AW: AW: AW: AW: [firebird-python] FDB v1.4.2 [1 Attachment]
Hi Pavel,
I believe that I have found a solution by shuffling the __wait_for_events()
invoke. This is the patch (also available as attachment):
--- C:/Python27/Lib/site-packages/fdb-1.4.3-py2.7.egg/fdb/fbcore.py Mo Nov
17 08:27:57 2014
+++ C:/Python27/Lib/site-packages/fdb-1.4.3-py2.7.egg/fdb/fbcore.py Mo Nov
17 08:29:08 2014
@@ -1729,9 +1729,10 @@ class EventBlock(object):
self.buf_length = api.isc_event_block(ctypes.pointer(self.event_buf),
ctypes.pointer(self.result_buf),
*[b(x) for x in event_names])
-self.__wait_for_events()
def __lt__(self,other):
return self.event_id.value other.event_id.value
+def begin(self):
+self.__wait_for_events()
def __wait_for_events(self):
api.isc_que_events(self._isc_status,self._db_handle,self.event_id,
self.buf_length,self.event_buf,
@@ -1841,7 +1842,9 @@ class EventConduit(object):
for block_events in self.__blocks:
-self.__event_blocks.append(EventBlock(self.__queue,
self._db_handle, block_events))
+event_block = EventBlock(self.__queue, self._db_handle,
block_events)
+self.__event_blocks.append(event_block)
+event_block.begin()
With this, even after 5+ events raised and processed and 2400+ event
conduits created and disposed, gc.garbage doesn't contain EventBlock items
anymore. *HOORAY!* :-)
While inspecting gc.garbage I have further noticed that it does contain several
fdb.fbcore.PreparedStatement items. According to
https://docs.python.org/2/library/gc.html#gc.garbage :
By default, this list contains only objects with __del__() methods. [...] Note
that these objects are kept alive even so by virtue of being in the garbage
list, so they should be removed from garbage too. For example, after breaking
cycles, do del gc.garbage[:] to empty the list. It’s generally better to avoid
the issue by not creating cycles containing objects with __del__() methods, and
garbage can be examined in that case to verify that no such cycles are being
created.
To me some obvious questions arise:
1] Is this behavior expected?
2] If [1], does fdb clean up gc.garbage from time to time?
3] If not [1], should fdb.fbcore.PreparedStatement better not override __del__
such that gc can do its job?
4] If not [2] and not [3], should fdb clean up gc.garbage?
Please note also that the memory foodprint of the fdb.fbcore.PreparedStatement
items in question is minimal because all containing data in
fdb.fbcore.PreparedStatement is freed. Due to this this issue is not
problematic in normal environments, but might become critical in an
environment that runs for a long time under heavy load.
Cheers,
Dominik
Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 15:03
An: firebird-python@yahoogroups.com
Betreff: Re: AW: AW: AW: [firebird-python] FDB v1.4.2
Hi,
Can you try to add a call to self.close() before exception is raised in
__wait_for_events ? Does it helps?
if db_api_error(self._isc_status):
self.close()
raise exception_from_status(DatabaseError, self._isc_status, Error
while waiting for events:)
best regards
Pavel
Dne 14.11.2014 v 14:08 Dominik Psenner dominik.psen...@topcontrol.it
[firebird-python] napsal(a):
I refreshed the package and it looks like the memory leak is almost gone. I
can observe that the number of EventBlock’s left in memory are equal to the
count of errors raised with the invoke of begin() (namely one of: unknown ISC
error 0, error reading from connection, error writing to connection).
So there’s still a troublesome execution path left and it took some time
until I had wrapped my head around it. According to the stack trace the
exception is thrown in the constructor of EventBlock:
File dbEvents.py, line 103, in run
event_listener.begin()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1844, in begin
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1732, in __init__
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1741, in __wait_for_events
Error while waiting for events:)
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error
reading data from the connection.', -902, 335544726)
That would be right here where the EventBlock should be appended to the list
of event blocks:
for block_events in self.__blocks:
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
But like this an initialized and broken EventBlock is never appended to the
__event_blocks list and therefore EventBlock.close() is never invoked. ☹
Von: firebird-python@yahoogroups.com
AW: AW: [firebird-python] FDB v1.4.2 [1 Attachment]
Ah, you were faster than I could write. Sorry for me being so persistent. Unfortunately I am unable to post a file in the bug tracker now that the issue is resolved, therefore I write you here. The good thing is, 1.4.3 improved the situation. One cyclic reference has gone away but there’s still one cyclic reference left to be broken. The attached objgraph shows that the one left might be the callback function which is stored in the EventBlock upon creation. You could break that by cleaning up the callback attribute on close(). Doing that should resolve this issue for good. Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 14. November 2014 12:06 An: firebird-python@yahoogroups.com Betreff: AW: AW: [firebird-python] FDB v1.4.2 Awesome! Thanks for the fast responses, you’re doing a good job. ☺ If you feel like sending me a patch that I can apply before you push 1.4.3 out of the door, I’ll gladly run my test scripts on a patched 1.4.2. Cheers Von: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 14. November 2014 11:50 An: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com Betreff: Re: AW: [firebird-python] FDB v1.4.2 Hi, Fixed in 1.4.3, see PyPI. Thanks for pointing this out. best regards Pavel Cisar IBPhoenix Dne 14.11.2014 v 11:42 Dominik Psenner dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.it [firebird-python] napsal(a): Hi Pavel, great to hear that you have got this release out of the door so quickly. However, I’m afraid that http://tracker.firebirdsql.org/browse/PYFB-43 is not yet resolved. As posted in the issue I was just now able to reproduce the memory leak with 1.4.2. Thus, people, beware from invoking event_conduit() in a loop! Are there any plans for fixing the memory leak? Best regards, Dominik Von: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 14. November 2014 10:36 An: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com Betreff: [firebird-python] FDB v1.4.2 Hi, FDB Python driver for Firebird v1.4.2 is available for download . This mostly bugfix version introduces a backward incompatible change to events API, so read the Release Notes before deployment. https://pypi.python.org/pypi/fdb best regards Pavel Cisar IBPhoenix
AW: AW: AW: [firebird-python] FDB v1.4.2
I refreshed the package and it looks like the memory leak is almost gone. I can
observe that the number of EventBlock’s left in memory are equal to the count
of errors raised with the invoke of begin() (namely one of: unknown ISC error
0, error reading from connection, error writing to connection).
So there’s still a troublesome execution path left and it took some time until
I had wrapped my head around it. According to the stack trace the exception is
thrown in the constructor of EventBlock:
File dbEvents.py, line 103, in run
event_listener.begin()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1844, in begin
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1732, in __init__
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1741, in __wait_for_events
Error while waiting for events:)
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error
reading data from the connection.', -902, 335544726)
That would be right here where the EventBlock should be appended to the list of
event blocks:
for block_events in self.__blocks:
self.__event_blocks.append(EventBlock(self.__queue,
self._db_handle, block_events))
But like this an initialized and broken EventBlock is never appended to the
__event_blocks list and therefore EventBlock.close() is never invoked. ☹
Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 13:38
An: firebird-python@yahoogroups.com
Betreff: Re: AW: AW: [firebird-python] FDB v1.4.2
Hi,
Fixed. This time I've just refreshed the 1.4.3 package, as I don't want
to make another point release in single day. If anyone already got
unpatched 1.4.3 (unlikely that anyone beside you get it so quickly),
then it will replaced by next version 1.5 (full Firebird 3.0 support)
scheduled for release before end of year.
best regards
Pavel Cisar
IBPhoenix
Dne 14.11.2014 v 12:24 Dominik Psenner
dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.it
[firebird-python] napsal(a):
Ah, you were faster than I could write. Sorry for me being so persistent.
Unfortunately I am unable to post a file in the bug tracker now that the
issue is resolved, therefore I write you here.
The good thing is, 1.4.3 improved the situation. One cyclic reference has
gone away but there’s still one cyclic reference left to be broken. The
attached objgraph shows that the one left might be the callback function
which is stored in the EventBlock upon creation. You could break that by
cleaning up the callback attribute on close(). Doing that should resolve this
issue for good.
Von: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com
[mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 12:06
An: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com
Betreff: AW: AW: [firebird-python] FDB v1.4.2
Awesome! Thanks for the fast responses, you’re doing a good job. ☺ If you
feel like sending me a patch that I can apply before you push 1.4.3 out of
the door, I’ll gladly run my test scripts on a patched 1.4.2.
Cheers
Von:
firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com%3cmailto:firebird-python@yahoogroups.com
[mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 11:50
An:
firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com%3cmailto:firebird-python@yahoogroups.com
Betreff: Re: AW: [firebird-python] FDB v1.4.2
Hi,
Fixed in 1.4.3, see PyPI. Thanks for pointing this out.
best regards
Pavel Cisar
IBPhoenix
Dne 14.11.2014 v 11:42 Dominik Psenner
dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.it%3cmailto:dominik.psen...@topcontrol.it
[firebird-python] napsal(a):
Hi Pavel,
great to hear that you have got this release out of the door so quickly.
However, I’m afraid that http://tracker.firebirdsql.org/browse/PYFB-43 is
not yet resolved. As posted in the issue I was just now able to reproduce
the memory leak with 1.4.2. Thus, people, beware from invoking
event_conduit() in a loop!
Are there any plans for fixing the memory leak?
Best regards,
Dominik
Von:
firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com%3cmailto:firebird-python@yahoogroups.com
[mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 10:36
An:
firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com%3cmailto:firebird-python@yahoogroups.com
Betreff: [firebird-python] FDB v1.4.2
Hi,
FDB Python driver
AW: AW: AW: AW: [firebird-python] FDB v1.4.2
*hm* .. not really.. then an AttributeError pops up:
2014-11-14 15:15:08.498000 50/51 (1|50:51|3|0)
event_conduit(['TC_IMPORT_QUEUE_UPDATED'])Exception in thread Thread-6:
Traceback (most recent call last):
File C:\Python27\lib\threading.py, line 810, in __bootstrap_inner
self.run()
File C:\Python27\lib\threading.py, line 763, in run
self.__target(*self.__args, **self.__kwargs)
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1831, in event_process
events = data.count_and_reregister()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1751, in count_and_reregister
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1742, in __wait_for_events
Error while waiting for events:)
DatabaseError: ('Error while waiting for events:\n- SQLCODE: 0\n- unknown ISC
error 0', 0, 0)
2014-11-14 15:15:08.608000 50/52 (2|25:26|4|1) Traceback (most recent call
last):
File dbEvents.py, line 103, in run
event_listener.begin()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1845, in begin
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1732, in __init__
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1740, in __wait_for_events
self.close()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1763, in close
del self.__callback
AttributeError: _EventBlock__callback
Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 15:03
An: firebird-python@yahoogroups.com
Betreff: Re: AW: AW: AW: [firebird-python] FDB v1.4.2
Hi,
Can you try to add a call to self.close() before exception is raised in
__wait_for_events ? Does it helps?
if db_api_error(self._isc_status):
self.close()
raise exception_from_status(DatabaseError, self._isc_status, Error
while waiting for events:)
best regards
Pavel
Dne 14.11.2014 v 14:08 Dominik Psenner
dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.it
[firebird-python] napsal(a):
I refreshed the package and it looks like the memory leak is almost gone. I
can observe that the number of EventBlock’s left in memory are equal to the
count of errors raised with the invoke of begin() (namely one of: unknown ISC
error 0, error reading from connection, error writing to connection).
So there’s still a troublesome execution path left and it took some time
until I had wrapped my head around it. According to the stack trace the
exception is thrown in the constructor of EventBlock:
File dbEvents.py, line 103, in run
event_listener.begin()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1844, in begin
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1732, in __init__
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1741, in __wait_for_events
Error while waiting for events:)
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error
reading data from the connection.', -902, 335544726)
That would be right here where the EventBlock should be appended to the list
of event blocks:
for block_events in self.__blocks:
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
But like this an initialized and broken EventBlock is never appended to the
__event_blocks list and therefore EventBlock.close() is never invoked. ☹
Von: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com
[mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 13:38
An: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com
Betreff: Re: AW: AW: [firebird-python] FDB v1.4.2
Hi,
Fixed. This time I've just refreshed the 1.4.3 package, as I don't want
to make another point release in single day. If anyone already got
unpatched 1.4.3 (unlikely that anyone beside you get it so quickly),
then it will replaced by next version 1.5 (full Firebird 3.0 support)
scheduled for release before end of year.
best regards
Pavel Cisar
IBPhoenix
Dne 14.11.2014 v 12:24 Dominik Psenner
dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.it%3cmailto:dominik.psen...@topcontrol.it
[firebird-python] napsal(a):
Ah, you were faster than I could write. Sorry for me being so persistent.
Unfortunately I am unable to post a file in the bug tracker now that the
issue is resolved, therefore I write you here.
The good thing is, 1.4.3 improved the situation. One cyclic reference has
gone away but there’s still one cyclic reference left to
AW: [firebird-python] Memory leak when using events [1 Attachment]
Hi Pavel, Happy to read you. Sure thing, come up with a fix and I'll do my best. However, I attach a little python script to this email that does reproduce the issue within a few cycles. This should ease your job a little. :-) For the records, I have reported this issue as bug #PYFB-43. Cheers and read you back soon, Dominik -Ursprüngliche Nachricht- Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 7. November 2014 12:05 An: firebird-python@yahoogroups.com Betreff: Re: [firebird-python] Memory leak when using events Hi, This is definitely a bug in FDB event handling, but tricky one. It seems to me that root of this problem is that exception raised in EventBlock.__wait_for_events is not properly handled in upper levels, which in turn causes accumulation of blocks in queue. The circular reference you pointed out just made it visible. So fixing the circular reference with weakref.proxy will probably not fix the problem. I'm working on proper fix, but it's hard to test. Can you help me with testing if I'll create a patch? best regards Pavel Cisar IBPhoenix Dne 7.11.2014 v 08:56 Dominik Psenner dominik.psen...@topcontrol.it [firebird-python] napsal(a): Good morning, as written yesterday we have recently implemented a way to react on database events in our python based program. But we are facing several outages with that implementation. First we encountered at least three errors when invoking event_conduit(events), namely: * Error reading data from the connection. * Error writing data to the connection. * unknown ISC error 0 Then I had the suspicion that the private memory was increasing and never dropping, thus I investigated a little further and used the dummy program from yesterday and let it run overnight. It produced and consumed about 1.250.000 events and the process memory raised from 27mb to roughly 400mb. Luckily I was wise enough to start the test script with pdb and this is the outcome: A memory leak in the class EventBlock(object), defined in dbcore.py at line 1687 caused by the cyclic reference introduced in its constructor at lines 1705 and 1709: 1705:self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self)) 1709: self.__queue = queue The events issue is now officially promoted to be a blocker for us. Looking at what the code does, it looks mostly like a chicken and egg problem. Ideas / suggestions? Is this the right place to come up with such problems (i.e. do the maintainers of the fdb python package read this mailinglist)? Best regards, Dominik Yahoo Groups Links
AW: AW: [firebird-python] Memory leak when using events
Apologies, I should have explained that better. :-) The good thing (or bad?) is, that the database can be any firebird database. It must not even raise events. Take any database you want and modify the first few variables in the __main__ section such that it can connect to a database you have on your computer. You could also adjust the timeout and minTimeout to smaller values to get the results faster. Cheers -Ursprüngliche Nachricht- Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 7. November 2014 12:28 An: firebird-python@yahoogroups.com Betreff: Re: AW: [firebird-python] Memory leak when using events Hi, Well, the script is nice but I can't use it without the database it uses :) best regards Pavel Cisar IBPhoenix Dne 7.11.2014 v 12:19 Dominik Psenner dominik.psen...@topcontrol.it [firebird-python] napsal(a): Hi Pavel, Happy to read you. Sure thing, come up with a fix and I'll do my best. However, I attach a little python script to this email that does reproduce the issue within a few cycles. This should ease your job a little. :-) For the records, I have reported this issue as bug #PYFB-43. Cheers and read you back soon, Dominik -Ursprüngliche Nachricht- Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 7. November 2014 12:05 An: firebird-python@yahoogroups.com Betreff: Re: [firebird-python] Memory leak when using events Hi, This is definitely a bug in FDB event handling, but tricky one. It seems to me that root of this problem is that exception raised in EventBlock.__wait_for_events is not properly handled in upper levels, which in turn causes accumulation of blocks in queue. The circular reference you pointed out just made it visible. So fixing the circular reference with weakref.proxy will probably not fix the problem. I'm working on proper fix, but it's hard to test. Can you help me with testing if I'll create a patch? best regards Pavel Cisar IBPhoenix Dne 7.11.2014 v 08:56 Dominik Psenner dominik.psen...@topcontrol.it [firebird-python] napsal(a): Good morning, as written yesterday we have recently implemented a way to react on database events in our python based program. But we are facing several outages with that implementation. First we encountered at least three errors when invoking event_conduit(events), namely: * Error reading data from the connection. * Error writing data to the connection. * unknown ISC error 0 Then I had the suspicion that the private memory was increasing and never dropping, thus I investigated a little further and used the dummy program from yesterday and let it run overnight. It produced and consumed about 1.250.000 events and the process memory raised from 27mb to roughly 400mb. Luckily I was wise enough to start the test script with pdb and this is the outcome: A memory leak in the class EventBlock(object), defined in dbcore.py at line 1687 caused by the cyclic reference introduced in its constructor at lines 1705 and 1709: 1705:self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self)) 1709: self.__queue = queue The events issue is now officially promoted to be a blocker for us. Looking at what the code does, it looks mostly like a chicken and egg problem. Ideas / suggestions? Is this the right place to come up with such problems (i.e. do the maintainers of the fdb python package read this mailinglist)? Best regards, Dominik Yahoo Groups Links Posted by: Dominik Psenner dominik.psen...@topcontrol.it Yahoo Groups Links Yahoo Groups Links
AW: AW: AW: [firebird-python] Memory leak when using events
Bug or pitfall does not matter since there's no way to work around the issue. The problem is mainly caused by the invoke of self.__wait_for_events() in the constructor, isn't it? So what was if you changed the API such that one would: eventNames = ['EVENT'] handle = conn.event_conduit(eventNames) try: # begin listening for events handle.begin() # wait until events are there events = handle.wait(timeout) # process events except: pass #or whatever would be appropriate finally: handle.close() or maybe even fancier by providing an __enter__ and __exit__ method: with conn.event_conduit(eventNames) as handle: events = handle.wait(timeout) # process events Where the __enter__ method called self.begin() and __exit__ called self.close(). This way the constructor would no longer have to do fancy things that could break. Handling exceptional cases would after all be left up to who uses the API. Cheers -Ursprüngliche Nachricht- Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 7. November 2014 13:33 An: firebird-python@yahoogroups.com Betreff: Re: AW: AW: [firebird-python] Memory leak when using events Hi, thanks for the app, it helped to look into this issue. However, after deep look I start to think that this is actually not a bug, just pitfall. The problem is that when some problem arises while (re)registering interest in event notifications (isc_que_events API call), there is no good way how to handle it and continue with event processing using given conduit (threading issues aside). So exception has to be propagated up to the application level and handled there properly, i.e. EventConduit has to be considered invalid and discarded. I can do next things to improve the situation: 1. Use weakref.proxy to queue from EventBlock, so circular reference through queue content would be broken. 2. Provide access to internal events dictionary in EventConduit (it would return a copy like wait does), so event counts accumulated so far could be retrieved in case of failure. The side issue is that events seems to be quite fragile (isc_que_events shouldn't fail so easily and especially with no good reason like Unknown ISC error I'm getting from your sample app), and I don't know why. I'll consult that with core developers and report here. best regards Pavel Cisar IBPhoenix Dne 7.11.2014 v 12:43 Dominik Psenner dominik.psen...@topcontrol.it [firebird-python] napsal(a): Apologies, I should have explained that better. :-) The good thing (or bad?) is, that the database can be any firebird database. It must not even raise events. Take any database you want and modify the first few variables in the __main__ section such that it can connect to a database you have on your computer. You could also adjust the timeout and minTimeout to smaller values to get the results faster. Cheers -Ursprüngliche Nachricht- Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 7. November 2014 12:28 An: firebird-python@yahoogroups.com Betreff: Re: AW: [firebird-python] Memory leak when using events Hi, Well, the script is nice but I can't use it without the database it uses :) best regards Pavel Cisar IBPhoenix Dne 7.11.2014 v 12:19 Dominik Psenner dominik.psen...@topcontrol.it [firebird-python] napsal(a): Hi Pavel, Happy to read you. Sure thing, come up with a fix and I'll do my best. However, I attach a little python script to this email that does reproduce the issue within a few cycles. This should ease your job a little. :-) For the records, I have reported this issue as bug #PYFB-43. Cheers and read you back soon, Dominik -Ursprüngliche Nachricht- Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] Gesendet: Freitag, 7. November 2014 12:05 An: firebird-python@yahoogroups.com Betreff: Re: [firebird-python] Memory leak when using events Hi, This is definitely a bug in FDB event handling, but tricky one. It seems to me that root of this problem is that exception raised in EventBlock.__wait_for_events is not properly handled in upper levels, which in turn causes accumulation of blocks in queue. The circular reference you pointed out just made it visible. So fixing the circular reference with weakref.proxy will probably not fix the problem. I'm working on proper fix, but it's hard to test. Can you help me with testing if I'll create a patch? best regards Pavel Cisar IBPhoenix Dne 7.11.2014 v 08:56 Dominik Psenner dominik.psen...@topcontrol.it [firebird-python] napsal(a): Good morning, as written yesterday we have recently implemented a way to react on database events in our python based program. But we are facing several outages with that implementation. First we encountered at least three errors when invoking event_conduit
[firebird-python] Broken firebird events? [1 Attachment]
Good morning,
we have recently implemented a worker that listens and reacts to database
events. It is written in python and uses the most recent python driver (1.4.1)
together with the latest firebird server (2.5.3). However, the following three
exceptions are catched regularily even if there are no events raised by the
database. The next cycle after one of those exceptions in the loop executes
just fine and even if the database raised an event there it is caught and
handled properly:
File fdb\fbcore.pyo, line 1601, in event_conduit
File fdb\fbcore.pyo, line 1811, in __init__
File fdb\fbcore.pyo, line 1726, in __init__
File fdb\fbcore.pyo, line 1735, in __wait_for_events
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error
reading data from the connection.', -902, 335544726)
File fdb\fbcore.pyo, line 1601, in event_conduit
File fdb\fbcore.pyo, line 1811, in __init__
File fdb\fbcore.pyo, line 1726, in __init__
File fdb\fbcore.pyo, line 1735, in __wait_for_events
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error
writing data from the connection.', -902, 335544726)
File fdb\fbcore.pyo, line 1601, in event_conduit
File fdb\fbcore.pyo, line 1811, in __init__
File fdb\fbcore.pyo, line 1726, in __init__
File fdb\fbcore.pyo, line 1735, in __wait_for_events
DatabaseError: ('Error while waiting for events:\n- SQLCODE: 0\n- unknown ISC
error 0', 0, 0)
I attached a small sample program that reproduces the issue. Any ideas what
could cause the issue? Can we safely ignore the error?
Best regards,
Dominik
[firebird-python] Memory leak when using events
Good morning, as written yesterday we have recently implemented a way to react on database events in our python based program. But we are facing several outages with that implementation. First we encountered at least three errors when invoking event_conduit(events), namely: * Error reading data from the connection. * Error writing data to the connection. * unknown ISC error 0 Then I had the suspicion that the private memory was increasing and never dropping, thus I investigated a little further and used the dummy program from yesterday and let it run overnight. It produced and consumed about 1.250.000 events and the process memory raised from 27mb to roughly 400mb. Luckily I was wise enough to start the test script with pdb and this is the outcome: A memory leak in the class EventBlock(object), defined in dbcore.py at line 1687 caused by the cyclic reference introduced in its constructor at lines 1705 and 1709: 1705:self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self)) 1709: self.__queue = queue The events issue is now officially promoted to be a blocker for us. Looking at what the code does, it looks mostly like a chicken and egg problem. Ideas / suggestions? Is this the right place to come up with such problems (i.e. do the maintainers of the fdb python package read this mailinglist)? Best regards, Dominik
