Hi,

thanks for the app, it helped to look into this issue. However, after 
deep look I start to think that this is actually not a bug, just 
pitfall. The problem is that when some problem arises while 
(re)registering interest in event notifications (isc_que_events API 
call), there is no good way how to handle it and continue with event 
processing using given conduit (threading issues aside). So exception 
has to be propagated up to the application level and handled there 
properly, i.e. EventConduit has to be considered invalid and discarded.

I can do next things to improve the situation:

1. Use weakref.proxy to queue from EventBlock, so circular reference 
through queue content would be broken.
2. Provide access to internal events dictionary in EventConduit (it 
would return a copy like wait does), so event counts accumulated so far 
could be retrieved in case of failure.

The side issue is that events seems to be quite fragile (isc_que_events 
shouldn't fail so easily and especially with no good reason like 
"Unknown ISC error" I'm getting from your sample app), and I don't know 
why. I'll consult that with core developers and report here.

best regards
Pavel Cisar
IBPhoenix

Dne 7.11.2014 v 12:43 Dominik Psenner dominik.psen...@topcontrol.it 
[firebird-python] napsal(a):
> Apologies, I should have explained that better. :-) The good thing (or bad?) 
> is, that the database can be any firebird database. It must not even raise 
> events. Take any database you want and modify the first few variables in the 
> __main__ section such that it can connect to a database you have on your 
> computer. You could also adjust the timeout and minTimeout to smaller values 
> to get the results faster.
>
> Cheers
>
> -----Urspr√ľngliche Nachricht-----
> Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com]
> Gesendet: Freitag, 7. November 2014 12:28
> An: firebird-python@yahoogroups.com
> Betreff: Re: AW: [firebird-python] Memory leak when using events
>
> Hi,
>
> Well, the script is nice but I can't use it without the database it uses :)
>
> best regards
> Pavel Cisar
> IBPhoenix
>
> Dne 7.11.2014 v 12:19 Dominik Psenner dominik.psen...@topcontrol.it 
> [firebird-python] napsal(a):
>> Hi Pavel,
>>
>> Happy to read you. Sure thing, come up with a fix and I'll do my best. 
>> However, I attach a little python script to this email that does reproduce 
>> the issue within a few cycles. This should ease your job a little. :-) For 
>> the records, I have reported this issue as bug #PYFB-43.
>>
>> Cheers and read you back soon,
>> Dominik
>>
>> -----Urspr√ľngliche Nachricht-----
>> Von: firebird-python@yahoogroups.com
>> [mailto:firebird-python@yahoogroups.com]
>> Gesendet: Freitag, 7. November 2014 12:05
>> An: firebird-python@yahoogroups.com
>> Betreff: Re: [firebird-python] Memory leak when using events
>>
>> Hi,
>>
>> This is definitely a bug in FDB event handling, but tricky one. It seems to 
>> me that root of this problem is that exception raised in 
>> EventBlock.__wait_for_events is not properly handled in upper levels, which 
>> in turn causes accumulation of blocks in queue. The circular reference you 
>> pointed out just made it visible. So fixing the circular reference with 
>> weakref.proxy will probably not fix the problem. I'm working on proper fix, 
>> but it's hard to test. Can you help me with testing if I'll create a patch?
>>
>> best regards
>> Pavel Cisar
>> IBPhoenix
>>
>> Dne 7.11.2014 v 08:56 Dominik Psenner dominik.psen...@topcontrol.it 
>> [firebird-python] napsal(a):
>>> Good morning,
>>>
>>> as written yesterday we have recently implemented a way to react on 
>>> database events in our python based program. But we are facing several 
>>> outages with that implementation. First we encountered at least three 
>>> errors when invoking event_conduit(events), namely:
>>>
>>> *         Error reading data from the connection.
>>>
>>> *         Error writing data to the connection.
>>>
>>> *         unknown ISC error 0
>>>
>>> Then I had the suspicion that the private memory was increasing and never 
>>> dropping, thus I investigated a little further and used the dummy program 
>>> from yesterday and let it run overnight. It produced and consumed about 
>>> 1.250.000 events and the process memory raised from 27mb to roughly 400mb. 
>>> Luckily I was wise enough to start the test script with pdb and this is the 
>>> outcome:
>>>
>>> A memory leak in the class EventBlock(object), defined in dbcore.py at line 
>>> 1687 caused by the cyclic reference introduced in its constructor at lines 
>>> 1705 and 1709:
>>>
>>> 1705:            self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self))
>>> 1709:     self.__queue = queue
>>>
>>> The events issue is now officially promoted to be a blocker for us. Looking 
>>> at what the code does, it looks mostly like a chicken and egg problem. 
>>> Ideas / suggestions? Is this the right place to come up with such problems 
>>> (i.e. do the maintainers of the fdb python package read this mailinglist)?
>>>
>>> Best regards,
>>> Dominik
>>>
>>
>>
>> ------------------------------------
>>
>> ------------------------------------
>>
>>
>> ------------------------------------
>>
>> Yahoo Groups Links
>>
>>
>>
>>
>>
>>
>> ------------------------------------
>> Posted by: Dominik Psenner <dominik.psen...@topcontrol.it>
>> ------------------------------------
>>
>>
>> ------------------------------------
>>
>> Yahoo Groups Links
>>
>>
>>
>>
>
>
> ------------------------------------
>
> ------------------------------------
>
>
> ------------------------------------
>
> Yahoo Groups Links
>
>
>
>
>
> ------------------------------------
> Posted by: Dominik Psenner <dominik.psen...@topcontrol.it>
> ------------------------------------
>
>
> ------------------------------------
>
> Yahoo Groups Links
>
>
>
>


------------------------------------

------------------------------------


------------------------------------

Yahoo Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-dig...@yahoogroups.com 
    firebird-python-fullfeatu...@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscr...@yahoogroups.com

<*> Your use of Yahoo Groups is subject to:
    https://info.yahoo.com/legal/us/yahoo/utos/terms/

  • [firebird-... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]
    • Re: [... Pavel Cisar pci...@ibphoenix.cz [firebird-python]
      • A... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]
        • ... Pavel Cisar pci...@ibphoenix.cz [firebird-python]
          • ... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]
            • ... Pavel Cisar pci...@ibphoenix.cz [firebird-python]
              • ... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]
                • ... Pavel Cisar pci...@ibphoenix.cz [firebird-python]
                • ... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]

Reply via email to