I'm well aware that changing the API is a pain for everyone and thus it might 
be required to introduce another method (i.e. conn.get_event_conduit()) instead 
of changing the behavior of conn.event_conduit().

Nevertheless, it's awesome that you're already into fixing this issue and I 
hope to hear from you soon.

Cheers

-----Ursprüngliche Nachricht-----
Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com] 
Gesendet: Freitag, 7. November 2014 15:24
An: firebird-python@yahoogroups.com
Betreff: Re: AW: AW: AW: [firebird-python] Memory leak when using events

Hi,

I have to think more about implementation details and consequences but I like 
your solution. If no problem arises I'm going to make a minor release with this 
fix next week. We're also working with Vlad on "Unknown error" issue which is 
most likely a bug in FB 2.5 engine.

best regards
Pavel Cisar
IBPhoenix

Dne 7.11.2014 v 13:57 Dominik Psenner dominik.psen...@topcontrol.it 
[firebird-python] napsal(a):
> Bug or pitfall does not matter since there's no way to work around the issue. 
> The problem is mainly caused by the invoke of self.__wait_for_events() in the 
> constructor, isn't it? So what was if you changed the API such that one would:
>
> eventNames = ['EVENT']
> handle = conn.event_conduit(eventNames)
> try:
>       # begin listening for events
>       handle.begin()
>       # wait until events are there
>       events = handle.wait(timeout)
>       # process events
> except:
>       pass #or whatever would be appropriate
> finally:
>       handle.close()
>
> or maybe even fancier by providing an __enter__ and __exit__ method:
>
> with conn.event_conduit(eventNames) as handle:
>       events = handle.wait(timeout)
>       # process events
>
> Where the __enter__ method called self.begin() and __exit__ called 
> self.close(). This way the constructor would no longer have to do fancy 
> things that could break. Handling exceptional cases would after all be left 
> up to who uses the API.
>
> Cheers
>
> -----Ursprüngliche Nachricht-----
> Von: firebird-python@yahoogroups.com 
> [mailto:firebird-python@yahoogroups.com]
> Gesendet: Freitag, 7. November 2014 13:33
> An: firebird-python@yahoogroups.com
> Betreff: Re: AW: AW: [firebird-python] Memory leak when using events
>
> Hi,
>
> thanks for the app, it helped to look into this issue. However, after deep 
> look I start to think that this is actually not a bug, just pitfall. The 
> problem is that when some problem arises while (re)registering interest in 
> event notifications (isc_que_events API call), there is no good way how to 
> handle it and continue with event processing using given conduit (threading 
> issues aside). So exception has to be propagated up to the application level 
> and handled there properly, i.e. EventConduit has to be considered invalid 
> and discarded.
>
> I can do next things to improve the situation:
>
> 1. Use weakref.proxy to queue from EventBlock, so circular reference through 
> queue content would be broken.
> 2. Provide access to internal events dictionary in EventConduit (it would 
> return a copy like wait does), so event counts accumulated so far could be 
> retrieved in case of failure.
>
> The side issue is that events seems to be quite fragile (isc_que_events 
> shouldn't fail so easily and especially with no good reason like "Unknown ISC 
> error" I'm getting from your sample app), and I don't know why. I'll consult 
> that with core developers and report here.
>
> best regards
> Pavel Cisar
> IBPhoenix
>
> Dne 7.11.2014 v 12:43 Dominik Psenner dominik.psen...@topcontrol.it 
> [firebird-python] napsal(a):
>> Apologies, I should have explained that better. :-) The good thing (or bad?) 
>> is, that the database can be any firebird database. It must not even raise 
>> events. Take any database you want and modify the first few variables in the 
>> __main__ section such that it can connect to a database you have on your 
>> computer. You could also adjust the timeout and minTimeout to smaller values 
>> to get the results faster.
>>
>> Cheers
>>
>> -----Ursprüngliche Nachricht-----
>> Von: firebird-python@yahoogroups.com
>> [mailto:firebird-python@yahoogroups.com]
>> Gesendet: Freitag, 7. November 2014 12:28
>> An: firebird-python@yahoogroups.com
>> Betreff: Re: AW: [firebird-python] Memory leak when using events
>>
>> Hi,
>>
>> Well, the script is nice but I can't use it without the database it 
>> uses :)
>>
>> best regards
>> Pavel Cisar
>> IBPhoenix
>>
>> Dne 7.11.2014 v 12:19 Dominik Psenner dominik.psen...@topcontrol.it 
>> [firebird-python] napsal(a):
>>> Hi Pavel,
>>>
>>> Happy to read you. Sure thing, come up with a fix and I'll do my best. 
>>> However, I attach a little python script to this email that does reproduce 
>>> the issue within a few cycles. This should ease your job a little. :-) For 
>>> the records, I have reported this issue as bug #PYFB-43.
>>>
>>> Cheers and read you back soon,
>>> Dominik
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: firebird-python@yahoogroups.com 
>>> [mailto:firebird-python@yahoogroups.com]
>>> Gesendet: Freitag, 7. November 2014 12:05
>>> An: firebird-python@yahoogroups.com
>>> Betreff: Re: [firebird-python] Memory leak when using events
>>>
>>> Hi,
>>>
>>> This is definitely a bug in FDB event handling, but tricky one. It seems to 
>>> me that root of this problem is that exception raised in 
>>> EventBlock.__wait_for_events is not properly handled in upper levels, which 
>>> in turn causes accumulation of blocks in queue. The circular reference you 
>>> pointed out just made it visible. So fixing the circular reference with 
>>> weakref.proxy will probably not fix the problem. I'm working on proper fix, 
>>> but it's hard to test. Can you help me with testing if I'll create a patch?
>>>
>>> best regards
>>> Pavel Cisar
>>> IBPhoenix
>>>
>>> Dne 7.11.2014 v 08:56 Dominik Psenner dominik.psen...@topcontrol.it 
>>> [firebird-python] napsal(a):
>>>> Good morning,
>>>>
>>>> as written yesterday we have recently implemented a way to react on 
>>>> database events in our python based program. But we are facing several 
>>>> outages with that implementation. First we encountered at least three 
>>>> errors when invoking event_conduit(events), namely:
>>>>
>>>> *         Error reading data from the connection.
>>>>
>>>> *         Error writing data to the connection.
>>>>
>>>> *         unknown ISC error 0
>>>>
>>>> Then I had the suspicion that the private memory was increasing and never 
>>>> dropping, thus I investigated a little further and used the dummy program 
>>>> from yesterday and let it run overnight. It produced and consumed about 
>>>> 1.250.000 events and the process memory raised from 27mb to roughly 400mb. 
>>>> Luckily I was wise enough to start the test script with pdb and this is 
>>>> the outcome:
>>>>
>>>> A memory leak in the class EventBlock(object), defined in dbcore.py at 
>>>> line 1687 caused by the cyclic reference introduced in its constructor at 
>>>> lines 1705 and 1709:
>>>>
>>>> 1705:            self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self))
>>>> 1709:     self.__queue = queue
>>>>
>>>> The events issue is now officially promoted to be a blocker for us. 
>>>> Looking at what the code does, it looks mostly like a chicken and egg 
>>>> problem. Ideas / suggestions? Is this the right place to come up with such 
>>>> problems (i.e. do the maintainers of the fdb python package read this 
>>>> mailinglist)?
>>>>
>>>> Best regards,
>>>> Dominik
>>>>
>>>
>>>
>>> ------------------------------------
>>>
>>> ------------------------------------
>>>
>>>
>>> ------------------------------------
>>>
>>> Yahoo Groups Links
>>>
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------
>>> Posted by: Dominik Psenner <dominik.psen...@topcontrol.it>
>>> ------------------------------------
>>>
>>>
>>> ------------------------------------
>>>
>>> Yahoo Groups Links
>>>
>>>
>>>
>>>
>>
>>
>> ------------------------------------
>>
>> ------------------------------------
>>
>>
>> ------------------------------------
>>
>> Yahoo Groups Links
>>
>>
>>
>>
>>
>> ------------------------------------
>> Posted by: Dominik Psenner <dominik.psen...@topcontrol.it>
>> ------------------------------------
>>
>>
>> ------------------------------------
>>
>> Yahoo Groups Links
>>
>>
>>
>>
>
>
> ------------------------------------
>
> ------------------------------------
>
>
> ------------------------------------
>
> Yahoo Groups Links
>
>
>
>
>
> ------------------------------------
> Posted by: Dominik Psenner <dominik.psen...@topcontrol.it>
> ------------------------------------
>
>
> ------------------------------------
>
> Yahoo Groups Links
>
>
>
>


------------------------------------

------------------------------------


------------------------------------

Yahoo Groups Links



  • [firebird-... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]
    • Re: [... Pavel Cisar pci...@ibphoenix.cz [firebird-python]
      • A... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]
        • ... Pavel Cisar pci...@ibphoenix.cz [firebird-python]
          • ... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]
            • ... Pavel Cisar pci...@ibphoenix.cz [firebird-python]
              • ... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]
                • ... Pavel Cisar pci...@ibphoenix.cz [firebird-python]
                • ... Dominik Psenner dominik.psen...@topcontrol.it [firebird-python]

Reply via email to