Re: Design thought for callbacks
Steven D'Aprano : > Marko Rauhamaa wrote: >> Grant Edwards : >>> the polite thing to do is to delete references to it when you're done >>> with it. >> >> I disagree with that recommendation. You should do the natural thing and >> not care who holds references to who. > > I don't understand this. What is "the natural thing" if not to delete > references to an object when you are done with it? Normally you just let > things go out of scope, but if that won't happen, you have to take active > steps, such as calling del or setting the reference to None. Obviously, if you are hoarding objects in a collection, you're going to land in trouble. What I mean, though, is that you shouldn't think you need to create object destructors where you routinely set all members to None. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sat, Feb 21, 2015 at 1:57 PM, Grant Edwards wrote: > On 2015-02-21, Cem Karan wrote: >> >> On Feb 21, 2015, at 12:42 AM, Chris Angelico wrote: >> >>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about? >>> >>> No, it's not. I would advise using strong references - if the callback >>> is a closure, for instance, you need to hang onto it, because there >>> are unlikely to be any other references to it. If I register a >>> callback with you, I expect it to be called; I expect, in fact, that >>> that *will* keep my object alive. >> >> OK, so it would violate the principle of least surprise for you. > > And me as well. I would expect to be able to pass a closure as a > callback and not have to keep a reference to it. Perhaps that just a > leftover from working with other languages (javascript, scheme, etc.). > It doesn't matter if it's a string, a float, a callback, a graphic or > whatever: if I pass your function/library an object, I expect _you_ to > keep track of it until you're done with it. > >> Interesting. Is this a general pattern in python? That is, >> callbacks are owned by what they are registered with? > > I'm not sure what you mean by "owned" or why it matters that it's a > callback: it's an object that was passed to you: you need to hold onto > a reference to it until you're done with it, and the polite thing to > do is to delete references to it when you're done with it. > >> So, what's the consensus on the list, strongly-held callbacks, or >> weakly-held ones? Count me in the weak-ref crowd. It may be a nuisance to keep a reference around on the object registering the callback, but it's preferable to the alternative of messing around with disposables in order to ensure that the callback gets cleaned up and doesn't create a memory leak. I would also rather have my code fail by losing a callback reference, which should be relatively easy to spot and diagnose, than to have said memory leak go unnoticed. -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
"Steven D'Aprano" wrote in message news:54e8af1b$0$12976$c3e8da3$54964...@news.astraweb.com... > Frank Millman wrote: > >> I tried something similar a while ago, and I did find a gotcha. >> >> The problem lies in this phrase - "if they are no longer alive, they are >> automatically removed from the WeakSet, preventing me from accidentally >> calling them when they are dead." >> >> I found that the reference was not removed immediately, but was waiting >> to >> be garbage collected. During that window, I could call the callback, >> which >> resulted in an error. > > I don't understand how this could possibly work. (Or fail to work, as the > case may be.) > > If the callback has been garbage collected, then you cannot call it, > because > you don't have any references to it and so cannot refer to it in any way. > > If the callback has *not* been garbage collected, then you can safely call > it. You have a reference to the callback, therefore it exists. (If Python > ever garbage collects an object that still has references to it, that > would > be a critical bug, and you would likely get some sort of seg fault). > > The only thing I can think of is you have a situation where your callback > refers to another object, B, via a weak reference. Once all the regular > strong references to the callback and B are gone, theoretically you could > have a race condition where the callback is waiting to be garbage > collected > but B has already been garbage collected. If, in that window, you call the > callback, *and* if the callback fails to correctly check that the weak > reference to B still exists, then you could get a Python exception. > > The solution is simple: anytime you have a weak reference, you must always > check it before you use it. > > Other than that, I cannot see how calling a function which has *not* yet > been garbage collected can fail, just because the only reference still > existing is a weak reference. > You are right. I tried to reproduce the problem and I can't. Before describing what I think was happening, I want to clarify something. Most of this thread uses the word 'callback' in the sense of an 'asynchronous' scenario - the caller wants something to happen some time in the future, and then forget about it, but it is important that it does actually happen eventually. That is not what I was doing, and it is not what I thought the OP was asking for. "In order to inform users that certain bits of state have changed, I require them to register a callback with my code." This sounds to me like a pub/sub scenario. When a 'listener' object comes into existence it is passed a reference to a 'controller' object that holds state. It wants to be informed when the state changes, so it registers a callback function with the controller. When the controller detects a change in state, it calls all the callback functions, thereby notifying each listener. When the listener goes out of scope, it is important that it deregisters with the controller. Now back to my scenario. You are right that so long as the controller maintains a reference to the callback function, the listener cannot be garbage collected, and therefore the callback will always succeed. As far as I can remember, I had a situation where the listener used the information to pass the information to a gui on a client. When the listener was no longer required, a close() fiunction was called which cleaned up and closed connections. When the callback was called after the listener was closed, whatever it was trying to do failed (I forget the details). Hope this makes sense. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 3:38 PM, Steven D'Aprano wrote: > But you are using it. You might not be using it by name, but you are using > it via the callback function. What did you expect, that Python should read > your mind and somehow intuit that you still care about this socket > listener, but not some other socket listener that you are done with? > > You don't have to bind the listener to a name. Any reference will do. You > can dump it in a bucket: > > bucket_of_stuff = [] > bucket_of_stuff.append(some_function(a, b, c)) > bucket_of_stuff.append(make_web_server()) > bucket_of_stuff.append(socket(23, on_accept=client_connected)) Sure, and whether it's a name or a list-element reference doesn't matter: it seems wrong to have to stash a thing in a bucket in order to keep its callbacks alive. I expect the callbacks _themselves_ to keep it alive. But I can understand the opposite POV. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How to design a search engine in Python?
On Sunday, February 22, 2015 at 11:08:47 AM UTC+5:30, Denis McMahon wrote: > On Sat, 21 Feb 2015 21:02:34 -0800, subhabangalore wrote: > > > Thank you for your suggestion. But I was looking for a small tutorial of > > algorithm of the whole engine. I would try to check it build individual > > modules and integrate them. I was getting some in google and youtube, > > but I tried to consult you as I do not know whether they would be fine. > > I am trying your way, let me see how much I go. There are so many search > > algorithms in our popular data structure books, that is not an issue but > > how a search engine is getting done, I am thinking bit on that. > > Presumably a search engine is simply a database of keyword -> result, > possibly with some scoring factor. > > Calculating scoring factor is going to be fun. > > Then of course result pages might have scoring factors too. What about a > search with multiple keywords. Some result pages might match more than > one keyword, so you might add their score for each keyword together to > get the ranking in that enquiry for that page. > > But then pages with lots and lots of different keywords might be low > scoring, because searchers are looking for content, not pages of keywords. > > Finally, What special, unique feature is your search engine going to have > that makes it better than all the existing ones? > > -- > Denis McMahon, Dear Sir, Thank you for your kind suggestion. Let me traverse one by one. My special feature is generally Semantic Search, but I am trying to build a search engine first and then go for semantic I feel that would give me a solid background to work around the problem. Regards, Subhabrata. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to design a search engine in Python?
On Sat, 21 Feb 2015 21:02:34 -0800, subhabangalore wrote: > Thank you for your suggestion. But I was looking for a small tutorial of > algorithm of the whole engine. I would try to check it build individual > modules and integrate them. I was getting some in google and youtube, > but I tried to consult you as I do not know whether they would be fine. > I am trying your way, let me see how much I go. There are so many search > algorithms in our popular data structure books, that is not an issue but > how a search engine is getting done, I am thinking bit on that. Presumably a search engine is simply a database of keyword -> result, possibly with some scoring factor. Calculating scoring factor is going to be fun. Then of course result pages might have scoring factors too. What about a search with multiple keywords. Some result pages might match more than one keyword, so you might add their score for each keyword together to get the ranking in that enquiry for that page. But then pages with lots and lots of different keywords might be low scoring, because searchers are looking for content, not pages of keywords. Finally, What special, unique feature is your search engine going to have that makes it better than all the existing ones? -- Denis McMahon, denismfmcma...@gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: How to design a search engine in Python?
On Sunday, February 22, 2015 at 10:12:39 AM UTC+5:30, Steven D'Aprano wrote: > wrote: > > > Dear Group, > > > > I am trying to build a search engine in Python. > > How to design a search engine in Python? > > First, design a search engine. > > Then, write Python code to implement that search engine. > > > > To do this, I have read tutorials and working methodologies from web and > > books like Stanford IR book [ http://www-nlp.stanford.edu/IR-book/]. I > > know how to design a crawler, I know PostgresSql, I am fluent with > > PageRank, TF-IDF, Zipf's law, etc. I came to know of > > Whoosh[https://pypi.python.org/pypi/Whoosh/] > > How does your search engine work? What does it do? > > You MUST be able to describe the workings of your search engine in English, > or the natural language of your choice. Write out the steps that it must > take, the tasks that it must perform. This is your algorithm. Without an > algorithm, how do you expect to write code? What will the code do? > > Once you have designed your search engine algorithm, then *and only then* > should you start to write code to implement that algorithm. > > > > > -- > Steven Dear Sir, Thank you for your suggestion. But I was looking for a small tutorial of algorithm of the whole engine. I would try to check it build individual modules and integrate them. I was getting some in google and youtube, but I tried to consult you as I do not know whether they would be fine. I am trying your way, let me see how much I go. There are so many search algorithms in our popular data structure books, that is not an issue but how a search engine is getting done, I am thinking bit on that. Regards, Subhabrata. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to design a search engine in Python?
subhabangal...@gmail.com wrote: > Dear Group, > > I am trying to build a search engine in Python. How to design a search engine in Python? First, design a search engine. Then, write Python code to implement that search engine. > To do this, I have read tutorials and working methodologies from web and > books like Stanford IR book [ http://www-nlp.stanford.edu/IR-book/]. I > know how to design a crawler, I know PostgresSql, I am fluent with > PageRank, TF-IDF, Zipf's law, etc. I came to know of > Whoosh[https://pypi.python.org/pypi/Whoosh/] How does your search engine work? What does it do? You MUST be able to describe the workings of your search engine in English, or the natural language of your choice. Write out the steps that it must take, the tasks that it must perform. This is your algorithm. Without an algorithm, how do you expect to write code? What will the code do? Once you have designed your search engine algorithm, then *and only then* should you start to write code to implement that algorithm. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Chris Angelico wrote:
> On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
> wrote:
>> Marko Rauhamaa wrote:
>>
>>> Grant Edwards :
>>>
the polite thing to do is to delete references to it when you're done
with it.
>>>
>>> I disagree with that recommendation. You should do the natural thing and
>>> not care who holds references to who.
>>
>> I don't understand this. What is "the natural thing" if not to delete
>> references to an object when you are done with it? Normally you just let
>> things go out of scope, but if that won't happen, you have to take active
>> steps, such as calling del or setting the reference to None.
>
> I think the disagreement here is over the interpretation of "done with
> it". If you drop all references to a connected socket object, Python
> can rightly assume that you're done with the file and want to close
> it; but what if you drop all references to a listening socket that's
> been configured to call a function whenever someone connects?
>
> def client_connected(sock):
> sock.send("Hello!\r\n")
> # whatever
>
> listener = socket(23, on_accept=client_connected)
>
> What should happen if that main socket isn't bound to a name? In my
> opinion, the fact that it's configured for callback mode should mean
> that it's kept alive. But it's also understandable to want to treat it
> as "done", that it can be disposed of. It seems weird to me that you
> should have to have a name somewhere that you'll never use, though.
But you are using it. You might not be using it by name, but you are using
it via the callback function. What did you expect, that Python should read
your mind and somehow intuit that you still care about this socket
listener, but not some other socket listener that you are done with?
If you have a servant that follows you around everywhere, throwing objects
away when you stop using them, then naturally if you stop using something
it will be thrown away. What did you expect?
You don't have to bind the listener to a name. Any reference will do. You
can dump it in a bucket:
bucket_of_stuff = []
bucket_of_stuff.append(some_function(a, b, c))
bucket_of_stuff.append(make_web_server())
bucket_of_stuff.append(socket(23, on_accept=client_connected))
So long as your bucket is still alive, the garbage collector won't collect
it or its contents.
Hypothetically we could have a system in place where you instruct the
garbage collector to not collect an object, then drop all references to it:
gc.never_collect(socket(23, on_accept=client_connected))
but that's a potential memory leak, because now you have no way of telling
the GC to collect it again once you've finished listening. If you never
finish listening, or at least not until your application shuts down, it
doesn't count as a leak. But in general, if listeners might come and go,
but you have no way for them to be garbage collected once they are done,
then that's a leak.
What's the easiest way for the GC to flag an object as "never collect this"?
It can keep a reference to it. Keeping a list of things you want to be kept
alive is simple, easy and obvious:
# hypothetically inside gc.py
_ALIVE = []
def never_collect(obj):
_ALIVE.append(obj)
but that's so trivial that it's not worth putting it in the gc module.
Besides, that means any other code could reach into the gc and remove your
listener from the "keep alive list" without your knowledge or permission.
It's simpler and safer to just keep it alive yourself:
alive = []
alive.append(socket(...))
but of course that's just my bucket of stuff under a different name.
Using the idiom "keep objects alive by keeping a reference to them" (e.g.
bind them to a name, or stick them in a list which you keep) makes things
much simpler. You can trivially flag objects as "collect" or "don't
collect" as needed, without having to import the gc module or memorise some
obscure API or worry about implementation details ("if I flag an object
as 'never collect' *twice*, do I have to unflag it twice to undo it?"). You
just use the regular interface to the garbage collector: the existence of a
reference, any reference, keeps an object alive.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements
TommyVee wrote:
> Start off with sets of elements as follows:
>
> 1. A,B,E,F
> 2. G,H,L,P,Q
> 3. C,D,E,F
> 4. E,X,Z
> 5. L,M,R
> 6. O,M,Y
>
> Note that sets 1, 3 and 4 all have the element 'E' in common, therefore
> they are "related" and form the following superset:
>
> A,B,C,D,E,F,X,Z
Sounds to me like you already have an algorithm in mind. It might not be
correct or complete, or it might not be efficient, but you performed some
steps in your own mind to generate that superset. Write down those steps
and you have an algorithm. Once you have the algorithm, you can then turn
it into Python code and start revising it from there.
I went back to first principles and drew a set diagram with multiple
disjoint sets of overlapping sets, and then thought about the process of
*adding* each new set to the set-of-supersets. It's hard to describe, but
if you start drawing an empty Venn diagram, then add one of your sets to
the Venn diagram at a time, merging that set to whatever superset it
belongs to, the process should appear quite obvious.
Let me walk through the process with your six sets above, plus one extra.
First, I create a list of supersets, initially empty:
supersets = []
I look at Set #1, and compare to all the supersets. There aren't any, so I
make a copy of Set #1 and add it to the supersets:
[{A,B,E,F}]
I look at Set #2, and compare it to all the supersets. It doesn't intersect
any of them, so I add it as a new superset:
[{A,B,E,F}, {G,H,L,P,Q}]
I look at Set #3. It intersects the first superset, but not the second, so I
merge them. Likewise for Set #4:
[{A,B,C,D,E,F}, {G,H,L,P,Q}]
[{A,B,C,D,E,F,X,Z}, {G,H,L,P,Q}]
Set #5 intersects the second superset, but not the first. Same for Set #6:
[{A,B,C,D,E,F,X,Z}, {G,H,L,M,P,Q,R}]
[{A,B,C,D,E,F,X,Z}, {G,H,L,M,O,P,Q,R,Y}]
Now let me add an extra Set #7, {A, R}, which intersects *both* supersets.
First I merge it with both:
[{A,B,C,D,E,F,R,X,Z}, {A,G,H,L,M,O,P,Q,R,Y}]
but I don't stop there. Then I recursively run through the supersets with
the same merging process to get:
[{A,B,C,D,E,F,G,H,L,M,O,P,Q,R,X,Y,Z}]
which is the single minimal, disjoint superset of all seven sets.
def merge(sets):
"""Merge a collection of sets into a list of minimal, disjoint
supersets.
"""
supersets = []
for a in sets:
count = 0
for s in supersets:
if a&s:
s.update(a)
count += 1
if count == 0:
supersets.append(a.copy())
elif count > 1:
supersets = merge(supersets)
return supersets
Testing this with your six sets gives:
py> sets = [set(s) for s in ("ABEF", "GHLPQ", "CDEF", "EXZ", "LMR", "OMY")]
py> for s in merge(sets):
... print(','.join(sorted(s)))
...
A,B,C,D,E,F,X,Z
G,H,L,M,O,P,Q,R,Y
Adding set #7 gives:
py> for s in merge(sets + [{'A', 'R'}]):
... print(','.join(sorted(s)))
...
A,B,C,D,E,F,G,H,L,M,O,P,Q,R,X,Y,Z
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
wrote:
> Marko Rauhamaa wrote:
>
>> Grant Edwards :
>>
>>> the polite thing to do is to delete references to it when you're done
>>> with it.
>>
>> I disagree with that recommendation. You should do the natural thing and
>> not care who holds references to who.
>
> I don't understand this. What is "the natural thing" if not to delete
> references to an object when you are done with it? Normally you just let
> things go out of scope, but if that won't happen, you have to take active
> steps, such as calling del or setting the reference to None.
I think the disagreement here is over the interpretation of "done with
it". If you drop all references to a connected socket object, Python
can rightly assume that you're done with the file and want to close
it; but what if you drop all references to a listening socket that's
been configured to call a function whenever someone connects?
def client_connected(sock):
sock.send("Hello!\r\n")
# whatever
listener = socket(23, on_accept=client_connected)
What should happen if that main socket isn't bound to a name? In my
opinion, the fact that it's configured for callback mode should mean
that it's kept alive. But it's also understandable to want to treat it
as "done", that it can be disposed of. It seems weird to me that you
should have to have a name somewhere that you'll never use, though.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Marko Rauhamaa wrote: > Grant Edwards : > >> the polite thing to do is to delete references to it when you're done >> with it. > > I disagree with that recommendation. You should do the natural thing and > not care who holds references to who. I don't understand this. What is "the natural thing" if not to delete references to an object when you are done with it? Normally you just let things go out of scope, but if that won't happen, you have to take active steps, such as calling del or setting the reference to None. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements
"TommyVee" wrote in message news:Bg5Gw.1344030$no4.494...@fx19.iad... Start off with sets of elements as follows: 1. A,B,E,F 2. G,H,L,P,Q 3. C,D,E,F 4. E,X,Z 5. L,M,R 6. O,M,Y Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they are "related" and form the following superset: A,B,C,D,E,F,X,Z Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have element 'M' in common, therefore they form the following superset: G,H,L,M,O,P,Q,R,Y I think you get the point. As long as sets have at least 1 common element, they combine to form a superset. Also "links" (common elements) between sets may go down multiple levels, as described in the second case above (2->5->6). Cycles thankfully, are not possible. BTW, the number of individual sets (and resultant supersets) will be very large. I don't know where to start with this. I thought about some type of recursive algorithm, but I'm not sure. I could figure out the Python implementation easy enough, I'm just stumped on the algorithm itself. Anybody have an idea? Thanks, Tom Thanks to all! Not only did you turn around my thinking on this, but I also learned about some new Python collection types. I think I have a way to get this to work now. -- https://mail.python.org/mailman/listinfo/python-list
Re: pypandoc and restructured text
Hi Chris, Chris Angelico wrote: [] >> Thanks a lot for the hint. Maybe I should seriously think about >> upgrading the whole distro. It's just that Gnome3 really sucks to my >> taste and I'm not in the mood to look for another Desktop >> Environment...(maybe I should go back to CDE). >> > > This is a bit off-topic for python-list, being more about Debian than > the language, but trying to install just one Wheezy package on Squeeze > is probably a bad idea; you'll probably find that it's built against a > different version of libc and such, so you'd need to upgrade pretty > much your whole system. I apologize for littering the group, thought somehow many users here might have had a similar issues with pypandoc. [] > 2) Upgrade the entire OS to either Wheezy or Jessie. Personally, I'm > quite happy with Jessie, even though she's not officially stable yet > (and have you _seen_ her? Definitely mentally unstable); but whether > you go Wheezy or Jessie, grab Xfce rather than Gnome 3. The latest > Jessie installers are giving an option on installation as to which > desktop environment(s) to install, but if you do it as an upgrade, you > may want to explicitly uninstall Gnome and install Xfce, either before > or after the upgrade. I finally upgraded! And I'm currently trying out xfce! Thanks again for the suggestions. Al p.s.: now pandoc works as expected. -- https://mail.python.org/mailman/listinfo/python-list
Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements
On 21/02/2015 19:46, TommyVee wrote:
Start off with sets of elements as follows:
1. A,B,E,F
2. G,H,L,P,Q
3. C,D,E,F
4. E,X,Z
5. L,M,R
6. O,M,Y
Note that sets 1, 3 and 4 all have the element 'E' in common, therefore
they are "related" and form the following superset:
A,B,C,D,E,F,X,Z
Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6
have element 'M' in common, therefore they form the following superset:
G,H,L,M,O,P,Q,R,Y
I think you get the point. As long as sets have at least 1 common
element, they combine to form a superset. Also "links" (common
elements) between sets may go down multiple levels, as described in the
second case above (2->5->6). Cycles thankfully, are not possible.
BTW, the number of individual sets (and resultant supersets) will be
very large.
I don't know where to start with this. I thought about some type of
recursive algorithm, but I'm not sure. I could figure out the Python
implementation easy enough, I'm just stumped on the algorithm itself.
Anybody have an idea?
Thanks, Tom
A naive approach but should give you something to think about.
from collections import defaultdict
sets = ({'A','B','E','F'},
{'G','H','L','P','Q'},
{'C','D','E','F'},
{'E','X','Z'},
{'L','M','R'},
{'O','M','Y'})
d = defaultdict(list)
for i, aSet in enumerate(sets):
for a in aSet:
d[a].append(i)
superSets = []
for k in d:
if len(d[k]) > 1:
superSet = set()
for i in d[k]:
superSet |= sets[i]
superSets.append(superSet)
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Grant Edwards : > the polite thing to do is to delete references to it when you're done > with it. I disagree with that recommendation. You should do the natural thing and not care who holds references to who. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On 2015-02-21, Cem Karan wrote: > > On Feb 21, 2015, at 12:42 AM, Chris Angelico wrote: > >> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: >>> In order to inform users that certain bits of state have changed, I require >>> them to register a callback with my code. The problem is that when I store >>> these callbacks, it naturally creates a strong reference to the objects, >>> which means that if they are deleted without unregistering themselves >>> first, my code will keep the callbacks alive. Since this could lead to >>> really weird and nasty situations, I would like to store all the callbacks >>> in a WeakSet >>> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That >>> way, my code isn't the reason why the objects are kept alive, and if they >>> are no longer alive, they are automatically removed from the WeakSet, >>> preventing me from accidentally calling them when they are dead. My >>> question is simple; is this a good design? If not, why not? Are there any >>> potential 'gotchas' I should be worried about? >>> >> >> No, it's not. I would advise using strong references - if the callback >> is a closure, for instance, you need to hang onto it, because there >> are unlikely to be any other references to it. If I register a >> callback with you, I expect it to be called; I expect, in fact, that >> that *will* keep my object alive. > > OK, so it would violate the principle of least surprise for you. And me as well. I would expect to be able to pass a closure as a callback and not have to keep a reference to it. Perhaps that just a leftover from working with other languages (javascript, scheme, etc.). It doesn't matter if it's a string, a float, a callback, a graphic or whatever: if I pass your function/library an object, I expect _you_ to keep track of it until you're done with it. > Interesting. Is this a general pattern in python? That is, > callbacks are owned by what they are registered with? I'm not sure what you mean by "owned" or why it matters that it's a callback: it's an object that was passed to you: you need to hold onto a reference to it until you're done with it, and the polite thing to do is to delete references to it when you're done with it. > So, what's the consensus on the list, strongly-held callbacks, or > weakly-held ones? -- Grant -- https://mail.python.org/mailman/listinfo/python-list
How to design a search engine in Python?
Dear Group, I am trying to build a search engine in Python. To do this, I have read tutorials and working methodologies from web and books like Stanford IR book [ http://www-nlp.stanford.edu/IR-book/]. I know how to design a crawler, I know PostgresSql, I am fluent with PageRank, TF-IDF, Zipf's law, etc. I came to know of Whoosh[https://pypi.python.org/pypi/Whoosh/] But I am looking for a total tutorial how to implement it. If any body may kindly direct me. I heard there are good source codes and prototypes, but I am not getting. Apology if this is not a question of the room. I tried to post as this is a room of Python bigwigs. Regards, Subhabrata. -- https://mail.python.org/mailman/listinfo/python-list
Re: subprocess command fails
On 02/20/2015 08:47 PM, Brad s wrote:
[...]
print("command = %r" % (cmdargv,))
sfmove = subprocess.call(cmdargv)
also, is the % to Python what precision is to C++?
No. It is like the % in C's printf().
-=- Larry -=-
--
https://mail.python.org/mailman/listinfo/python-list
Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements
On 02/21/2015 02:46 PM, TommyVee wrote: Start off with sets of elements as follows: 1. A,B,E,F 2. G,H,L,P,Q 3. C,D,E,F 4. E,X,Z 5. L,M,R 6. O,M,Y Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they are "related" and form the following superset: A,B,C,D,E,F,X,Z Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have element 'M' in common, therefore they form the following superset: G,H,L,M,O,P,Q,R,Y I think you get the point. As long as sets have at least 1 common element, they combine to form a superset. Also "links" (common elements) between sets may go down multiple levels, as described in the second case above (2->5->6). Cycles thankfully, are not possible. BTW, the number of individual sets (and resultant supersets) will be very large. I don't know where to start with this. I can't see where you've defined "this" yet. I think you're trying to find all combinations of sets that are "related", and that you're assuming some of them won't be. So perhaps you're trying to build "islands" of related sets, and you want each of those islands to be of maximal size. I thought about some type of recursive algorithm, but I'm not sure. I could figure out the Python implementation easy enough, I'm just stumped on the algorithm itself. Seems like you can have a doubly-nested loop, where your inner loop collects all the sets that are related to the one in the outer loop. You label each group with an island-number, where the island number is simply the lowest number set that's on the island. It's still not clear what you do with each of those islands, but the "superset" you refer to before is simply the OR of everything on the island. Perhaps you want to show all smaller supersets that are composed of at least two sets of an island. That becomes a fairly simple power-of-two iteration of those sets, where you throw out the case where the index is a power of two (ie. contains only one set). As you say, calculating the union or intersection of the various sets in python is trivial. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements
On 02/21/2015 11:46 AM, TommyVee wrote: > Start off with sets of elements as follows: > > 1. A,B,E,F > 2. G,H,L,P,Q > 3. C,D,E,F > 4. E,X,Z > 5. L,M,R > 6. O,M,Y > > Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they > are "related" and form the following superset: > > A,B,C,D,E,F,X,Z > > Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have > element 'M' in common, therefore they form > the following superset: > > G,H,L,M,O,P,Q,R,Y > > I think you get the point. As long as sets have at least 1 common element, > they combine to form a superset. Also > "links" (common elements) between sets may go down multiple levels, as > described in the second case above (2->5->6). > Cycles thankfully, are not possible. > > BTW, the number of individual sets (and resultant supersets) will be very > large. > > I don't know where to start with this. I thought about some type of > recursive algorithm, but I'm not sure. I could > figure out the Python implementation easy enough, I'm just stumped on the > algorithm itself. > > Anybody have an idea? Use a Counter (collections.Counter), add all sets, then keep any keys with a count of 2 or more. -- ~Ethan~ signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements
On Sat, Feb 21, 2015 at 2:46 PM, TommyVee wrote: > Start off with sets of elements as follows: > > 1. A,B,E,F > 2. G,H,L,P,Q > 3. C,D,E,F > 4. E,X,Z > 5. L,M,R > 6. O,M,Y > > Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they > are "related" and form the following superset: > > A,B,C,D,E,F,X,Z > > Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have > element 'M' in common, therefore they form the following superset: > > G,H,L,M,O,P,Q,R,Y > > I think you get the point. As long as sets have at least 1 common element, > they combine to form a superset. Also "links" (common elements) between > sets may go down multiple levels, as described in the second case above > (2->5->6). Cycles thankfully, are not possible. > > BTW, the number of individual sets (and resultant supersets) will be very > large. > > I don't know where to start with this. I thought about some type of > recursive algorithm, but I'm not sure. I could figure out the Python > implementation easy enough, I'm just stumped on the algorithm itself. > > Anybody have an idea? > start with reading about python sets. If you take the intersection of two sets it will return a set with common elements. If that is empty, they don't pass your test. If you take the union, you get a set with the set values in each. > Thanks, Tom > -- > https://mail.python.org/mailman/listinfo/python-list -- Joel Goldstick http://joelgoldstick.com -- https://mail.python.org/mailman/listinfo/python-list
Algorithm for Creating Supersets of Smaller Sets Based on Common Elements
Start off with sets of elements as follows: 1. A,B,E,F 2. G,H,L,P,Q 3. C,D,E,F 4. E,X,Z 5. L,M,R 6. O,M,Y Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they are "related" and form the following superset: A,B,C,D,E,F,X,Z Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have element 'M' in common, therefore they form the following superset: G,H,L,M,O,P,Q,R,Y I think you get the point. As long as sets have at least 1 common element, they combine to form a superset. Also "links" (common elements) between sets may go down multiple levels, as described in the second case above (2->5->6). Cycles thankfully, are not possible. BTW, the number of individual sets (and resultant supersets) will be very large. I don't know where to start with this. I thought about some type of recursive algorithm, but I'm not sure. I could figure out the Python implementation easy enough, I'm just stumped on the algorithm itself. Anybody have an idea? Thanks, Tom -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Lite' Databases (Re: sqlite3 and dates)
On 02/21/2015 01:22 AM, Ned Deily wrote: SQLite is one of the most widely-used, best-documented, best-tested, and well-respected software packages in the world. yes but is still sql. there are a couple of small scale not-sql databases that look interesting. problem with them is that the creator seem to subscribe to the build-and-toss-into-the-wild school of development. http://buzhug.sourceforge.net/ http://www.pydblite.net/en/index.html both are useful, both could use multi-writer support, and both need some love from the python world. --- eric -- https://mail.python.org/mailman/listinfo/python-list
Re: Accessible tools
On 2015-02-21 10:21, Bryan Duarte wrote: > those of us who rely on screen readers to interact with our > computers have a few things we do, and tend to not do. [snip] While my experience has shown most of your items to be true, I'd contend that >• Do not, have access to debugging tools. is mistaken or at least misinformed. For Python, I use the "pdb" module all the time, and it's command-line driven. Combined with a multi-terminal (whether multiple windows, virtual consoles, or a tmux/screen session), I can easily bounce back and forth between a "pdb" debugging session and the source code to make edits. Just to check, I fired up the "yasr" terminal screen-reader, launched tmux (using my quiet config, since it updates information on the screen like the time on a regular basis, making it chatty), and stepped through some Python code, checked variables, and walked up/down the call-stack. I know most other languages have similar functionality such as gdb for C code. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan wrote: > > On Feb 21, 2015, at 8:15 AM, Chris Angelico wrote: > >> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan wrote: >>> OK, so it would violate the principle of least surprise for you. >>> Interesting. Is this a general pattern in python? That is, callbacks >>> are owned by what they are registered with? >>> >>> In the end, I want to make a library that offers as few surprises to the >>> user as possible, and no matter how I think about callbacks, they are >>> surprising to me. If callbacks are strongly-held, then calling 'del >>> foo' on a callable object may not make it go away, which can lead to >>> weird and nasty situations. How? The whole point of callbacks is that you hand over responsibility to another piece of code, and then forget about your callback. The library will call it, when and if necessary, and when the library no longer needs your callback, it is free to throw it away. (If I wish the callback to survive beyond the lifetime of your library's use of it, I have to keep a reference to the function.) >>> Weakly-held callbacks mean that I (as the >>> programmer), know that objects will go away after the next garbage >>> collection (see Frank's earlier message), so I don't get 'dead' >>> callbacks coming back from the grave to haunt me. I'm afraid this makes no sense to me. Can you explain, or better still demonstrate, a scenario where "dead callbacks rise from the grave", so to speak? >>> So, what's the consensus on the list, strongly-held callbacks, or >>> weakly-held ones? >> >> I don't know about Python specifically, but it's certainly a general >> pattern in other languages. They most definitely are owned, and it's >> the only model that makes sense when you use closures (which won't >> have any other references anywhere). > > I agree about closures; its the only way they could work. *scratches head* There's nothing special about closures. You can assign them to a name like any other object. def make_closure(): x = 23 def closure(): return x + 1 return closure func = make_closure() Now you can register func as a callback, and de-register it when your done: register(func) unregister(func) Of course, if you thrown away your reference to func, you have no (easy) way of de-registering it. That's no different to any other object which is registered by identity. (Registering functions by name is a bad idea, since multiple functions can have the same name.) As an alternative, your callback registration function might return a ticket for the function: ticket = register(func) del func unregister(ticket) but that strikes me as over-kill. And of course, the simplest ticket is to return the function itself :-) > When I was > originally thinking about the library, I was trying to include all types > of callbacks, including closures and callable objects. The callable > objects may pass themselves, or one of their methods to the library, or > may do something really weird. I don't think they can do anything too weird. They have to pass a callable object. Your library just calls that object. You shouldn't need to care whether it is a function, a method, a type, a callable instance, or something else. You just call it, and when you're done calling it forever, you just throw it away. > Although I just realized that closures may cause another problem. In my > code, I expect that many different callbacks can be registered for the > same event. Unregistering means you request to be unregistered for the > event. How do you do that with a closure? Aren't they anonymous? Not unless you create them using lambda. Using the make_closure function above: py> func = make_closure() py> func.__name__ 'closure' Of course, if you call make_closure twice, both functions will have the same internal name. You can set the function __name__ and __qualname__ to fix that. This is how the functools.wraps decorator works. But that's a red herring. Don't register functions by name! Not all callable objects have names, and those that do, you may have multiple *distinct* callbacks with the same name. There are two reasonable approaches: unregister by identity, or by returning a ticket which uniquely identifies the callback. The user is responsible for keeping track of their own ticket. If I lose it, I can't unregister my callback any more. So sad, sucks to be me. The simplest possible identity-based scheme would be something like this: # don't hate me for using a global variable CALLBACKS = [] def register(func): if func not in CALLBACKS: CALLBACKS.append(func) def unregister(func): try: CALLBACKS.remove(func) except ValueError: pass That's probably a bit too simple, since it won't behave as expected with bound methods. The problem is that bound methods are generated on the fly, so this won't work: register(instance.spam) # later unregister(instance.spam) # a different instance! I would have to do this: boun
Re: Accessible tools
Tim, I am also on the blind linux list. I do not often post there as I predominately use a Mac and the Unix terminal but I am using Linux Kali on the side for some side tinkering and learning. I would use Linux a lot more if the screen reader was not so robotic... Would you be willing to be included in some accessibility Q&A or to bounce some ideas off of? > On Feb 19, 2015, at 11:43 AM, Tim Chase wrote: > > While not blind, I have an interest in accessibility and answer a > number of questions on the Blinux (Blind Linux Users) mailing list. > > On 2015-02-19 08:33, Bryan Duarte wrote: >> A professor and I have been throwing around the idea of developing >> a completely text based IDE. There are a lot of reasons this could >> be beneficial to a blind developer and maybe even some sighted >> developers who are comfortable in the terminal. The idea would be >> really just to provide a way of easily navigating blocks of code >> using some kind of tabular formatting, and being able to collapse >> blocks of code and hearing from a high level information about the >> code within. All tools and features would obviously be spoken or >> output in some kind of audio manor. > > It would seem that the traditional Unix-as-IDE[1] would serve you well > here. This is my method of choice, and it allows me to pick my > components and combine them. I usually use tmux, though GNU screen > would do too. Within that, I usually have the following: > > - vim to edit my code. Though swap in your favorite, whether > emacs/emacspeak, ed/edbrowse, joe, nano, or whatever. I know that > at least Vim and emacs support "folding" away blocks of code (what > you describe as "collapsing") which I usually prefix with a comment > that would give you a description of the block > > - a command-line (I use bash, some prefer zsh or tcsh or whatever) > for things like version-control, running my code, and file > management (move/copy/delete/rename/link/etc) > > - a Python command-line REPL that allows me to do quick tests on a > line of code as well as well as make extensive use of Python's > built-in dir() and help() commands which are invaluable. > > - when doing web-development (Django in my case), I'll often have the > dev-server running in one pane, and a console browser like > lynx/links/links2/elinks/w3m in another pane so that I can put my > code through its paces > > Another benefit of this is that I can run this on my development > machine, but then SSH into the machine from anywhere, reattach to the > tmux/screen session, and have the same configuration right as I left > it. > > The entire tmux/screen session can be run within an accessible > terminal window (I know that some are more accessible than others), > within a terminal screen-reader session (like yasr, screader, or > emacspeak), or even remoted into via an accessible SSH program on your > platform of choice. > > -tkc > > [1] > http://blog.sanctum.geek.nz/series/unix-as-ide/ > > > > > > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Accessible tools
Eric, Although I would most likely enjoy the former option I feel the latter would be most appropriate for contacting you. Thanks for getting back to me and explaining some of this. I will contact you off list for sure so as not to fill up the lists mailboxes with this topic. I will say that Jacob is absolutely correct in the fact that those of us who rely on screen readers to interact with our computers have a few things we do, and tend to not do. • Do type very efficiently • Do listen to our speech output very fast • Do interact with the text line, word, and character at a time. • Do not, use a mouse • Do not, enjoy reading through several lines of code at a time to locate a specific issue, method, or block of code. • Do not, have access to debugging tools. This is just a few things we do and do not do typically, but the point is that even though some things are similar there are some things that are quite different. Here is an example. I would not use a voice driven editor because I tend to build out a layout of code before I begin filling in methods, functions, and constructors. For me to do this with voice would require the use of a mouse to place the cursor where I want each line of code to be placed, and since I do not have access to the mouse in that way this would not work well. On the other hand if the program could interpret what you are speaking to build out method signatures and individual lines of code accurately, then there is a good chance that same system could provide a great interface for a screen reader. Thank you again for the response and be expecting an email from me shortly sir. > On Feb 20, 2015, at 11:39 AM, Jacob Kruger wrote: > > Eric, issue is that with screenreaders, we're generally way more into > navigating code and interface character by character/by keyboard, so , yes, > keeping interface relatively simple is a good thing, but, we also would > prefer to primarily keep all interface elements to make use of standard UI > controls, and make sure tab index/order is suitable/relevant at times, etc. > etc. > > As in, I think we'd primarily want to avoid having to use a mouse at all if > possible, but anyway. > > Stay well > > Jacob Kruger > Blind Biker > Skype: BlindZA > "Roger Wilco wants to welcome you...to the space janitor's closet..." > > - Original Message - From: "Eric S. Johansson" > To: > Sent: Friday, February 20, 2015 7:22 PM > Subject: Re: Accessible tools > > >> >> On 2/19/2015 10:33 AM, Bryan Duarte wrote: >>> Thank you jwi, and Jacob, >>> >>> I took a look at that posting and it seems pretty unique. I am not much >>> interested in the speech driven development, but I am very interested in >>> developing an accessible IDE. >> >> Well you should be because it looks like an aural interface (uses speech >> instead of keyboards) uses the same kinds of data to present to either a >> text to speech or speech recognition driven environment. >>> A professor and I have been throwing around the idea of developing a >>> completely text based IDE. There are a lot of reasons this could be >>> beneficial to a blind developer and maybe even some sighted developers who >>> are comfortable in the terminal. The idea would be really just to provide a >>> way of easily navigating blocks of code using some kind of tabular >>> formatting, and being able to collapse blocks of code and hearing from a >>> high level information about the code within. All tools and features would >>> obviously be spoken or output in some kind of audio manor. >> I've been working with another professor working on some of these issues as >> well. His focus has been mostly blind young adults in India. come up with >> some pretty cool concepts that looks very usable. The challenge now is to >> make them work and, quite frankly monetize the effort to pay for the >> development. >> >> Again, this shows the similarities in functionality used by both speech >> recognition and text-to-speech. All I care about is text and what I can say. >> We're now working with constructs such as with-open, argument by number, >> plaintext symbol names (with bidirectional transform to and from code form), >> guided construct generation for things like classes, methods, comprehensions >> etc. >> >> All of these things would be useful to handed programmers as well as a way >> of accelerating co-creation and editing. Unfortunately, like with disabled >> people stove piping text-to-speech versus speech recognition, handed >> developers stovepipe keyboard interfaces and don't really think about what >> they are trying to do, only how they are doing it. >> >> Yes yes, it's a broadbrush that you can probably slap me with. :-) >>> >>> Oh and before I forget does anyone know how to contact Eric who was >>> developing that accessible speech driven IDE? Thanks >> >> Well, you could try looking in a mirror and speaking my name three times at >> midnight But you would g
Re: Best practice: Sharing object between different objects
pfranke...@gmail.com writes: > ADC, DAC components. As I said, I would like to derive the > corresponding classes from one common class, let's say I2CDevice, so > that they can share the same bus connection (I don't want to do a > import smbus, ..., self.bus = smbus.SMBus(1) all the time). I don't really understand the question but it sounds like you might want a class variable: import smbus class I2CDevice bus = smbus.SMBus(1) def __init__ (self): ... This puts the "bus" object into the I2CDevice class itself. It is created when the I2CDevice class is defined. Methods wanting to access it can then say do_something_with(I2CDevice.bus) is that what you wanted? -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Steven D'Aprano : > Other than that, I cannot see how calling a function which has *not* > yet been garbage collected can fail, just because the only reference > still existing is a weak reference. Maybe the logic of the receiving object isn't prepared for the callback anymore after an intervening event. The problem then, of course, is in the logic and not in the callbacks. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: try pattern for database connection with the close method
Ian Kelly wrote:
> On Sat, Feb 21, 2015 at 8:27 AM, Peter Otten <__pete...@web.de> wrote:
>> Ian Kelly wrote:
>>
>>> On Sat, Feb 21, 2015 at 5:22 AM, Mark Lawrence
>>> wrote:
try:
with lite.connect('data.db') as db:
try:
db.execute(sql, parms)
except lite.IntegrityError:
raise ValueError('invalid data')
except lite.DatabaseError:
raise OSError('database file corrupt or not found.')
>>>
>>> This could result in the OSError being misleadingly raised due to some
>>> DatabaseError raised by the execute rather than the connect.
>>
>> The OP probably wants to catch these DatabaseErrors, too. Also, the
>> chance of a misleading traceback has been greatly reduced with the advent
>> of chained exceptions.
>>
>
> Yes, but the point is that OSError is probably inappropriate in that case.
Perhaps, but the example I gave in my other post:
>>> with open("data.db", "wb") as f:
... f.write(os.urandom(1024)) # put random bytes into data.db
>>> db = sqlite3.connect("data.db")
>>> db.execute("create table foo (bar, baz);")
Traceback (most recent call last):
File "", line 1, in
sqlite3.DatabaseError: file is encrypted or is not a database
matches the "database file corrupt" part of the error message provided by
the OP.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Frank Millman wrote: > I tried something similar a while ago, and I did find a gotcha. > > The problem lies in this phrase - "if they are no longer alive, they are > automatically removed from the WeakSet, preventing me from accidentally > calling them when they are dead." > > I found that the reference was not removed immediately, but was waiting to > be garbage collected. During that window, I could call the callback, which > resulted in an error. I don't understand how this could possibly work. (Or fail to work, as the case may be.) If the callback has been garbage collected, then you cannot call it, because you don't have any references to it and so cannot refer to it in any way. If the callback has *not* been garbage collected, then you can safely call it. You have a reference to the callback, therefore it exists. (If Python ever garbage collects an object that still has references to it, that would be a critical bug, and you would likely get some sort of seg fault). The only thing I can think of is you have a situation where your callback refers to another object, B, via a weak reference. Once all the regular strong references to the callback and B are gone, theoretically you could have a race condition where the callback is waiting to be garbage collected but B has already been garbage collected. If, in that window, you call the callback, *and* if the callback fails to correctly check that the weak reference to B still exists, then you could get a Python exception. The solution is simple: anytime you have a weak reference, you must always check it before you use it. Other than that, I cannot see how calling a function which has *not* yet been garbage collected can fail, just because the only reference still existing is a weak reference. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Chris Angelico : > On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: >> In order to inform users that certain bits of state have changed, I >> require them to register a callback with my code. The problem is that >> when I store these callbacks, it naturally creates a strong reference >> to the objects, which means that if they are deleted without >> unregistering themselves first, my code will keep the callbacks >> alive. Since this could lead to really weird and nasty situations, >> [...] > > No, it's not. I would advise using strong references - if the callback > is a closure, for instance, you need to hang onto it, because there > are unlikely to be any other references to it. If I register a > callback with you, I expect it to be called; I expect, in fact, that > that *will* keep my object alive. I use callbacks all the time but haven't had any problems with strong references. I am careful to move my objects to a zombie state after they're done so they can absorb any potential loose callbacks that are lingering in the system. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 2:45 AM, Cem Karan wrote: > OK, so if I'm reading your code correctly, you're breaking the cycle in your > object graph by making the GUI the owner of the callback, correct? No other > chunk of code has a reference to the callback, correct? Correct. The GUI engine ultimately owns everything. Of course, this is a very simple case (imagine a little notification popup; you don't care about it, you don't need to know when it's been closed, the only event on it is "hit Close to destroy the window"), and most usage would have other complications, but it's not uncommon for me to build a GUI program that leaves everything owned by the GUI engine. Everything is done through callbacks. Destroy a window, clean up its callbacks. The main window will have an "on-deletion" callback that terminates the program, perhaps. It's pretty straight-forward. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: try pattern for database connection with the close method
On Sat, Feb 21, 2015 at 8:27 AM, Peter Otten <__pete...@web.de> wrote:
> Ian Kelly wrote:
>
>> On Sat, Feb 21, 2015 at 5:22 AM, Mark Lawrence
>> wrote:
>>> try:
>>> with lite.connect('data.db') as db:
>>> try:
>>> db.execute(sql, parms)
>>> except lite.IntegrityError:
>>> raise ValueError('invalid data')
>>> except lite.DatabaseError:
>>> raise OSError('database file corrupt or not found.')
>>
>> This could result in the OSError being misleadingly raised due to some
>> DatabaseError raised by the execute rather than the connect.
>
> The OP probably wants to catch these DatabaseErrors, too. Also, the chance
> of a misleading traceback has been greatly reduced with the advent of
> chained exceptions.
>
Yes, but the point is that OSError is probably inappropriate in that case.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 9:36 AM, Chris Angelico wrote:
> On Sun, Feb 22, 2015 at 1:07 AM, Cem Karan wrote:
>> I agree about closures; its the only way they could work. When I was
>> originally thinking about the library, I was trying to include all types of
>> callbacks, including closures and callable objects. The callable objects
>> may pass themselves, or one of their methods to the library, or may do
>> something really weird.
>>
>> Although I just realized that closures may cause another problem. In my
>> code, I expect that many different callbacks can be registered for the same
>> event. Unregistering means you request to be unregistered for the event.
>> How do you do that with a closure? Aren't they anonymous?
>>
>
> They're objects, same as any other, so the caller can hang onto a
> reference and then say "now remove this one". Simple example:
>
> callbacks = []
> def register_callback(f): callbacks.append(f)
> def unregister_callback(f): callbacks.remove(f)
> def do_callbacks():
>for f in callbacks:
>f()
>
> def make_callback(i):
>def inner():
>print("Callback! %d"%i)
>register_callback(inner)
>return inner
>
> make_callback(5)
> remove_me = make_callback(6)
> make_callback(7)
> unregister_callback(remove_me)
> do_callbacks()
Yeah, that's pretty much what I thought you'd have to do, which kind of defeats
the purpose of closures (fire-and-forget things). BUT it does answer my
question, so no complaints about it!
So, either you keep a reference to your own closure, which means that the
library doesn't really need to, or the library keeps hold of it for you, in
which case you don't have a reasonable way of removing it.
> The other option is for your callback registration to return some kind
> of identifier, which can later be used to unregister the callback.
> This is a good way of avoiding reference cycles (the ID could be a
> simple integer - maybe the length of the list prior to the new
> callback being appended, and then the unregistration process is simply
> "callbacks[id] = None", and you skip the Nones when iterating), and
> even allows you to register the exact same function more than once,
> for what that's worth.
That would work. In the cases where someone might register & unregister many
callbacks, you might use UUIDs as keys instead (avoids the ABA problem).
> When I do GUI programming, this is usually how things work. For
> instance, I use GTK2 (though usually with Pike rather than Python),
> and I can connect a signal to a callback function. Any given signal
> could have multiple callbacks attached to it, so it's similar to your
> case. I frequently depend on the GTK engine retaining a reference to
> my function (and thus to any data it requires), as I tend not to hang
> onto any inner objects that don't need retention. Once the parent
> object is destroyed, all its callbacks get dereferenced. Consider this
> simplified form:
>
> def popup_window():
>w = Window()
># Add layout, info, whatever it takes
>btn = Button("Close")
>w.add(btn) # actually it'd be added to a layout
>btn.signal_connect("clicked", lambda *args: w.destroy())
>
> The GUI back end will hang onto a reference to the window, because
> it's currently on screen; to the button, because it's attached to the
> window; and to my function, because it's connected to a button signal.
> Then when you click the button, the window gets destroyed, which
> destroys the button, which unregisters all its callbacks. At that
> point, there are no refs to the function, so it can get disposed of.
> That button function was the last external reference to the window,
> and now that it's not on screen, its Python object can also be
> disposed of, as can the button inside. So it'll all clean up fairly
> nicely; as long as the callback gets explicitly deregistered, that's
> the end of everything.
OK, so if I'm reading your code correctly, you're breaking the cycle in your
object graph by making the GUI the owner of the callback, correct? No other
chunk of code has a reference to the callback, correct?
Thanks,
Cem Karan
--
https://mail.python.org/mailman/listinfo/python-list
Re: try pattern for database connection with the close method
Ian Kelly wrote:
> On Sat, Feb 21, 2015 at 5:22 AM, Mark Lawrence
> wrote:
>> On 21/02/2015 02:42, Mario Figueiredo wrote:
>>>
>>> Hello all,
>>>
>>> I'm using the following pattern for db access that requires me to
>>> close the connection as soon as it is not needed:
>>>
>>> import sqlite3 as lite
>>>
>>> try:
>>> db = lite.connect('data.db')
>>> except lite.DatabaseError:
>>> raise OSError('database file corrupt or not found.')
>>> else:
>>> try:
>>> with db:
>>> db.execute(sql, parms)
>>> except lite.IntegrityError:
>>> raise ValueError('invalid data')
>>> finally:
>>> db.close()
>>>
>>> Since it's a bit verbose, is there a better way?
>>>
>>> Note: The user of this API has the whole database functionality
>>> abstracted away. Hence the exception channeling in the except clauses.
>>>
>>
>> Use your context manager at the outer level.
>>
>> import sqlite3 as lite
>>
>> try:
>> with lite.connect('data.db') as db:
>> try:
>> db.execute(sql, parms)
>> except lite.IntegrityError:
>> raise ValueError('invalid data')
>> except lite.DatabaseError:
>> raise OSError('database file corrupt or not found.')
>
> This could result in the OSError being misleadingly raised due to some
> DatabaseError raised by the execute rather than the connect.
The OP probably wants to catch these DatabaseErrors, too. Also, the chance
of a misleading traceback has been greatly reduced with the advent of
chained exceptions.
--
https://mail.python.org/mailman/listinfo/python-list
Re: try pattern for database connection with the close method
Mario Figueiredo wrote:
> Hello all,
>
> I'm using the following pattern for db access that requires me to
> close the connection as soon as it is not needed:
>
> import sqlite3 as lite
>
> try:
> db = lite.connect('data.db')
> except lite.DatabaseError:
> raise OSError('database file corrupt or not found.')
> else:
> try:
> with db:
> db.execute(sql, parms)
> except lite.IntegrityError:
> raise ValueError('invalid data')
> finally:
> db.close()
>
> Since it's a bit verbose,
Why would you care about a few lines? You don't repeat them, do you? Put the
code into a function or a context manager and invoke it with
>>> my_execute(sql, parms)
or
>>> with my_db() as db:
... db.execute(sql, parms)
> is there a better way?
>
> Note: The user of this API has the whole database functionality
> abstracted away. Hence the exception channeling in the except clauses.
db.execute() may trigger other sqlite-related errors including
DatabaseError:
>>> import sqlite3
>>> db = sqlite3.connect("/dev/full")
>>> db.execute("create table foo (bar, baz);")
Traceback (most recent call last):
File "", line 1, in
sqlite3.OperationalError: unable to open database file
>>> import os
>>> with open("data.db", "wb") as f:
... f.write(os.urandom(1024)) # put random bytes into data.db
... # chances of creating a valid db
... # left as an exercise ;)
...
1024
>>> db = sqlite3.connect("data.db")
>>> db.execute("create table foo (bar, baz);")
Traceback (most recent call last):
File "", line 1, in
sqlite3.DatabaseError: file is encrypted or is not a database
If you want to catch these, too:
@contextlib.contextmanager
def my_db():
db = None
try:
db = sqlite3.connect("data.db")
with db:
yield db # db.execute() if you don't buy
# into the contextmanager idea
except sqlite3.IntegrityError:
raise ValueError
except sqlite3.DatabaseError:
raise OSError
except sqlite3.Error:
raise WhateverYouNeed
finally:
if db is not None:
db.close()
with my_db() as db:
db.execute("select * from sqlite_master;")
--
https://mail.python.org/mailman/listinfo/python-list
Re: try pattern for database connection with the close method
On Sat, Feb 21, 2015 at 5:22 AM, Mark Lawrence wrote:
> On 21/02/2015 02:42, Mario Figueiredo wrote:
>>
>> Hello all,
>>
>> I'm using the following pattern for db access that requires me to
>> close the connection as soon as it is not needed:
>>
>> import sqlite3 as lite
>>
>> try:
>> db = lite.connect('data.db')
>> except lite.DatabaseError:
>> raise OSError('database file corrupt or not found.')
>> else:
>> try:
>> with db:
>> db.execute(sql, parms)
>> except lite.IntegrityError:
>> raise ValueError('invalid data')
>> finally:
>> db.close()
>>
>> Since it's a bit verbose, is there a better way?
>>
>> Note: The user of this API has the whole database functionality
>> abstracted away. Hence the exception channeling in the except clauses.
>>
>
> Use your context manager at the outer level.
>
> import sqlite3 as lite
>
> try:
> with lite.connect('data.db') as db:
> try:
> db.execute(sql, parms)
> except lite.IntegrityError:
> raise ValueError('invalid data')
> except lite.DatabaseError:
> raise OSError('database file corrupt or not found.')
This could result in the OSError being misleadingly raised due to some
DatabaseError raised by the execute rather than the connect.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python path on windows
On Sun, Feb 22, 2015 at 1:10 AM, Dave Angel wrote: > But you could always write your own py.exe, which interprets the shebang > differently. > > Or run python.exe yourself, which as far as I know, pays no attention to > shebang lines. Or, my preferred solution: Fix the shebangs :) It's usually not hard to put together a script that zips through your hard drive and fixed them... or else you just fix 'em when you notice 'em, which means your shebangs become a kind of historic marker. I used to have drive letters all over my code, and I could track a program's history through them: this used to be on D:, then it moved to E:, etc. Discovering something that still said "D:\eddie" was like digging up a Second World War bullet casing. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 1:07 AM, Cem Karan wrote:
> I agree about closures; its the only way they could work. When I was
> originally thinking about the library, I was trying to include all types of
> callbacks, including closures and callable objects. The callable objects may
> pass themselves, or one of their methods to the library, or may do something
> really weird.
>
> Although I just realized that closures may cause another problem. In my
> code, I expect that many different callbacks can be registered for the same
> event. Unregistering means you request to be unregistered for the event. How
> do you do that with a closure? Aren't they anonymous?
>
They're objects, same as any other, so the caller can hang onto a
reference and then say "now remove this one". Simple example:
callbacks = []
def register_callback(f): callbacks.append(f)
def unregister_callback(f): callbacks.remove(f)
def do_callbacks():
for f in callbacks:
f()
def make_callback(i):
def inner():
print("Callback! %d"%i)
register_callback(inner)
return inner
make_callback(5)
remove_me = make_callback(6)
make_callback(7)
unregister_callback(remove_me)
do_callbacks()
The other option is for your callback registration to return some kind
of identifier, which can later be used to unregister the callback.
This is a good way of avoiding reference cycles (the ID could be a
simple integer - maybe the length of the list prior to the new
callback being appended, and then the unregistration process is simply
"callbacks[id] = None", and you skip the Nones when iterating), and
even allows you to register the exact same function more than once,
for what that's worth.
When I do GUI programming, this is usually how things work. For
instance, I use GTK2 (though usually with Pike rather than Python),
and I can connect a signal to a callback function. Any given signal
could have multiple callbacks attached to it, so it's similar to your
case. I frequently depend on the GTK engine retaining a reference to
my function (and thus to any data it requires), as I tend not to hang
onto any inner objects that don't need retention. Once the parent
object is destroyed, all its callbacks get dereferenced. Consider this
simplified form:
def popup_window():
w = Window()
# Add layout, info, whatever it takes
btn = Button("Close")
w.add(btn) # actually it'd be added to a layout
btn.signal_connect("clicked", lambda *args: w.destroy())
The GUI back end will hang onto a reference to the window, because
it's currently on screen; to the button, because it's attached to the
window; and to my function, because it's connected to a button signal.
Then when you click the button, the window gets destroyed, which
destroys the button, which unregisters all its callbacks. At that
point, there are no refs to the function, so it can get disposed of.
That button function was the last external reference to the window,
and now that it's not on screen, its Python object can also be
disposed of, as can the button inside. So it'll all clean up fairly
nicely; as long as the callback gets explicitly deregistered, that's
the end of everything.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best practice: Sharing object between different objects
On 02/21/2015 07:15 AM, pfranke...@gmail.com wrote: Hello! I have a best-practice question: Imagine I have several hardware devices that I work with on the same I2C bus and I am using the python smbus module for that purpose. The individual devices are sensors, ADC, DAC components. As I said, I would like to derive the corresponding classes from one common class, let's say I2CDevice, so that they can share the same bus connection (I don't want to do a import smbus, ..., self.bus = smbus.SMBus(1) all the time). Of course, I could just pass the the bus reference to each instance, but I am pretty sure that there must be a nicer way to do this. In particular, I imagine the following: It should be possible that I have two classes, ADC_I2C, DAC_I2C which share the same base class. Once I create an instance of ADC_I2C or DAC_I2C it should check whether a bus object exists, if not, it should create one and the other class should be able to use this bus reference as well. Do you get my point? I am pretty sure that such a design pattern should exist, maybe also in the reference of DB connections? Unfortunately I did not find something like this. I think you should write the code (or at least a skeleton version). I suspect the answer will become obvious to you, once you're not worrying about design patterns, or being java compatible. Regardless of whether you inherit from a common base class, both classes can have an attribute with a "bus object" in it. The question becomes who initializes it, and who decides to reuse the "same one". Your wording doesn't make that the least bit clear to me; maybe your sample code will. Many times when you'd have a common base class in Java or C++, you just have to have a common member or two in Python. On the other hand, once you start implementing, maybe it'll become obvious that there's enough shared code for a common base class. The first question is whether an ADC "is-a" bus, or "has-a" bus. Stated that way it sounds like you should have an attribute in ADC which is a bus object. Any hints are highly appreciated! Thanks! -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Fri, Feb 20, 2015 at 9:42 PM, Chris Angelico wrote: > No, it's not. I would advise using strong references - if the callback > is a closure, for instance, you need to hang onto it, because there > are unlikely to be any other references to it. If I register a > callback with you, I expect it to be called; I expect, in fact, that > that *will* keep my object alive. For that matter, if the callback is a method, you need to hang onto it, because method wrappers are generated on demand, so the method would be removed from the valid callbacks instantly. Weak references for callbacks are broken. -- Devin -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 8:37 AM, Mark Lawrence wrote: > On 21/02/2015 05:41, Frank Millman wrote: >> >> "Cem Karan" wrote in message >> news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com... >>> Hi all, I'm working on a project that will involve the use of callbacks, >>> and I want to bounce an idea I had off of everyone to make sure I'm not >>> developing a bad idea. Note that this is for python 3.4 code; I don't >>> need to worry about any version of python earlier than that. >>> >>> In order to inform users that certain bits of state have changed, I >>> require them to register a callback with my code. The problem is that >>> when I store these callbacks, it naturally creates a strong reference to >>> the objects, which means that if they are deleted without unregistering >>> themselves first, my code will keep the callbacks alive. Since this could >>> lead to really weird and nasty situations, I would like to store all the >>> callbacks in a WeakSet >>> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That >>> way, my code isn't the reason why the objects are kept alive, and if they >>> are no longer alive, they are automatically removed from the WeakSet, >>> preventing me from accidentally calling them when they are dead. My >>> question is simple; is this a good design? If not, why not? >>> Are there any potential 'gotchas' I should be worried about? >>> >> >> I tried something similar a while ago, and I did find a gotcha. >> >> The problem lies in this phrase - "if they are no longer alive, they are >> automatically removed from the WeakSet, preventing me from accidentally >> calling them when they are dead." >> >> I found that the reference was not removed immediately, but was waiting to >> be garbage collected. During that window, I could call the callback, which >> resulted in an error. >> >> There may have been a simple workaround. Perhaps someone else can comment. >> >> Frank Millman >> > > https://docs.python.org/3/library/gc.html has a collect function. That seems > like a simple workaround, but whether or not it classifies as a good solution > I'll leave to others, I'm not qualified to say. Unfortunately, depending on how many objects you have in your object graph, it can slow your code down a fair amount. I think Frank is right about how a WeakSet might be a bad idea in this case. You really need to know if an object is alive or dead, and not some indeterminate state. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Python path on windows
On 02/21/2015 06:05 AM, Gisle Vanem wrote: Dave Angel wrote: Finally, when py.exe starts, it reads that first (shebang) line, and decides which python interpreter to actually use. py.exe? Do you mean python.exe? Reread my post, or read Mark's reply to yours. The job of py.exe or pyw.exe is to examine the shebang line and then exec the appropriate python.exe Is there a way to make python.exe ignore all Shebang lines in all scripts? I had many generated .py-files with: #!g:\ProgramFiler\Python27\python.EXE That's what symlinks are for, on Unix-like systems. Somewhere I read that Windows has had them for a little while now. Those generated files should have pointed to a symlink on your system drive, rather than pointing directly to drive G: But you could always write your own py.exe, which interprets the shebang differently. Or run python.exe yourself, which as far as I know, pays no attention to shebang lines. After transferring my Python tree to a new Win-8.1 PC, my Python tree was installed under "f:\ProgramFiler\" and my CD-ROM on "g:\" This caused those scripts to access my CD-ROM or pop-up a Win-GUI error dialogue if no CD-ROM was in the drive. Irritating. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 8:15 AM, Chris Angelico wrote: > On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan wrote: >> OK, so it would violate the principle of least surprise for you. >> Interesting. Is this a general pattern in python? That is, callbacks are >> owned by what they are registered with? >> >> In the end, I want to make a library that offers as few surprises to the >> user as possible, and no matter how I think about callbacks, they are >> surprising to me. If callbacks are strongly-held, then calling 'del foo' on >> a callable object may not make it go away, which can lead to weird and nasty >> situations. Weakly-held callbacks mean that I (as the programmer), know >> that objects will go away after the next garbage collection (see Frank's >> earlier message), so I don't get 'dead' callbacks coming back from the grave >> to haunt me. >> >> So, what's the consensus on the list, strongly-held callbacks, or >> weakly-held ones? > > I don't know about Python specifically, but it's certainly a general > pattern in other languages. They most definitely are owned, and it's > the only model that makes sense when you use closures (which won't > have any other references anywhere). I agree about closures; its the only way they could work. When I was originally thinking about the library, I was trying to include all types of callbacks, including closures and callable objects. The callable objects may pass themselves, or one of their methods to the library, or may do something really weird. Although I just realized that closures may cause another problem. In my code, I expect that many different callbacks can be registered for the same event. Unregistering means you request to be unregistered for the event. How do you do that with a closure? Aren't they anonymous? > If you're expecting 'del foo' to destroy the object, then you have a > bigger problem than callbacks, because that's simply not how Python > works. You can't _ever_ assume that deleting something from your local > namespace will destroy the object, because there can always be more > references. So maybe you need a more clear way of saying "I'm done > with this, get rid of it". Agreed about 'del', and I don't assume that the object goes away at the point. The problem is debugging and determining WHY your object is still around. I know a combination of logging and gc.get_referrers() will probably help you figure out why something is still around, but I'm trying to avoid that headache. I guess the real problem is how this creates cycles in the call graph. User code effectively owns the library code, which via callbacks owns the user code. I have no idea what the best point the cycle is to break it, and not surprise someone down the road. The only idea I have is to redesign the library a little, and make anything that accepts a callback actually be a subclass of collections.abc.Container, or even collections.abc.MutableSet. That makes it very obvious that the object owns the callback, and that you will need to remove your object to unregister it. The only problem is how to handle closures; since they are anonymous, how do you decide which one to remove? Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: pypandoc and restructured text
On Sun, Feb 22, 2015 at 12:39 AM, alb wrote: > Hi Wolfgang, > > Wolfgang Maier wrote: > [] >> I have pandoc 1.12.2.1 and it recognizes the figure directive just fine >> (tested with html output so I cannot say anything about LaTeX). > > This reminds me that I need to move sooner or later from squeeze to > wheezy... > > Do you know of anyway to install wheezy packages on squeeze? On > backports there's nothing more than what comes with the squeeze sources. > > Thanks a lot for the hint. Maybe I should seriously think about > upgrading the whole distro. It's just that Gnome3 really sucks to my > taste and I'm not in the mood to look for another Desktop > Environment...(maybe I should go back to CDE). > This is a bit off-topic for python-list, being more about Debian than the language, but trying to install just one Wheezy package on Squeeze is probably a bad idea; you'll probably find that it's built against a different version of libc and such, so you'd need to upgrade pretty much your whole system. Your two best options are, IMO: 1) Install pandoc from source, which will then build it against your existing libraries. This may not work (if it depends on a newer version of something), but it should tell you if that's the case. You won't disturb your system in any way. 2) Upgrade the entire OS to either Wheezy or Jessie. Personally, I'm quite happy with Jessie, even though she's not officially stable yet (and have you _seen_ her? Definitely mentally unstable); but whether you go Wheezy or Jessie, grab Xfce rather than Gnome 3. The latest Jessie installers are giving an option on installation as to which desktop environment(s) to install, but if you do it as an upgrade, you may want to explicitly uninstall Gnome and install Xfce, either before or after the upgrade. Happily using a mix of Debian versions here, but all with Xfce. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: pypandoc and restructured text
On 21.02.2015 14:39, alb wrote: Do you know of anyway to install wheezy packages on squeeze? No need for a new distro. Use virtualenv, this is simple and great: http://simononsoftware.com/virtualenv-tutorial-part-2/ Cheers, Fabien -- https://mail.python.org/mailman/listinfo/python-list
Re: pypandoc and restructured text
Hi Wolfgang, Wolfgang Maier wrote: [] > I have pandoc 1.12.2.1 and it recognizes the figure directive just fine > (tested with html output so I cannot say anything about LaTeX). This reminds me that I need to move sooner or later from squeeze to wheezy... Do you know of anyway to install wheezy packages on squeeze? On backports there's nothing more than what comes with the squeeze sources. > I've found an old forum post from 2011: > > https://groups.google.com/forum/#!topic/pandoc-discuss/OmGYDycaMjs > > confirming that the figure directive was not supported at that time. > So, yes, I think upgrading pandoc could help. Thanks a lot for the hint. Maybe I should seriously think about upgrading the whole distro. It's just that Gnome3 really sucks to my taste and I'm not in the mood to look for another Desktop Environment...(maybe I should go back to CDE). Al -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On 21/02/2015 05:41, Frank Millman wrote: "Cem Karan" wrote in message news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com... Hi all, I'm working on a project that will involve the use of callbacks, and I want to bounce an idea I had off of everyone to make sure I'm not developing a bad idea. Note that this is for python 3.4 code; I don't need to worry about any version of python earlier than that. In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about? I tried something similar a while ago, and I did find a gotcha. The problem lies in this phrase - "if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead." I found that the reference was not removed immediately, but was waiting to be garbage collected. During that window, I could call the callback, which resulted in an error. There may have been a simple workaround. Perhaps someone else can comment. Frank Millman https://docs.python.org/3/library/gc.html has a collect function. That seems like a simple workaround, but whether or not it classifies as a good solution I'll leave to others, I'm not qualified to say. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 12:41 AM, Frank Millman wrote: > > "Cem Karan" wrote in message > news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com... >> Hi all, I'm working on a project that will involve the use of callbacks, >> and I want to bounce an idea I had off of everyone to make sure I'm not >> developing a bad idea. Note that this is for python 3.4 code; I don't >> need to worry about any version of python earlier than that. >> >> In order to inform users that certain bits of state have changed, I >> require them to register a callback with my code. The problem is that >> when I store these callbacks, it naturally creates a strong reference to >> the objects, which means that if they are deleted without unregistering >> themselves first, my code will keep the callbacks alive. Since this could >> lead to really weird and nasty situations, I would like to store all the >> callbacks in a WeakSet >> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That >> way, my code isn't the reason why the objects are kept alive, and if they >> are no longer alive, they are automatically removed from the WeakSet, >> preventing me from accidentally calling them when they are dead. My >> question is simple; is this a good design? If not, why not? >> Are there any potential 'gotchas' I should be worried about? >> > > I tried something similar a while ago, and I did find a gotcha. > > The problem lies in this phrase - "if they are no longer alive, they are > automatically removed from the WeakSet, preventing me from accidentally > calling them when they are dead." > > I found that the reference was not removed immediately, but was waiting to > be garbage collected. During that window, I could call the callback, which > resulted in an error. > > There may have been a simple workaround. Perhaps someone else can comment. THAT would be one heck of a gotcha! Must have been fun debugging that one! Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan wrote: > OK, so it would violate the principle of least surprise for you. > Interesting. Is this a general pattern in python? That is, callbacks are > owned by what they are registered with? > > In the end, I want to make a library that offers as few surprises to the user > as possible, and no matter how I think about callbacks, they are surprising > to me. If callbacks are strongly-held, then calling 'del foo' on a callable > object may not make it go away, which can lead to weird and nasty situations. > Weakly-held callbacks mean that I (as the programmer), know that objects > will go away after the next garbage collection (see Frank's earlier message), > so I don't get 'dead' callbacks coming back from the grave to haunt me. > > So, what's the consensus on the list, strongly-held callbacks, or weakly-held > ones? I don't know about Python specifically, but it's certainly a general pattern in other languages. They most definitely are owned, and it's the only model that makes sense when you use closures (which won't have any other references anywhere). If you're expecting 'del foo' to destroy the object, then you have a bigger problem than callbacks, because that's simply not how Python works. You can't _ever_ assume that deleting something from your local namespace will destroy the object, because there can always be more references. So maybe you need a more clear way of saying "I'm done with this, get rid of it". ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 12:42 AM, Chris Angelico wrote: > On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: >> In order to inform users that certain bits of state have changed, I require >> them to register a callback with my code. The problem is that when I store >> these callbacks, it naturally creates a strong reference to the objects, >> which means that if they are deleted without unregistering themselves first, >> my code will keep the callbacks alive. Since this could lead to really >> weird and nasty situations, I would like to store all the callbacks in a >> WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). >> That way, my code isn't the reason why the objects are kept alive, and if >> they are no longer alive, they are automatically removed from the WeakSet, >> preventing me from accidentally calling them when they are dead. My >> question is simple; is this a good design? If not, why not? Are there any >> potential 'gotchas' I should be worried about? >> > > No, it's not. I would advise using strong references - if the callback > is a closure, for instance, you need to hang onto it, because there > are unlikely to be any other references to it. If I register a > callback with you, I expect it to be called; I expect, in fact, that > that *will* keep my object alive. OK, so it would violate the principle of least surprise for you. Interesting. Is this a general pattern in python? That is, callbacks are owned by what they are registered with? In the end, I want to make a library that offers as few surprises to the user as possible, and no matter how I think about callbacks, they are surprising to me. If callbacks are strongly-held, then calling 'del foo' on a callable object may not make it go away, which can lead to weird and nasty situations. Weakly-held callbacks mean that I (as the programmer), know that objects will go away after the next garbage collection (see Frank's earlier message), so I don't get 'dead' callbacks coming back from the grave to haunt me. So, what's the consensus on the list, strongly-held callbacks, or weakly-held ones? Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Python path on windows
On 21/02/2015 11:05, Gisle Vanem wrote: Dave Angel wrote: Finally, when py.exe starts, it reads that first (shebang) line, and decides which python interpreter to actually use. py.exe? Do you mean python.exe? py.exe or pyw.exe come with the Python launcher on Windows and work out which version of the interpreter to use. https://www.python.org/dev/peps/pep-0397/ https://docs.python.org/3/using/windows.html#python-launcher-for-windows I think it was first installed with Python 3.3. It's available here https://bitbucket.org/vinay.sajip/pylauncher if you're on an older version. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Best practice: Sharing object between different objects
Hello! I have a best-practice question: Imagine I have several hardware devices that I work with on the same I2C bus and I am using the python smbus module for that purpose. The individual devices are sensors, ADC, DAC components. As I said, I would like to derive the corresponding classes from one common class, let's say I2CDevice, so that they can share the same bus connection (I don't want to do a import smbus, ..., self.bus = smbus.SMBus(1) all the time). Of course, I could just pass the the bus reference to each instance, but I am pretty sure that there must be a nicer way to do this. In particular, I imagine the following: It should be possible that I have two classes, ADC_I2C, DAC_I2C which share the same base class. Once I create an instance of ADC_I2C or DAC_I2C it should check whether a bus object exists, if not, it should create one and the other class should be able to use this bus reference as well. Do you get my point? I am pretty sure that such a design pattern should exist, maybe also in the reference of DB connections? Unfortunately I did not find something like this. Any hints are highly appreciated! Thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: try pattern for database connection with the close method
On 21/02/2015 02:42, Mario Figueiredo wrote:
Hello all,
I'm using the following pattern for db access that requires me to
close the connection as soon as it is not needed:
import sqlite3 as lite
try:
db = lite.connect('data.db')
except lite.DatabaseError:
raise OSError('database file corrupt or not found.')
else:
try:
with db:
db.execute(sql, parms)
except lite.IntegrityError:
raise ValueError('invalid data')
finally:
db.close()
Since it's a bit verbose, is there a better way?
Note: The user of this API has the whole database functionality
abstracted away. Hence the exception channeling in the except clauses.
Use your context manager at the outer level.
import sqlite3 as lite
try:
with lite.connect('data.db') as db:
try:
db.execute(sql, parms)
except lite.IntegrityError:
raise ValueError('invalid data')
except lite.DatabaseError:
raise OSError('database file corrupt or not found.')
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python path on windows
Dave Angel wrote: Finally, when py.exe starts, it reads that first (shebang) line, and decides which python interpreter to actually use. py.exe? Do you mean python.exe? Is there a way to make python.exe ignore all Shebang lines in all scripts? I had many generated .py-files with: #!g:\ProgramFiler\Python27\python.EXE After transferring my Python tree to a new Win-8.1 PC, my Python tree was installed under "f:\ProgramFiler\" and my CD-ROM on "g:\" This caused those scripts to access my CD-ROM or pop-up a Win-GUI error dialogue if no CD-ROM was in the drive. Irritating. -- --gv -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Lite' Databases (Re: sqlite3 and dates)
Am 20.02.2015 um 15:16 schrieb Dennis Lee Bieber: The middle-ground is probably something like the embedded version of Firebird (pity there has been no updated book -- "The Firebird Book" came out in 2004, v1.5 while 2.5 is current [Whoops, looks like there /is/ an update, print-on-demand in three overpriced volumes]) The second edition as PDF EBook is part of the IBPhoenix DVD (Developer Edition $100). But I can't find out from the website what that DVD contains in its current edition. It used to have Firebird itself and quite a lot of additional software and documentation. Firebird embedded has no network access and multi-user capabilities at all if I read the documentation correctly (http://www.firebirdsql.org/manual/ufb-cs-embedded.html). But "A Firebird embedded server DLL can also be used as a network client" to a regular Firebird server (from the same page). On the other hand triggers and stored procedures should work exactly like the client/server versions (and the procedure language is pretty similar to PostgreSQL, so migration between these two isn't too difficult). So this is a middle ground with other pros and cons than SQLite versus PostgreSQL. -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Lite' Databases (Re: sqlite3 and dates)
Ned Deily writes: >> Same reason lots of people have forked Postgres. Or you might just want >> to customize it. > Well, for whatever reason one might have, one can: it's public domain > software. Yes, but unlike with most FOSS software, your version has much lower quality assurance than the "official" version because you don't have the big test suite. Even if you don't fork or change the code at all, but you just port it to a new platform or compiler, you really should run the full set of tests, but you can't. > It seems like [the SQLite consortium] was an approach Richard Hipp > and major users of SQLite took to ensure a sustaining funding model > for the project while ensuring its independence. The main benefit of the consortium seems to be very close an intense support from the SQLite core developers, including customization and porting services. Consortium membership apparently starts at $75K a year so I doubt anyone joins just to get the test suite. There is some mention of separate licenses just for the test suite but I didn't see a price tag and I wonder if that generates significant revenue compared to the consortium. > given the immense good that the SQLite project has done for so many > other projects over the years. Maybe I should look into it more. I've never particularly felt the need for it since I've either used client/server databases or else simpler embedded databases like bsddb or even flat files. I actually think Macid/Happstack-state is brilliant but I haven't used it yet. > Many other less important projects have foundered for lack of > sustained funding. Other FOSS databases like Postgres, MySQL, MongoDB, Riak, Cassandra, etc. all seem to be doing fine. GNAT Ada (GCC-based Ada compiler and surrounding ecosystem) is split into free and proprietary components and you have to pay pretty big to get the proprietary parts, but the free parts aren't hobbled in any way (like missing tests) AFAIK. -- https://mail.python.org/mailman/listinfo/python-list
Re: 'Lite' Databases (Re: sqlite3 and dates)
In article <871tljepea@jester.gateway.pace.com>, Paul Rubin wrote: > Ned Deily writes: > > (though I don't know why anyone would want to fork it). > > Same reason lots of people have forked Postgres. Or you might just want > to customize it. Well, for whatever reason one might have, one can: it's public domain software. > > I imagine that is done as an incentive to help > > finance the on-going development and maintenance of SQLite. > It's a pretty unusual and annoying trick that other projects have not > felt they had to resort to. > > > https://www.sqlite.org/testing.html > > Thanks, that's the page I remember. The TH3 test suite is the > interesting one and you can't get it without paying a lot of $. I > guess there is some semantic quibble possible about whether you pay for > the test suite, or (as they put it) pay for Consortium membership and > then (as a member) get the test suite for free. I don't have any special knowledge of the history or current status of the Consortium but it's not difficult to find information and blog posts about it. It seems like this was an approach Richard Hipp and major users of SQLite took to ensure a sustaining funding model for the project while ensuring its independence. It strikes me as a very reasonable and modest constraint given the immense good that the SQLite project has done for so many other projects over the years. Many other less important projects have foundered for lack of sustained funding. https://www.sqlite.org/consortium.html https://blog.lizardwrangler.com/2008/02/27/the-sqlite-consortium/ > Do you know the situation with the SQL Logic Test (SLT) also mentioned > on that page? No -- Ned Deily, n...@acm.org -- https://mail.python.org/mailman/listinfo/python-list
