Re: Design thought for callbacks

2015-02-21 Thread Marko Rauhamaa
Steven D'Aprano :

> Marko Rauhamaa wrote:
>> Grant Edwards :
>>> the polite thing to do is to delete references to it when you're done
>>> with it.
>> 
>> I disagree with that recommendation. You should do the natural thing and
>> not care who holds references to who.
>
> I don't understand this. What is "the natural thing" if not to delete
> references to an object when you are done with it? Normally you just let
> things go out of scope, but if that won't happen, you have to take active
> steps, such as calling del or setting the reference to None.

Obviously, if you are hoarding objects in a collection, you're going to
land in trouble.

What I mean, though, is that you shouldn't think you need to create
object destructors where you routinely set all members to None.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Ian Kelly
On Sat, Feb 21, 2015 at 1:57 PM, Grant Edwards  wrote:
> On 2015-02-21, Cem Karan  wrote:
>>
>> On Feb 21, 2015, at 12:42 AM, Chris Angelico  wrote:
>>
>>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan  wrote:
 In order to inform users that certain bits of state have changed, I 
 require them to register a callback with my code.  The problem is that 
 when I store these callbacks, it naturally creates a strong reference to 
 the objects, which means that if they are deleted without unregistering 
 themselves first, my code will keep the callbacks alive.  Since this could 
 lead to really weird and nasty situations, I would like to store all the 
 callbacks in a WeakSet 
 (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That 
 way, my code isn't the reason why the objects are kept alive, and if they 
 are no longer alive, they are automatically removed from the WeakSet, 
 preventing me from accidentally calling them when they are dead.  My 
 question is simple; is this a good design?  If not, why not?  Are there 
 any potential 'gotchas' I should be worried about?

>>>
>>> No, it's not. I would advise using strong references - if the callback
>>> is a closure, for instance, you need to hang onto it, because there
>>> are unlikely to be any other references to it. If I register a
>>> callback with you, I expect it to be called; I expect, in fact, that
>>> that *will* keep my object alive.
>>
>> OK, so it would violate the principle of least surprise for you.
>
> And me as well.  I would expect to be able to pass a closure as a
> callback and not have to keep a reference to it.  Perhaps that just a
> leftover from working with other languages (javascript, scheme, etc.).
> It doesn't matter if it's a string, a float, a callback, a graphic or
> whatever: if I pass your function/library an object, I expect _you_ to
> keep track of it until you're done with it.
>
>> Interesting.  Is this a general pattern in python?  That is,
>> callbacks are owned by what they are registered with?
>
> I'm not sure what you mean by "owned" or why it matters that it's a
> callback: it's an object that was passed to you: you need to hold onto
> a reference to it until you're done with it, and the polite thing to
> do is to delete references to it when you're done with it.
>
>> So, what's the consensus on the list, strongly-held callbacks, or
>> weakly-held ones?

Count me in the weak-ref crowd. It may be a nuisance to keep a
reference around on the object registering the callback, but it's
preferable to the alternative of messing around with disposables in
order to ensure that the callback gets cleaned up and doesn't create a
memory leak. I would also rather have my code fail by losing a
callback reference, which should be relatively easy to spot and
diagnose, than to have said memory leak go unnoticed.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Frank Millman

"Steven D'Aprano"  wrote in message 
news:54e8af1b$0$12976$c3e8da3$54964...@news.astraweb.com...
> Frank Millman wrote:
>
>> I tried something similar a while ago, and I did find a gotcha.
>>
>> The problem lies in this phrase - "if they are no longer alive, they are
>> automatically removed from the WeakSet, preventing me from accidentally
>> calling them when they are dead."
>>
>> I found that the reference was not removed immediately, but was waiting 
>> to
>> be garbage collected. During that window, I could call the callback, 
>> which
>> resulted in an error.
>
> I don't understand how this could possibly work. (Or fail to work, as the
> case may be.)
>
> If the callback has been garbage collected, then you cannot call it, 
> because
> you don't have any references to it and so cannot refer to it in any way.
>
> If the callback has *not* been garbage collected, then you can safely call
> it. You have a reference to the callback, therefore it exists. (If Python
> ever garbage collects an object that still has references to it, that 
> would
> be a critical bug, and you would likely get some sort of seg fault).
>
> The only thing I can think of is you have a situation where your callback
> refers to another object, B, via a weak reference. Once all the regular
> strong references to the callback and B are gone, theoretically you could
> have a race condition where the callback is waiting to be garbage 
> collected
> but B has already been garbage collected. If, in that window, you call the
> callback, *and* if the callback fails to correctly check that the weak
> reference to B still exists, then you could get a Python exception.
>
> The solution is simple: anytime you have a weak reference, you must always
> check it before you use it.
>
> Other than that, I cannot see how calling a function which has *not* yet
> been garbage collected can fail, just because the only reference still
> existing is a weak reference.
>

You are right. I tried to reproduce the problem and I can't.

Before describing what I think was happening, I want to clarify something.

Most of this thread uses the word 'callback' in the sense of an 
'asynchronous' scenario - the caller wants something to happen some time in 
the future, and then forget about it, but it is important that it does 
actually happen eventually.

That is not what I was doing, and it is not what I thought the OP was asking 
for.

"In order to inform users that certain bits of state have changed, I require 
them to register a callback with my code."

This sounds to me like a pub/sub scenario. When a 'listener' object comes 
into existence it is passed a reference to a 'controller' object that holds 
state. It wants to be informed when the state changes, so it registers a 
callback function with the controller. When the controller detects a change 
in state, it calls all the callback functions, thereby notifying each 
listener. When the listener goes out of scope, it is important that it 
deregisters with the controller.

Now back to my scenario. You are right that so long as the controller 
maintains a reference to the callback function, the listener cannot be 
garbage collected, and therefore the callback will always succeed.

As far as I can remember, I had a situation where the listener used the 
information to pass the information to a gui on a client. When the listener 
was no longer required, a close() fiunction was called which cleaned up and 
closed connections. When the callback was called after the listener was 
closed, whatever it was trying to do failed (I forget the details).

Hope this makes sense.

Frank



-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Chris Angelico
On Sun, Feb 22, 2015 at 3:38 PM, Steven D'Aprano
 wrote:
> But you are using it. You might not be using it by name, but you are using
> it via the callback function. What did you expect, that Python should read
> your mind and somehow intuit that you still care about this socket
> listener, but not some other socket listener that you are done with?
>
> You don't have to bind the listener to a name. Any reference will do. You
> can dump it in a bucket:
>
> bucket_of_stuff = []
> bucket_of_stuff.append(some_function(a, b, c))
> bucket_of_stuff.append(make_web_server())
> bucket_of_stuff.append(socket(23, on_accept=client_connected))

Sure, and whether it's a name or a list-element reference doesn't
matter: it seems wrong to have to stash a thing in a bucket in order
to keep its callbacks alive. I expect the callbacks _themselves_ to
keep it alive. But I can understand the opposite POV.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: How to design a search engine in Python?

2015-02-21 Thread subhabangalore
On Sunday, February 22, 2015 at 11:08:47 AM UTC+5:30, Denis McMahon wrote:
> On Sat, 21 Feb 2015 21:02:34 -0800, subhabangalore wrote:
> 
> > Thank you for your suggestion. But I was looking for a small tutorial of
> > algorithm of the whole engine. I would try to check it build individual
> > modules and integrate them. I was getting some in google and youtube,
> > but I tried to consult you as I do not know whether they would be fine.
> > I am trying your way, let me see how much I go. There are so many search
> > algorithms in our popular data structure books, that is not an issue but
> > how a search engine is getting done, I am thinking bit on that.
> 
> Presumably a search engine is simply a database of keyword -> result, 
> possibly with some scoring factor.
> 
> Calculating scoring factor is going to be fun.
> 
> Then of course result pages might have scoring factors too. What about a 
> search with multiple keywords. Some result pages might match more than 
> one keyword, so you might add their score for each keyword together to 
> get the ranking in that enquiry for that page.
> 
> But then pages with lots and lots of different keywords might be low 
> scoring, because searchers are looking for content, not pages of keywords.
> 
> Finally, What special, unique feature is your search engine going to have 
> that makes it better than all the existing ones?
> 
> -- 
> Denis McMahon,
Dear Sir,

Thank you for your kind suggestion. Let me traverse one by one. 
My special feature is generally Semantic Search, but I am trying to build
a search engine first and then go for semantic I feel that would give me a 
solid background to work around the problem. 

Regards,
Subhabrata. 


-- 
https://mail.python.org/mailman/listinfo/python-list


Re: How to design a search engine in Python?

2015-02-21 Thread Denis McMahon
On Sat, 21 Feb 2015 21:02:34 -0800, subhabangalore wrote:

> Thank you for your suggestion. But I was looking for a small tutorial of
> algorithm of the whole engine. I would try to check it build individual
> modules and integrate them. I was getting some in google and youtube,
> but I tried to consult you as I do not know whether they would be fine.
> I am trying your way, let me see how much I go. There are so many search
> algorithms in our popular data structure books, that is not an issue but
> how a search engine is getting done, I am thinking bit on that.

Presumably a search engine is simply a database of keyword -> result, 
possibly with some scoring factor.

Calculating scoring factor is going to be fun.

Then of course result pages might have scoring factors too. What about a 
search with multiple keywords. Some result pages might match more than 
one keyword, so you might add their score for each keyword together to 
get the ranking in that enquiry for that page.

But then pages with lots and lots of different keywords might be low 
scoring, because searchers are looking for content, not pages of keywords.

Finally, What special, unique feature is your search engine going to have 
that makes it better than all the existing ones?

-- 
Denis McMahon, denismfmcma...@gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: How to design a search engine in Python?

2015-02-21 Thread subhabangalore
On Sunday, February 22, 2015 at 10:12:39 AM UTC+5:30, Steven D'Aprano wrote:
> wrote:
> 
> > Dear Group,
> > 
> > I am trying to build a search engine in Python.
> 
> How to design a search engine in Python?
> 
> First, design a search engine.
> 
> Then, write Python code to implement that search engine.
> 
> 
> > To do this, I have read tutorials and working methodologies from web and
> > books like Stanford IR book [ http://www-nlp.stanford.edu/IR-book/]. I
> > know how to design a crawler, I know PostgresSql, I am fluent with
> > PageRank, TF-IDF, Zipf's law, etc. I came to know of
> > Whoosh[https://pypi.python.org/pypi/Whoosh/]
> 
> How does your search engine work? What does it do?
> 
> You MUST be able to describe the workings of your search engine in English,
> or the natural language of your choice. Write out the steps that it must
> take, the tasks that it must perform. This is your algorithm. Without an
> algorithm, how do you expect to write code? What will the code do?
> 
> Once you have designed your search engine algorithm, then *and only then*
> should you start to write code to implement that algorithm.
> 
> 
> 
> 
> -- 
> Steven

Dear Sir,

Thank you for your suggestion. But I was looking for a small tutorial of 
algorithm of the whole engine. I would try to check it build individual modules 
and integrate them. I was getting some in google and youtube, but I tried to 
consult you as I do not know whether they would be fine. I am trying your way, 
let me see how much I go. There are so many search algorithms in our popular 
data structure books, that is not an issue but how a search engine is getting 
done, I am thinking bit on that. 

Regards,
Subhabrata.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: How to design a search engine in Python?

2015-02-21 Thread Steven D'Aprano
subhabangal...@gmail.com wrote:

> Dear Group,
> 
> I am trying to build a search engine in Python.

How to design a search engine in Python?

First, design a search engine.

Then, write Python code to implement that search engine.


> To do this, I have read tutorials and working methodologies from web and
> books like Stanford IR book [ http://www-nlp.stanford.edu/IR-book/]. I
> know how to design a crawler, I know PostgresSql, I am fluent with
> PageRank, TF-IDF, Zipf's law, etc. I came to know of
> Whoosh[https://pypi.python.org/pypi/Whoosh/]

How does your search engine work? What does it do?

You MUST be able to describe the workings of your search engine in English,
or the natural language of your choice. Write out the steps that it must
take, the tasks that it must perform. This is your algorithm. Without an
algorithm, how do you expect to write code? What will the code do?

Once you have designed your search engine algorithm, then *and only then*
should you start to write code to implement that algorithm.




-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Steven D'Aprano
Chris Angelico wrote:

> On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
>  wrote:
>> Marko Rauhamaa wrote:
>>
>>> Grant Edwards :
>>>
 the polite thing to do is to delete references to it when you're done
 with it.
>>>
>>> I disagree with that recommendation. You should do the natural thing and
>>> not care who holds references to who.
>>
>> I don't understand this. What is "the natural thing" if not to delete
>> references to an object when you are done with it? Normally you just let
>> things go out of scope, but if that won't happen, you have to take active
>> steps, such as calling del or setting the reference to None.
> 
> I think the disagreement here is over the interpretation of "done with
> it". If you drop all references to a connected socket object, Python
> can rightly assume that you're done with the file and want to close
> it; but what if you drop all references to a listening socket that's
> been configured to call a function whenever someone connects?
> 
> def client_connected(sock):
> sock.send("Hello!\r\n")
> # whatever
> 
> listener = socket(23, on_accept=client_connected)
> 
> What should happen if that main socket isn't bound to a name? In my
> opinion, the fact that it's configured for callback mode should mean
> that it's kept alive. But it's also understandable to want to treat it
> as "done", that it can be disposed of. It seems weird to me that you
> should have to have a name somewhere that you'll never use, though.

But you are using it. You might not be using it by name, but you are using
it via the callback function. What did you expect, that Python should read
your mind and somehow intuit that you still care about this socket
listener, but not some other socket listener that you are done with?

If you have a servant that follows you around everywhere, throwing objects
away when you stop using them, then naturally if you stop using something
it will be thrown away. What did you expect?

You don't have to bind the listener to a name. Any reference will do. You
can dump it in a bucket:

bucket_of_stuff = []
bucket_of_stuff.append(some_function(a, b, c))
bucket_of_stuff.append(make_web_server())
bucket_of_stuff.append(socket(23, on_accept=client_connected))


So long as your bucket is still alive, the garbage collector won't collect
it or its contents.

Hypothetically we could have a system in place where you instruct the
garbage collector to not collect an object, then drop all references to it:

gc.never_collect(socket(23, on_accept=client_connected))

but that's a potential memory leak, because now you have no way of telling
the GC to collect it again once you've finished listening. If you never
finish listening, or at least not until your application shuts down, it
doesn't count as a leak. But in general, if listeners might come and go,
but you have no way for them to be garbage collected once they are done,
then that's a leak.

What's the easiest way for the GC to flag an object as "never collect this"?
It can keep a reference to it. Keeping a list of things you want to be kept
alive is simple, easy and obvious:

# hypothetically inside gc.py
_ALIVE = []
def never_collect(obj):
_ALIVE.append(obj)


but that's so trivial that it's not worth putting it in the gc module.
Besides, that means any other code could reach into the gc and remove your
listener from the "keep alive list" without your knowledge or permission.

It's simpler and safer to just keep it alive yourself:

alive = []
alive.append(socket(...))

but of course that's just my bucket of stuff under a different name.

Using the idiom "keep objects alive by keeping a reference to them" (e.g.
bind them to a name, or stick them in a list which you keep) makes things
much simpler. You can trivially flag objects as "collect" or "don't
collect" as needed, without having to import the gc module or memorise some
obscure API or worry about implementation details ("if I flag an object
as 'never collect' *twice*, do I have to unflag it twice to undo it?"). You
just use the regular interface to the garbage collector: the existence of a
reference, any reference, keeps an object alive.




-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements

2015-02-21 Thread Steven D'Aprano
TommyVee wrote:

> Start off with sets of elements as follows:
> 
> 1. A,B,E,F
> 2. G,H,L,P,Q
> 3. C,D,E,F
> 4. E,X,Z
> 5. L,M,R
> 6. O,M,Y
> 
> Note that sets 1, 3 and 4 all have the element 'E' in common, therefore
> they are "related" and form the following superset:
> 
> A,B,C,D,E,F,X,Z

Sounds to me like you already have an algorithm in mind. It might not be
correct or complete, or it might not be efficient, but you performed some
steps in your own mind to generate that superset. Write down those steps
and you have an algorithm. Once you have the algorithm, you can then turn
it into Python code and start revising it from there.

I went back to first principles and drew a set diagram with multiple
disjoint sets of overlapping sets, and then thought about the process of
*adding* each new set to the set-of-supersets. It's hard to describe, but
if you start drawing an empty Venn diagram, then add one of your sets to
the Venn diagram at a time, merging that set to whatever superset it
belongs to, the process should appear quite obvious.

Let me walk through the process with your six sets above, plus one extra.
First, I create a list of supersets, initially empty:

supersets = []

I look at Set #1, and compare to all the supersets. There aren't any, so I
make a copy of Set #1 and add it to the supersets:

[{A,B,E,F}]

I look at Set #2, and compare it to all the supersets. It doesn't intersect
any of them, so I add it as a new superset:

[{A,B,E,F}, {G,H,L,P,Q}]

I look at Set #3. It intersects the first superset, but not the second, so I
merge them. Likewise for Set #4:

[{A,B,C,D,E,F}, {G,H,L,P,Q}]
[{A,B,C,D,E,F,X,Z}, {G,H,L,P,Q}]

Set #5 intersects the second superset, but not the first. Same for Set #6:

[{A,B,C,D,E,F,X,Z}, {G,H,L,M,P,Q,R}]
[{A,B,C,D,E,F,X,Z}, {G,H,L,M,O,P,Q,R,Y}]

Now let me add an extra Set #7, {A, R}, which intersects *both* supersets.
First I merge it with both:

[{A,B,C,D,E,F,R,X,Z}, {A,G,H,L,M,O,P,Q,R,Y}]

but I don't stop there. Then I recursively run through the supersets with
the same merging process to get:

[{A,B,C,D,E,F,G,H,L,M,O,P,Q,R,X,Y,Z}]

which is the single minimal, disjoint superset of all seven sets.



def merge(sets):
"""Merge a collection of sets into a list of minimal, disjoint
supersets.
"""
supersets = []
for a in sets:
count = 0
for s in supersets:
if a&s:
s.update(a)
count += 1
if count == 0:
supersets.append(a.copy())
elif count > 1:
supersets = merge(supersets)
return supersets


Testing this with your six sets gives:

py> sets = [set(s) for s in ("ABEF", "GHLPQ", "CDEF", "EXZ", "LMR", "OMY")]
py> for s in merge(sets):
... print(','.join(sorted(s)))
...
A,B,C,D,E,F,X,Z
G,H,L,M,O,P,Q,R,Y



Adding set #7 gives:

py> for s in merge(sets + [{'A', 'R'}]):
... print(','.join(sorted(s)))
...
A,B,C,D,E,F,G,H,L,M,O,P,Q,R,X,Y,Z





-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Chris Angelico
On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
 wrote:
> Marko Rauhamaa wrote:
>
>> Grant Edwards :
>>
>>> the polite thing to do is to delete references to it when you're done
>>> with it.
>>
>> I disagree with that recommendation. You should do the natural thing and
>> not care who holds references to who.
>
> I don't understand this. What is "the natural thing" if not to delete
> references to an object when you are done with it? Normally you just let
> things go out of scope, but if that won't happen, you have to take active
> steps, such as calling del or setting the reference to None.

I think the disagreement here is over the interpretation of "done with
it". If you drop all references to a connected socket object, Python
can rightly assume that you're done with the file and want to close
it; but what if you drop all references to a listening socket that's
been configured to call a function whenever someone connects?

def client_connected(sock):
sock.send("Hello!\r\n")
# whatever

listener = socket(23, on_accept=client_connected)

What should happen if that main socket isn't bound to a name? In my
opinion, the fact that it's configured for callback mode should mean
that it's kept alive. But it's also understandable to want to treat it
as "done", that it can be disposed of. It seems weird to me that you
should have to have a name somewhere that you'll never use, though.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Steven D'Aprano
Marko Rauhamaa wrote:

> Grant Edwards :
> 
>> the polite thing to do is to delete references to it when you're done
>> with it.
> 
> I disagree with that recommendation. You should do the natural thing and
> not care who holds references to who.

I don't understand this. What is "the natural thing" if not to delete
references to an object when you are done with it? Normally you just let
things go out of scope, but if that won't happen, you have to take active
steps, such as calling del or setting the reference to None.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements

2015-02-21 Thread TommyVee

"TommyVee"  wrote in message news:Bg5Gw.1344030$no4.494...@fx19.iad...

Start off with sets of elements as follows:

1. A,B,E,F
2. G,H,L,P,Q
3. C,D,E,F
4. E,X,Z
5. L,M,R
6. O,M,Y

Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they
are "related" and form the following superset:

A,B,C,D,E,F,X,Z

Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have
element 'M' in common, therefore they form the following superset:

G,H,L,M,O,P,Q,R,Y

I think you get the point.  As long as sets have at least 1 common element,
they combine to form a superset.  Also "links" (common elements) between
sets may go down multiple levels, as described in the second case above
(2->5->6).  Cycles thankfully, are not possible.

BTW, the number of individual sets (and resultant supersets) will be very
large.

I don't know where to start with this.  I thought about some type of
recursive algorithm, but I'm not sure.  I could figure out the Python
implementation easy enough, I'm just stumped on the algorithm itself.

Anybody have an idea?

Thanks, Tom

Thanks to all!  Not only did you turn around my thinking on this, but I also 
learned about some new Python collection types.  I think I have a way to get 
this to work now.


--
https://mail.python.org/mailman/listinfo/python-list


Re: pypandoc and restructured text

2015-02-21 Thread alb
Hi Chris,

Chris Angelico  wrote:
[]
>> Thanks a lot for the hint. Maybe I should seriously think about
>> upgrading the whole distro. It's just that Gnome3 really sucks to my
>> taste and I'm not in the mood to look for another Desktop
>> Environment...(maybe I should go back to CDE).
>>
> 
> This is a bit off-topic for python-list, being more about Debian than
> the language, but trying to install just one Wheezy package on Squeeze
> is probably a bad idea; you'll probably find that it's built against a
> different version of libc and such, so you'd need to upgrade pretty
> much your whole system.

I apologize for littering the group, thought somehow many users here 
might have had a similar issues with pypandoc.

[]
> 2) Upgrade the entire OS to either Wheezy or Jessie. Personally, I'm
> quite happy with Jessie, even though she's not officially stable yet
> (and have you _seen_ her? Definitely mentally unstable); but whether
> you go Wheezy or Jessie, grab Xfce rather than Gnome 3. The latest
> Jessie installers are giving an option on installation as to which
> desktop environment(s) to install, but if you do it as an upgrade, you
> may want to explicitly uninstall Gnome and install Xfce, either before
> or after the upgrade.

I finally upgraded! And I'm currently trying out xfce!
Thanks again for the suggestions.

Al

p.s.: now pandoc works as expected.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements

2015-02-21 Thread Mark Lawrence

On 21/02/2015 19:46, TommyVee wrote:

Start off with sets of elements as follows:

1. A,B,E,F
2. G,H,L,P,Q
3. C,D,E,F
4. E,X,Z
5. L,M,R
6. O,M,Y

Note that sets 1, 3 and 4 all have the element 'E' in common, therefore
they are "related" and form the following superset:

A,B,C,D,E,F,X,Z

Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6
have element 'M' in common, therefore they form the following superset:

G,H,L,M,O,P,Q,R,Y

I think you get the point.  As long as sets have at least 1 common
element, they combine to form a superset.  Also "links" (common
elements) between sets may go down multiple levels, as described in the
second case above (2->5->6).  Cycles thankfully, are not possible.

BTW, the number of individual sets (and resultant supersets) will be
very large.

I don't know where to start with this.  I thought about some type of
recursive algorithm, but I'm not sure.  I could figure out the Python
implementation easy enough, I'm just stumped on the algorithm itself.

Anybody have an idea?

Thanks, Tom


A naive approach but should give you something to think about.

from collections import defaultdict

sets = ({'A','B','E','F'},
{'G','H','L','P','Q'},
{'C','D','E','F'},
{'E','X','Z'},
{'L','M','R'},
{'O','M','Y'})

d = defaultdict(list)

for i, aSet in enumerate(sets):
for a in aSet:
d[a].append(i)

superSets = []
for k in d:
if len(d[k]) > 1:
superSet = set()
for i in d[k]:
superSet |= sets[i]
superSets.append(superSet)

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Marko Rauhamaa
Grant Edwards :

> the polite thing to do is to delete references to it when you're done
> with it.

I disagree with that recommendation. You should do the natural thing and
not care who holds references to who.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Grant Edwards
On 2015-02-21, Cem Karan  wrote:
>
> On Feb 21, 2015, at 12:42 AM, Chris Angelico  wrote:
>
>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan  wrote:
>>> In order to inform users that certain bits of state have changed, I require 
>>> them to register a callback with my code.  The problem is that when I store 
>>> these callbacks, it naturally creates a strong reference to the objects, 
>>> which means that if they are deleted without unregistering themselves 
>>> first, my code will keep the callbacks alive.  Since this could lead to 
>>> really weird and nasty situations, I would like to store all the callbacks 
>>> in a WeakSet 
>>> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That 
>>> way, my code isn't the reason why the objects are kept alive, and if they 
>>> are no longer alive, they are automatically removed from the WeakSet, 
>>> preventing me from accidentally calling them when they are dead.  My 
>>> question is simple; is this a good design?  If not, why not?  Are there any 
>>> potential 'gotchas' I should be worried about?
>>> 
>> 
>> No, it's not. I would advise using strong references - if the callback
>> is a closure, for instance, you need to hang onto it, because there
>> are unlikely to be any other references to it. If I register a
>> callback with you, I expect it to be called; I expect, in fact, that
>> that *will* keep my object alive.
>
> OK, so it would violate the principle of least surprise for you.

And me as well.  I would expect to be able to pass a closure as a
callback and not have to keep a reference to it.  Perhaps that just a
leftover from working with other languages (javascript, scheme, etc.).
It doesn't matter if it's a string, a float, a callback, a graphic or
whatever: if I pass your function/library an object, I expect _you_ to
keep track of it until you're done with it.

> Interesting.  Is this a general pattern in python?  That is,
> callbacks are owned by what they are registered with?

I'm not sure what you mean by "owned" or why it matters that it's a
callback: it's an object that was passed to you: you need to hold onto
a reference to it until you're done with it, and the polite thing to
do is to delete references to it when you're done with it.

> So, what's the consensus on the list, strongly-held callbacks, or
> weakly-held ones?

--
Grant

-- 
https://mail.python.org/mailman/listinfo/python-list


How to design a search engine in Python?

2015-02-21 Thread subhabangalore
Dear Group, 

I am trying to build a search engine in Python. 

To do this, I have read tutorials and working methodologies from web and books 
like Stanford IR book [ http://www-nlp.stanford.edu/IR-book/]. I know how to 
design a crawler, I know PostgresSql, I am fluent with PageRank, TF-IDF, Zipf's 
law, etc. 
I came to know of Whoosh[https://pypi.python.org/pypi/Whoosh/]

But I am looking for a total tutorial how to implement it. If any body may 
kindly direct me. 

I heard there are good source codes and prototypes, but I am not getting. 

Apology if this is not a question of the room. I tried to post as this is a 
room of Python bigwigs. 

Regards,
Subhabrata. 
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subprocess command fails

2015-02-21 Thread Larry Hudson

On 02/20/2015 08:47 PM, Brad s wrote:
[...]

print("command = %r" % (cmdargv,))
sfmove = subprocess.call(cmdargv)



also, is the % to Python what precision is to C++?



No.  It is like the % in C's printf().

 -=- Larry -=-

--
https://mail.python.org/mailman/listinfo/python-list


Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements

2015-02-21 Thread Dave Angel

On 02/21/2015 02:46 PM, TommyVee wrote:

Start off with sets of elements as follows:

1. A,B,E,F
2. G,H,L,P,Q
3. C,D,E,F
4. E,X,Z
5. L,M,R
6. O,M,Y

Note that sets 1, 3 and 4 all have the element 'E' in common, therefore
they are "related" and form the following superset:

A,B,C,D,E,F,X,Z

Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6
have element 'M' in common, therefore they form the following superset:

G,H,L,M,O,P,Q,R,Y

I think you get the point.  As long as sets have at least 1 common
element, they combine to form a superset.  Also "links" (common
elements) between sets may go down multiple levels, as described in the
second case above (2->5->6).  Cycles thankfully, are not possible.

BTW, the number of individual sets (and resultant supersets) will be
very large.

I don't know where to start with this.


I can't see where you've defined "this" yet.  I think you're trying to 
find all combinations of sets that are "related", and that you're 
assuming some of them won't be.  So perhaps you're trying to build 
"islands" of related sets, and you want each of those islands to be of 
maximal size.



 I thought about some type of
recursive algorithm, but I'm not sure.  I could figure out the Python
implementation easy enough, I'm just stumped on the algorithm itself.



Seems like you can have a doubly-nested loop, where your inner loop 
collects all the sets that are related to the one in the outer loop. 
You label each group with an island-number, where the island number is 
simply the lowest number set that's on the island.


It's still not clear what you do with each of those islands, but the 
"superset" you refer to before is simply the OR of everything on the island.


Perhaps you want to show all smaller supersets that are composed of at 
least two sets of an island.  That becomes a fairly simple power-of-two 
iteration of those sets, where you throw out the case where the index is 
a power of two (ie. contains only one set).


As you say, calculating the union or intersection of the various sets in 
python is trivial.


--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list


Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements

2015-02-21 Thread Ethan Furman
On 02/21/2015 11:46 AM, TommyVee wrote:
> Start off with sets of elements as follows:
> 
> 1. A,B,E,F
> 2. G,H,L,P,Q
> 3. C,D,E,F
> 4. E,X,Z
> 5. L,M,R
> 6. O,M,Y
> 
> Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they 
> are "related" and form the following superset:
> 
> A,B,C,D,E,F,X,Z
> 
> Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have 
> element 'M' in common, therefore they form
> the following superset:
> 
> G,H,L,M,O,P,Q,R,Y
> 
> I think you get the point.  As long as sets have at least 1 common element, 
> they combine to form a superset.  Also
> "links" (common elements) between sets may go down multiple levels, as 
> described in the second case above (2->5->6). 
> Cycles thankfully, are not possible.
> 
> BTW, the number of individual sets (and resultant supersets) will be very 
> large.
> 
> I don't know where to start with this.  I thought about some type of 
> recursive algorithm, but I'm not sure.  I could
> figure out the Python implementation easy enough, I'm just stumped on the 
> algorithm itself.
> 
> Anybody have an idea?

Use a Counter (collections.Counter), add all sets, then keep any keys with a 
count of 2 or more.

--
~Ethan~



signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Algorithm for Creating Supersets of Smaller Sets Based on Common Elements

2015-02-21 Thread Joel Goldstick
On Sat, Feb 21, 2015 at 2:46 PM, TommyVee  wrote:
> Start off with sets of elements as follows:
>
> 1. A,B,E,F
> 2. G,H,L,P,Q
> 3. C,D,E,F
> 4. E,X,Z
> 5. L,M,R
> 6. O,M,Y
>
> Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they
> are "related" and form the following superset:
>
> A,B,C,D,E,F,X,Z
>
> Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have
> element 'M' in common, therefore they form the following superset:
>
> G,H,L,M,O,P,Q,R,Y
>
> I think you get the point.  As long as sets have at least 1 common element,
> they combine to form a superset.  Also "links" (common elements) between
> sets may go down multiple levels, as described in the second case above
> (2->5->6).  Cycles thankfully, are not possible.
>
> BTW, the number of individual sets (and resultant supersets) will be very
> large.
>
> I don't know where to start with this.  I thought about some type of
> recursive algorithm, but I'm not sure.  I could figure out the Python
> implementation easy enough, I'm just stumped on the algorithm itself.
>
> Anybody have an idea?
>

start with reading about python sets.  If you take the intersection of
two sets it will return a set with common elements.  If that is empty,
they don't pass your test.  If you take the union, you get a set with
the set values in each.


> Thanks, Tom
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Joel Goldstick
http://joelgoldstick.com
-- 
https://mail.python.org/mailman/listinfo/python-list


Algorithm for Creating Supersets of Smaller Sets Based on Common Elements

2015-02-21 Thread TommyVee

Start off with sets of elements as follows:

1. A,B,E,F
2. G,H,L,P,Q
3. C,D,E,F
4. E,X,Z
5. L,M,R
6. O,M,Y

Note that sets 1, 3 and 4 all have the element 'E' in common, therefore they 
are "related" and form the following superset:


A,B,C,D,E,F,X,Z

Likewise, sets 2 and 5 have the element 'L' in common, then set 5 and 6 have 
element 'M' in common, therefore they form the following superset:


G,H,L,M,O,P,Q,R,Y

I think you get the point.  As long as sets have at least 1 common element, 
they combine to form a superset.  Also "links" (common elements) between 
sets may go down multiple levels, as described in the second case above 
(2->5->6).  Cycles thankfully, are not possible.


BTW, the number of individual sets (and resultant supersets) will be very 
large.


I don't know where to start with this.  I thought about some type of 
recursive algorithm, but I'm not sure.  I could figure out the Python 
implementation easy enough, I'm just stumped on the algorithm itself.


Anybody have an idea?

Thanks, Tom 


--
https://mail.python.org/mailman/listinfo/python-list


Re: 'Lite' Databases (Re: sqlite3 and dates)

2015-02-21 Thread Eric S. Johansson


On 02/21/2015 01:22 AM, Ned Deily wrote:
SQLite is one of the most widely-used, best-documented, best-tested, 
and well-respected software packages in the world. 
yes but is still sql.  there are a couple of small scale not-sql 
databases that look interesting.  problem with them is that the creator 
seem to subscribe to the build-and-toss-into-the-wild school of development.


http://buzhug.sourceforge.net/
http://www.pydblite.net/en/index.html

both are useful, both could use multi-writer support, and both need some 
love from the python world.


--- eric
--
https://mail.python.org/mailman/listinfo/python-list


Re: Accessible tools

2015-02-21 Thread Tim Chase
On 2015-02-21 10:21, Bryan Duarte wrote:
> those of us who rely on screen readers to interact with our
> computers have a few things we do, and tend to not do. 
[snip]

While my experience has shown most of your items to be true, I'd
contend that

>• Do not, have access to debugging tools.

is mistaken or at least misinformed.  For Python, I use the "pdb"
module all the time, and it's command-line driven.  Combined with a
multi-terminal (whether multiple windows, virtual consoles, or a
tmux/screen session), I can easily bounce back and forth between a
"pdb" debugging session and the source code to make edits.  Just to
check, I fired up the "yasr" terminal screen-reader, launched tmux
(using my quiet config, since it updates information on the screen
like the time on a regular basis, making it chatty), and stepped
through some Python code, checked variables, and walked up/down the
call-stack.  I know most other languages have similar functionality
such as gdb for C code.

-tkc





-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Steven D'Aprano
Cem Karan wrote:

> 
> On Feb 21, 2015, at 8:15 AM, Chris Angelico  wrote:
> 
>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan  wrote:
>>> OK, so it would violate the principle of least surprise for you. 
>>> Interesting.  Is this a general pattern in python?  That is, callbacks
>>> are owned by what they are registered with?
>>> 
>>> In the end, I want to make a library that offers as few surprises to the
>>> user as possible, and no matter how I think about callbacks, they are
>>> surprising to me.  If callbacks are strongly-held, then calling 'del
>>> foo' on a callable object may not make it go away, which can lead to
>>> weird and nasty situations.

How?

The whole point of callbacks is that you hand over responsibility to another
piece of code, and then forget about your callback. The library will call
it, when and if necessary, and when the library no longer needs your
callback, it is free to throw it away. (If I wish the callback to survive
beyond the lifetime of your library's use of it, I have to keep a reference
to the function.)


>>> Weakly-held callbacks mean that I (as the 
>>> programmer), know that objects will go away after the next garbage
>>> collection (see Frank's earlier message), so I don't get 'dead'
>>> callbacks coming back from the grave to haunt me.

I'm afraid this makes no sense to me. Can you explain, or better still
demonstrate, a scenario where "dead callbacks rise from the grave", so to
speak?


>>> So, what's the consensus on the list, strongly-held callbacks, or
>>> weakly-held ones?
>> 
>> I don't know about Python specifically, but it's certainly a general
>> pattern in other languages. They most definitely are owned, and it's
>> the only model that makes sense when you use closures (which won't
>> have any other references anywhere).
> 
> I agree about closures; its the only way they could work.

*scratches head* There's nothing special about closures. You can assign them
to a name like any other object.

def make_closure():
x = 23
def closure():
return x + 1
return closure

func = make_closure()

Now you can register func as a callback, and de-register it when your done:

register(func)
unregister(func)


Of course, if you thrown away your reference to func, you have no (easy) way
of de-registering it. That's no different to any other object which is
registered by identity. (Registering functions by name is a bad idea, since
multiple functions can have the same name.)

As an alternative, your callback registration function might return a ticket
for the function:

ticket = register(func)
del func
unregister(ticket)

but that strikes me as over-kill. And of course, the simplest ticket is to
return the function itself :-)



> When I was 
> originally thinking about the library, I was trying to include all types
> of callbacks, including closures and callable objects.  The callable
> objects may pass themselves, or one of their methods to the library, or
> may do something really weird.

I don't think they can do anything too weird. They have to pass a callable
object. Your library just calls that object. You shouldn't need to care
whether it is a function, a method, a type, a callable instance, or
something else. You just call it, and when you're done calling it forever,
you just throw it away.


> Although I just realized that closures may cause another problem.  In my
> code, I expect that many different callbacks can be registered for the
> same event.  Unregistering means you request to be unregistered for the
> event. How do you do that with a closure?  Aren't they anonymous?

Not unless you create them using lambda. Using the make_closure function
above:


py> func = make_closure()
py> func.__name__
'closure'

Of course, if you call make_closure twice, both functions will have the same
internal name. You can set the function __name__ and __qualname__ to fix
that. This is how the functools.wraps decorator works.

But that's a red herring. Don't register functions by name! Not all callable
objects have names, and those that do, you may have multiple *distinct*
callbacks with the same name.

There are two reasonable approaches: unregister by identity, or by returning
a ticket which uniquely identifies the callback. The user is responsible
for keeping track of their own ticket. If I lose it, I can't unregister my
callback any more. So sad, sucks to be me.


The simplest possible identity-based scheme would be something like this:


# don't hate me for using a global variable
CALLBACKS = []

def register(func):
if func not in CALLBACKS:
 CALLBACKS.append(func)

def unregister(func):
try:
CALLBACKS.remove(func)
except ValueError:
pass


That's probably a bit too simple, since it won't behave as expected with
bound methods. The problem is that bound methods are generated on the fly,
so this won't work:

register(instance.spam)
# later
unregister(instance.spam)  # a different instance!

I would have to do this:

boun

Re: Accessible tools

2015-02-21 Thread Bryan Duarte
Tim,

I am also on the blind linux list. I do not often post there as I predominately 
use a Mac and the Unix terminal but I am using Linux Kali on the side for some 
side tinkering and learning. I would use Linux a lot more if the screen reader 
was not so robotic... Would you be willing to be included in some accessibility 
Q&A or to bounce some ideas off of? 
> On Feb 19, 2015, at 11:43 AM, Tim Chase  wrote:
> 
> While not blind, I have an interest in accessibility and answer a
> number of questions on the Blinux (Blind Linux Users) mailing list.
> 
> On 2015-02-19 08:33, Bryan Duarte wrote:
>> A professor and I have been throwing around the idea of developing
>> a completely text based IDE. There are a lot of reasons this could
>> be beneficial to a blind developer and maybe even some sighted
>> developers who are comfortable in the terminal. The idea would be
>> really just to provide a way of easily navigating blocks of code
>> using some kind of tabular formatting, and being able to collapse
>> blocks of code and hearing from a high level information about the
>> code within. All tools and features would obviously be spoken or
>> output in some kind of audio manor. 
> 
> It would seem that the traditional Unix-as-IDE[1] would serve you well
> here.  This is my method of choice, and it allows me to pick my
> components and combine them.  I usually use tmux, though GNU screen
> would do too.  Within that, I usually have the following:
> 
> - vim to edit my code.  Though swap in your favorite, whether
>  emacs/emacspeak, ed/edbrowse, joe, nano, or whatever.  I know that
>  at least Vim and emacs support "folding" away blocks of code (what
>  you describe as "collapsing") which I usually prefix with a comment
>  that would give you a description of the block
> 
> - a command-line (I use bash, some prefer zsh or tcsh or whatever)
>  for things like version-control, running my code, and file
>  management (move/copy/delete/rename/link/etc)
> 
> - a Python command-line REPL that allows me to do quick tests on a
>  line of code as well as well as make extensive use of Python's
>  built-in dir() and help() commands which are invaluable.
> 
> - when doing web-development (Django in my case), I'll often have the
>  dev-server running in one pane, and a console browser like
>  lynx/links/links2/elinks/w3m in another pane so that I can put my
>  code through its paces
> 
> Another benefit of this is that I can run this on my development
> machine, but then SSH into the machine from anywhere, reattach to the
> tmux/screen session, and have the same configuration right as I left
> it.
> 
> The entire tmux/screen session can be run within an accessible
> terminal window (I know that some are more accessible than others),
> within a terminal screen-reader session (like yasr, screader, or
> emacspeak), or even remoted into via an accessible SSH program on your
> platform of choice.
> 
> -tkc
> 
> [1]
> http://blog.sanctum.geek.nz/series/unix-as-ide/
> 
> 
> 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Accessible tools

2015-02-21 Thread Bryan Duarte
Eric,

Although I would most likely enjoy the former option I feel the latter would be 
most appropriate for contacting you. Thanks for getting back to me and 
explaining some of this. I will contact you off list for sure so as not to fill 
up the lists mailboxes with this topic. 

I will say that Jacob is absolutely correct in the fact that those of us who 
rely on screen readers to interact with our computers have a few things we do, 
and tend to not do.
• Do type very efficiently
• Do listen to our speech output very fast
• Do interact with the text line, word, and character at a time.
• Do not, use a mouse
• Do not, enjoy reading through several lines of code at a time to locate a 
specific issue, method, or block of code.
• Do not, have access to debugging tools.

This is just a few things we do and do not do typically, but the point is that 
even though some things are similar there are some things that are quite 
different. Here is an example. I would not use a voice driven editor because I 
tend to build out a layout of code before I begin filling in methods, 
functions, and constructors. For me to do this with voice would require the use 
of a mouse to place the cursor where I want each line of code to be placed, and 
since I do not have access to the mouse in that way this would not work well. 
On the other hand if the program could interpret what you are speaking to build 
out method signatures and individual lines of code accurately, then there is a 
good chance that same system could provide a great interface for a screen 
reader. Thank you again for the response and be expecting an email from me 
shortly sir.  
> On Feb 20, 2015, at 11:39 AM, Jacob Kruger  wrote:
> 
> Eric, issue is that with screenreaders, we're generally way more into
> navigating code and interface character by character/by keyboard, so , yes,
> keeping interface relatively simple is a good thing, but, we also would
> prefer to primarily keep all interface elements to make use of standard UI
> controls, and make sure tab index/order is suitable/relevant at times, etc.
> etc.
> 
> As in, I think we'd primarily want to avoid having to use a mouse at all if
> possible, but anyway.
> 
> Stay well
> 
> Jacob Kruger
> Blind Biker
> Skype: BlindZA
> "Roger Wilco wants to welcome you...to the space janitor's closet..."
> 
> - Original Message - From: "Eric S. Johansson" 
> To: 
> Sent: Friday, February 20, 2015 7:22 PM
> Subject: Re: Accessible tools
> 
> 
>> 
>> On 2/19/2015 10:33 AM, Bryan Duarte wrote:
>>> Thank you jwi, and Jacob,
>>> 
>>> I took a look at that posting and it seems pretty unique. I am not much 
>>> interested in the speech driven development, but I am very interested in 
>>> developing an accessible IDE.
>> 
>> Well you should be because it looks like an aural interface (uses speech 
>> instead of keyboards) uses the same kinds of data to present to either a 
>> text to speech or speech recognition driven environment.
>>> A professor and I have been throwing around the idea of developing a 
>>> completely text based IDE. There are a lot of reasons this could be 
>>> beneficial to a blind developer and maybe even some sighted developers who 
>>> are comfortable in the terminal. The idea would be really just to provide a 
>>> way of easily navigating blocks of code using some kind of tabular 
>>> formatting, and being able to collapse blocks of code and hearing from a 
>>> high level information about the code within. All tools and features would 
>>> obviously be spoken or output in some kind of audio manor.
>> I've been working with another professor working on some of these issues as 
>> well. His focus has been mostly blind young adults in India.  come up with 
>> some pretty cool concepts that looks very usable. The challenge now is to 
>> make them work and, quite frankly monetize the effort to pay for the 
>> development.
>> 
>> Again, this shows the similarities in functionality used by both speech 
>> recognition and text-to-speech. All I care about is text and what I can say. 
>> We're now working with constructs such as with-open, argument by number, 
>> plaintext symbol names (with bidirectional transform to and from code form), 
>> guided construct generation for things like classes, methods, comprehensions 
>> etc.
>> 
>> All of these things would be useful to handed programmers as well as a way 
>> of accelerating co-creation and editing. Unfortunately, like with disabled 
>> people stove piping text-to-speech versus speech recognition, handed 
>> developers stovepipe keyboard interfaces and don't really think about what 
>> they are trying to do, only how they are doing it.
>> 
>> Yes yes, it's a broadbrush that you can probably slap me with. :-)
>>> 
>>> Oh and before I forget does anyone know how to contact Eric who was 
>>> developing that accessible speech driven IDE? Thanks
>> 
>> Well, you could try looking in a mirror and speaking my name three times at 
>> midnight But you would g

Re: Best practice: Sharing object between different objects

2015-02-21 Thread Paul Rubin
pfranke...@gmail.com writes:
> ADC, DAC components. As I said, I would like to derive the
> corresponding classes from one common class, let's say I2CDevice, so
> that they can share the same bus connection (I don't want to do a
> import smbus, ..., self.bus = smbus.SMBus(1) all the time). 

I don't really understand the question but it sounds like you might
want a class variable:

import smbus

class I2CDevice
   bus = smbus.SMBus(1)
   def __init__ (self): ...

This puts the "bus" object into the I2CDevice class itself.  It is
created when the I2CDevice class is defined.  Methods wanting to access
it can then say

   do_something_with(I2CDevice.bus)

is that what you wanted?
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Marko Rauhamaa
Steven D'Aprano :

> Other than that, I cannot see how calling a function which has *not*
> yet been garbage collected can fail, just because the only reference
> still existing is a weak reference.

Maybe the logic of the receiving object isn't prepared for the callback
anymore after an intervening event.

The problem then, of course, is in the logic and not in the callbacks.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: try pattern for database connection with the close method

2015-02-21 Thread Peter Otten
Ian Kelly wrote:

> On Sat, Feb 21, 2015 at 8:27 AM, Peter Otten <__pete...@web.de> wrote:
>> Ian Kelly wrote:
>>
>>> On Sat, Feb 21, 2015 at 5:22 AM, Mark Lawrence 
>>> wrote:
 try:
 with lite.connect('data.db') as db:
 try:
 db.execute(sql, parms)
 except lite.IntegrityError:
 raise ValueError('invalid data')
 except lite.DatabaseError:
 raise OSError('database file corrupt or not found.')
>>>
>>> This could result in the OSError being misleadingly raised due to some
>>> DatabaseError raised by the execute rather than the connect.
>>
>> The OP probably wants to catch these DatabaseErrors, too. Also, the
>> chance of a misleading traceback has been greatly reduced with the advent
>> of chained exceptions.
>>
> 
> Yes, but the point is that OSError is probably inappropriate in that case.

Perhaps, but the example I gave in my other post:

>>> with open("data.db", "wb") as f:
... f.write(os.urandom(1024)) # put random bytes into data.db

>>> db = sqlite3.connect("data.db")
>>> db.execute("create table foo (bar, baz);")
Traceback (most recent call last):
  File "", line 1, in 
sqlite3.DatabaseError: file is encrypted or is not a database

matches the "database file corrupt" part of the error message provided by 
the OP.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Steven D'Aprano
Frank Millman wrote:

> I tried something similar a while ago, and I did find a gotcha.
> 
> The problem lies in this phrase - "if they are no longer alive, they are
> automatically removed from the WeakSet, preventing me from accidentally
> calling them when they are dead."
> 
> I found that the reference was not removed immediately, but was waiting to
> be garbage collected. During that window, I could call the callback, which
> resulted in an error.

I don't understand how this could possibly work. (Or fail to work, as the
case may be.)

If the callback has been garbage collected, then you cannot call it, because
you don't have any references to it and so cannot refer to it in any way.

If the callback has *not* been garbage collected, then you can safely call
it. You have a reference to the callback, therefore it exists. (If Python
ever garbage collects an object that still has references to it, that would
be a critical bug, and you would likely get some sort of seg fault).

The only thing I can think of is you have a situation where your callback
refers to another object, B, via a weak reference. Once all the regular
strong references to the callback and B are gone, theoretically you could
have a race condition where the callback is waiting to be garbage collected
but B has already been garbage collected. If, in that window, you call the
callback, *and* if the callback fails to correctly check that the weak
reference to B still exists, then you could get a Python exception.

The solution is simple: anytime you have a weak reference, you must always
check it before you use it.

Other than that, I cannot see how calling a function which has *not* yet
been garbage collected can fail, just because the only reference still
existing is a weak reference.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Marko Rauhamaa
Chris Angelico :

> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan  wrote:

>> In order to inform users that certain bits of state have changed, I
>> require them to register a callback with my code. The problem is that
>> when I store these callbacks, it naturally creates a strong reference
>> to the objects, which means that if they are deleted without
>> unregistering themselves first, my code will keep the callbacks
>> alive. Since this could lead to really weird and nasty situations,
>> [...]
>
> No, it's not. I would advise using strong references - if the callback
> is a closure, for instance, you need to hang onto it, because there
> are unlikely to be any other references to it. If I register a
> callback with you, I expect it to be called; I expect, in fact, that
> that *will* keep my object alive.

I use callbacks all the time but haven't had any problems with strong
references.

I am careful to move my objects to a zombie state after they're done so
they can absorb any potential loose callbacks that are lingering in the
system.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Chris Angelico
On Sun, Feb 22, 2015 at 2:45 AM, Cem Karan  wrote:
> OK, so if I'm reading your code correctly, you're breaking the cycle in your 
> object graph by making the GUI the owner of the callback, correct?  No other 
> chunk of code has a reference to the callback, correct?

Correct. The GUI engine ultimately owns everything. Of course, this is
a very simple case (imagine a little notification popup; you don't
care about it, you don't need to know when it's been closed, the only
event on it is "hit Close to destroy the window"), and most usage
would have other complications, but it's not uncommon for me to build
a GUI program that leaves everything owned by the GUI engine.
Everything is done through callbacks. Destroy a window, clean up its
callbacks. The main window will have an "on-deletion" callback that
terminates the program, perhaps. It's pretty straight-forward.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: try pattern for database connection with the close method

2015-02-21 Thread Ian Kelly
On Sat, Feb 21, 2015 at 8:27 AM, Peter Otten <__pete...@web.de> wrote:
> Ian Kelly wrote:
>
>> On Sat, Feb 21, 2015 at 5:22 AM, Mark Lawrence 
>> wrote:
>>> try:
>>> with lite.connect('data.db') as db:
>>> try:
>>> db.execute(sql, parms)
>>> except lite.IntegrityError:
>>> raise ValueError('invalid data')
>>> except lite.DatabaseError:
>>> raise OSError('database file corrupt or not found.')
>>
>> This could result in the OSError being misleadingly raised due to some
>> DatabaseError raised by the execute rather than the connect.
>
> The OP probably wants to catch these DatabaseErrors, too. Also, the chance
> of a misleading traceback has been greatly reduced with the advent of
> chained exceptions.
>

Yes, but the point is that OSError is probably inappropriate in that case.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Cem Karan

On Feb 21, 2015, at 9:36 AM, Chris Angelico  wrote:

> On Sun, Feb 22, 2015 at 1:07 AM, Cem Karan  wrote:
>> I agree about closures; its the only way they could work.  When I was 
>> originally thinking about the library, I was trying to include all types of 
>> callbacks, including closures and callable objects.  The callable objects 
>> may pass themselves, or one of their methods to the library, or may do 
>> something really weird.
>> 
>> Although I just realized that closures may cause another problem.  In my 
>> code, I expect that many different callbacks can be registered for the same 
>> event.  Unregistering means you request to be unregistered for the event. 
>> How do you do that with a closure?  Aren't they anonymous?
>> 
> 
> They're objects, same as any other, so the caller can hang onto a
> reference and then say "now remove this one". Simple example:
> 
> callbacks = []
> def register_callback(f): callbacks.append(f)
> def unregister_callback(f): callbacks.remove(f)
> def do_callbacks():
>for f in callbacks:
>f()
> 
> def make_callback(i):
>def inner():
>print("Callback! %d"%i)
>register_callback(inner)
>return inner
> 
> make_callback(5)
> remove_me = make_callback(6)
> make_callback(7)
> unregister_callback(remove_me)
> do_callbacks()

Yeah, that's pretty much what I thought you'd have to do, which kind of defeats 
the purpose of closures (fire-and-forget things).  BUT it does answer my 
question, so no complaints about it!

So, either you keep a reference to your own closure, which means that the 
library doesn't really need to, or the library keeps hold of it for you, in 
which case you don't have a reasonable way of removing it.

> The other option is for your callback registration to return some kind
> of identifier, which can later be used to unregister the callback.
> This is a good way of avoiding reference cycles (the ID could be a
> simple integer - maybe the length of the list prior to the new
> callback being appended, and then the unregistration process is simply
> "callbacks[id] = None", and you skip the Nones when iterating), and
> even allows you to register the exact same function more than once,
> for what that's worth.

That would work.  In the cases where someone might register & unregister many 
callbacks, you might use UUIDs as keys instead (avoids the ABA problem).

> When I do GUI programming, this is usually how things work. For
> instance, I use GTK2 (though usually with Pike rather than Python),
> and I can connect a signal to a callback function. Any given signal
> could have multiple callbacks attached to it, so it's similar to your
> case. I frequently depend on the GTK engine retaining a reference to
> my function (and thus to any data it requires), as I tend not to hang
> onto any inner objects that don't need retention. Once the parent
> object is destroyed, all its callbacks get dereferenced. Consider this
> simplified form:
> 
> def popup_window():
>w = Window()
># Add layout, info, whatever it takes
>btn = Button("Close")
>w.add(btn) # actually it'd be added to a layout
>btn.signal_connect("clicked", lambda *args: w.destroy())
> 
> The GUI back end will hang onto a reference to the window, because
> it's currently on screen; to the button, because it's attached to the
> window; and to my function, because it's connected to a button signal.
> Then when you click the button, the window gets destroyed, which
> destroys the button, which unregisters all its callbacks. At that
> point, there are no refs to the function, so it can get disposed of.
> That button function was the last external reference to the window,
> and now that it's not on screen, its Python object can also be
> disposed of, as can the button inside. So it'll all clean up fairly
> nicely; as long as the callback gets explicitly deregistered, that's
> the end of everything.

OK, so if I'm reading your code correctly, you're breaking the cycle in your 
object graph by making the GUI the owner of the callback, correct?  No other 
chunk of code has a reference to the callback, correct?  

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: try pattern for database connection with the close method

2015-02-21 Thread Peter Otten
Ian Kelly wrote:

> On Sat, Feb 21, 2015 at 5:22 AM, Mark Lawrence 
> wrote:
>> On 21/02/2015 02:42, Mario Figueiredo wrote:
>>>
>>> Hello all,
>>>
>>> I'm using the following pattern for db access  that requires me to
>>> close the connection as soon as it is not needed:
>>>
>>>  import sqlite3 as lite
>>>
>>>  try:
>>>  db = lite.connect('data.db')
>>>  except lite.DatabaseError:
>>>  raise OSError('database file corrupt or not found.')
>>>  else:
>>>  try:
>>>  with db:
>>>  db.execute(sql, parms)
>>>  except lite.IntegrityError:
>>>  raise ValueError('invalid data')
>>>  finally:
>>>  db.close()
>>>
>>> Since it's a bit verbose, is there a better way?
>>>
>>> Note: The user of this API has the whole database functionality
>>> abstracted away. Hence the exception channeling in the except clauses.
>>>
>>
>> Use your context manager at the outer level.
>>
>> import sqlite3 as lite
>>
>> try:
>> with lite.connect('data.db') as db:
>> try:
>> db.execute(sql, parms)
>> except lite.IntegrityError:
>> raise ValueError('invalid data')
>> except lite.DatabaseError:
>> raise OSError('database file corrupt or not found.')
> 
> This could result in the OSError being misleadingly raised due to some
> DatabaseError raised by the execute rather than the connect.

The OP probably wants to catch these DatabaseErrors, too. Also, the chance 
of a misleading traceback has been greatly reduced with the advent of 
chained exceptions.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: try pattern for database connection with the close method

2015-02-21 Thread Peter Otten
Mario Figueiredo wrote:

> Hello all,
> 
> I'm using the following pattern for db access  that requires me to
> close the connection as soon as it is not needed:
> 
> import sqlite3 as lite
> 
> try:
> db = lite.connect('data.db')
> except lite.DatabaseError:
> raise OSError('database file corrupt or not found.')
> else:
> try:
> with db:
> db.execute(sql, parms)
> except lite.IntegrityError:
> raise ValueError('invalid data')
> finally:
> db.close()
> 
> Since it's a bit verbose, 

Why would you care about a few lines? You don't repeat them, do you? Put the 
code into a function or a context manager and invoke it with

>>> my_execute(sql, parms)

or

>>> with my_db() as db:
... db.execute(sql, parms)


> is there a better way?
> 
> Note: The user of this API has the whole database functionality
> abstracted away. Hence the exception channeling in the except clauses.

db.execute() may trigger other sqlite-related errors including 
DatabaseError:

>>> import sqlite3
>>> db = sqlite3.connect("/dev/full")
>>> db.execute("create table foo (bar, baz);")
Traceback (most recent call last):
  File "", line 1, in 
sqlite3.OperationalError: unable to open database file


>>> import os
>>> with open("data.db", "wb") as f:
... f.write(os.urandom(1024)) # put random bytes into data.db
...   # chances of creating a valid db
...   # left as an exercise ;)
... 
1024
>>> db = sqlite3.connect("data.db")
>>> db.execute("create table foo (bar, baz);")
Traceback (most recent call last):
  File "", line 1, in 
sqlite3.DatabaseError: file is encrypted or is not a database

If you want to catch these, too:

@contextlib.contextmanager
def my_db():
db = None
try:
db = sqlite3.connect("data.db")
with db:
yield db  # db.execute() if you don't buy 
  # into the contextmanager idea
except sqlite3.IntegrityError:
raise ValueError
except sqlite3.DatabaseError:
raise OSError
except sqlite3.Error:
raise WhateverYouNeed
finally:
if db is not None:
db.close()

with my_db() as db:
db.execute("select * from sqlite_master;")


-- 
https://mail.python.org/mailman/listinfo/python-list


Re: try pattern for database connection with the close method

2015-02-21 Thread Ian Kelly
On Sat, Feb 21, 2015 at 5:22 AM, Mark Lawrence  wrote:
> On 21/02/2015 02:42, Mario Figueiredo wrote:
>>
>> Hello all,
>>
>> I'm using the following pattern for db access  that requires me to
>> close the connection as soon as it is not needed:
>>
>>  import sqlite3 as lite
>>
>>  try:
>>  db = lite.connect('data.db')
>>  except lite.DatabaseError:
>>  raise OSError('database file corrupt or not found.')
>>  else:
>>  try:
>>  with db:
>>  db.execute(sql, parms)
>>  except lite.IntegrityError:
>>  raise ValueError('invalid data')
>>  finally:
>>  db.close()
>>
>> Since it's a bit verbose, is there a better way?
>>
>> Note: The user of this API has the whole database functionality
>> abstracted away. Hence the exception channeling in the except clauses.
>>
>
> Use your context manager at the outer level.
>
> import sqlite3 as lite
>
> try:
> with lite.connect('data.db') as db:
> try:
> db.execute(sql, parms)
> except lite.IntegrityError:
> raise ValueError('invalid data')
> except lite.DatabaseError:
> raise OSError('database file corrupt or not found.')

This could result in the OSError being misleadingly raised due to some
DatabaseError raised by the execute rather than the connect.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python path on windows

2015-02-21 Thread Chris Angelico
On Sun, Feb 22, 2015 at 1:10 AM, Dave Angel  wrote:
> But you could always write your own py.exe, which interprets the shebang
> differently.
>
> Or run python.exe yourself, which as far as I know, pays no attention to
> shebang lines.

Or, my preferred solution: Fix the shebangs :) It's usually not hard
to put together a script that zips through your hard drive and fixed
them... or else you just fix 'em when you notice 'em, which means your
shebangs become a kind of historic marker. I used to have drive
letters all over my code, and I could track a program's history
through them: this used to be on D:, then it moved to E:, etc.
Discovering something that still said "D:\eddie" was like digging up a
Second World War bullet casing.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Chris Angelico
On Sun, Feb 22, 2015 at 1:07 AM, Cem Karan  wrote:
> I agree about closures; its the only way they could work.  When I was 
> originally thinking about the library, I was trying to include all types of 
> callbacks, including closures and callable objects.  The callable objects may 
> pass themselves, or one of their methods to the library, or may do something 
> really weird.
>
> Although I just realized that closures may cause another problem.  In my 
> code, I expect that many different callbacks can be registered for the same 
> event.  Unregistering means you request to be unregistered for the event. How 
> do you do that with a closure?  Aren't they anonymous?
>

They're objects, same as any other, so the caller can hang onto a
reference and then say "now remove this one". Simple example:

callbacks = []
def register_callback(f): callbacks.append(f)
def unregister_callback(f): callbacks.remove(f)
def do_callbacks():
for f in callbacks:
f()

def make_callback(i):
def inner():
print("Callback! %d"%i)
register_callback(inner)
return inner

make_callback(5)
remove_me = make_callback(6)
make_callback(7)
unregister_callback(remove_me)
do_callbacks()

The other option is for your callback registration to return some kind
of identifier, which can later be used to unregister the callback.
This is a good way of avoiding reference cycles (the ID could be a
simple integer - maybe the length of the list prior to the new
callback being appended, and then the unregistration process is simply
"callbacks[id] = None", and you skip the Nones when iterating), and
even allows you to register the exact same function more than once,
for what that's worth.

When I do GUI programming, this is usually how things work. For
instance, I use GTK2 (though usually with Pike rather than Python),
and I can connect a signal to a callback function. Any given signal
could have multiple callbacks attached to it, so it's similar to your
case. I frequently depend on the GTK engine retaining a reference to
my function (and thus to any data it requires), as I tend not to hang
onto any inner objects that don't need retention. Once the parent
object is destroyed, all its callbacks get dereferenced. Consider this
simplified form:

def popup_window():
w = Window()
# Add layout, info, whatever it takes
btn = Button("Close")
w.add(btn) # actually it'd be added to a layout
btn.signal_connect("clicked", lambda *args: w.destroy())

The GUI back end will hang onto a reference to the window, because
it's currently on screen; to the button, because it's attached to the
window; and to my function, because it's connected to a button signal.
Then when you click the button, the window gets destroyed, which
destroys the button, which unregisters all its callbacks. At that
point, there are no refs to the function, so it can get disposed of.
That button function was the last external reference to the window,
and now that it's not on screen, its Python object can also be
disposed of, as can the button inside. So it'll all clean up fairly
nicely; as long as the callback gets explicitly deregistered, that's
the end of everything.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Best practice: Sharing object between different objects

2015-02-21 Thread Dave Angel

On 02/21/2015 07:15 AM, pfranke...@gmail.com wrote:

Hello!

I have a best-practice question: Imagine I have several hardware devices that I 
work with on the same I2C bus and I am using the python smbus module for that 
purpose. The individual devices are sensors, ADC, DAC components. As I said, I 
would like to derive the corresponding classes from one common class, let's say 
I2CDevice, so that they can share the same bus connection (I don't want to do a 
import smbus, ..., self.bus = smbus.SMBus(1) all the time). Of course, I could 
just pass the the bus reference to each instance, but I am pretty sure that 
there must be a nicer way to do this.

In particular, I imagine the following: It should be possible that I have two 
classes, ADC_I2C, DAC_I2C which share the same base class. Once I create an 
instance of ADC_I2C or DAC_I2C it should check whether a bus object exists, if 
not, it should create one and the other class should be able to use this bus 
reference as well. Do you get my point? I am pretty sure that such a design 
pattern should exist, maybe also in the reference of DB connections? 
Unfortunately I did not find something like this.



I think you should write the code (or at least a skeleton version).  I 
suspect the answer will become obvious to you, once you're not worrying 
about design patterns, or being java compatible.


Regardless of whether you inherit from a common base class, both classes 
can have an attribute with a "bus object" in it.  The question becomes 
who initializes it, and who decides to reuse the "same one".  Your 
wording doesn't make that the least bit clear to me;  maybe your sample 
code will.


Many times when you'd have a common base class in Java or C++, you just 
have to have a common member or two in Python.


On the other hand, once you start implementing, maybe it'll become 
obvious that there's enough shared code for a common base class.


The first question is whether an ADC "is-a" bus, or "has-a" bus.  Stated 
that way it sounds like you should have an attribute in ADC which is a 
bus object.



Any hints are highly appreciated!

Thanks!




--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Devin Jeanpierre
On Fri, Feb 20, 2015 at 9:42 PM, Chris Angelico  wrote:
> No, it's not. I would advise using strong references - if the callback
> is a closure, for instance, you need to hang onto it, because there
> are unlikely to be any other references to it. If I register a
> callback with you, I expect it to be called; I expect, in fact, that
> that *will* keep my object alive.

For that matter, if the callback is a method, you need to hang onto
it, because method wrappers are generated on demand, so the method
would be removed from the valid callbacks instantly.

Weak references for callbacks are broken.

-- Devin
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Cem Karan

On Feb 21, 2015, at 8:37 AM, Mark Lawrence  wrote:

> On 21/02/2015 05:41, Frank Millman wrote:
>> 
>> "Cem Karan"  wrote in message
>> news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com...
>>> Hi all, I'm working on a project that will involve the use of callbacks,
>>> and I want to bounce an idea I had off of everyone to make sure I'm not
>>> developing a bad idea.  Note that this is for python 3.4 code; I don't
>>> need to worry about any version of python earlier than that.
>>> 
>>> In order to inform users that certain bits of state have changed, I
>>> require them to register a callback with my code.  The problem is that
>>> when I store these callbacks, it naturally creates a strong reference to
>>> the objects, which means that if they are deleted without unregistering
>>> themselves first, my code will keep the callbacks alive.  Since this could
>>> lead to really weird and nasty situations, I would like to store all the
>>> callbacks in a WeakSet
>>> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That
>>> way, my code isn't the reason why the objects are kept alive, and if they
>>> are no longer alive, they are automatically removed from the WeakSet,
>>> preventing me from accidentally calling them when they are dead.  My
>>> question is simple; is this a good design?  If not, why not?
>>>   Are there any potential 'gotchas' I should be worried about?
>>> 
>> 
>> I tried something similar a while ago, and I did find a gotcha.
>> 
>> The problem lies in this phrase - "if they are no longer alive, they are
>> automatically removed from the WeakSet, preventing me from accidentally
>> calling them when they are dead."
>> 
>> I found that the reference was not removed immediately, but was waiting to
>> be garbage collected. During that window, I could call the callback, which
>> resulted in an error.
>> 
>> There may have been a simple workaround. Perhaps someone else can comment.
>> 
>> Frank Millman
>> 
> 
> https://docs.python.org/3/library/gc.html has a collect function.  That seems 
> like a simple workaround, but whether or not it classifies as a good solution 
> I'll leave to others, I'm not qualified to say.


Unfortunately, depending on how many objects you have in your object graph, it 
can slow your code down a fair amount.  I think Frank is right about how a 
WeakSet might be a bad idea in this case.  You really need to know if an object 
is alive or dead, and not some indeterminate state.

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python path on windows

2015-02-21 Thread Dave Angel

On 02/21/2015 06:05 AM, Gisle Vanem wrote:

Dave Angel wrote:


Finally, when py.exe starts, it reads that first (shebang) line, and
decides which python interpreter to actually use.


py.exe? Do you mean python.exe?


Reread my post, or read Mark's reply to yours.  The job of py.exe or 
pyw.exe is to examine the shebang line and then exec the appropriate 
python.exe




Is there a way to make python.exe ignore all Shebang lines
in all scripts? I had many generated .py-files with:
   #!g:\ProgramFiler\Python27\python.EXE


That's what symlinks are for, on Unix-like systems.  Somewhere I read 
that Windows has had them for a little while now.  Those generated files 
should have pointed to a symlink on your system drive, rather than 
pointing directly to drive G:


But you could always write your own py.exe, which interprets the shebang 
differently.


Or run python.exe yourself, which as far as I know, pays no attention to 
shebang lines.




After transferring my Python tree to a new Win-8.1 PC, my Python
tree was installed under "f:\ProgramFiler\" and my CD-ROM on "g:\"

This caused those scripts to access my CD-ROM or pop-up a Win-GUI
error dialogue if no CD-ROM was in the drive. Irritating.





--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Cem Karan

On Feb 21, 2015, at 8:15 AM, Chris Angelico  wrote:

> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan  wrote:
>> OK, so it would violate the principle of least surprise for you.  
>> Interesting.  Is this a general pattern in python?  That is, callbacks are 
>> owned by what they are registered with?
>> 
>> In the end, I want to make a library that offers as few surprises to the 
>> user as possible, and no matter how I think about callbacks, they are 
>> surprising to me.  If callbacks are strongly-held, then calling 'del foo' on 
>> a callable object may not make it go away, which can lead to weird and nasty 
>> situations.  Weakly-held callbacks mean that I (as the programmer), know 
>> that objects will go away after the next garbage collection (see Frank's 
>> earlier message), so I don't get 'dead' callbacks coming back from the grave 
>> to haunt me.
>> 
>> So, what's the consensus on the list, strongly-held callbacks, or 
>> weakly-held ones?
> 
> I don't know about Python specifically, but it's certainly a general
> pattern in other languages. They most definitely are owned, and it's
> the only model that makes sense when you use closures (which won't
> have any other references anywhere).

I agree about closures; its the only way they could work.  When I was 
originally thinking about the library, I was trying to include all types of 
callbacks, including closures and callable objects.  The callable objects may 
pass themselves, or one of their methods to the library, or may do something 
really weird.  

Although I just realized that closures may cause another problem.  In my code, 
I expect that many different callbacks can be registered for the same event.  
Unregistering means you request to be unregistered for the event. How do you do 
that with a closure?  Aren't they anonymous?

> If you're expecting 'del foo' to destroy the object, then you have a
> bigger problem than callbacks, because that's simply not how Python
> works. You can't _ever_ assume that deleting something from your local
> namespace will destroy the object, because there can always be more
> references. So maybe you need a more clear way of saying "I'm done
> with this, get rid of it".

Agreed about 'del', and I don't assume that the object goes away at the point.  
The problem is debugging and determining WHY your object is still around.  I 
know a combination of logging and gc.get_referrers() will probably help you 
figure out why something is still around, but I'm trying to avoid that 
headache.  

I guess the real problem is how this creates cycles in the call graph.  User 
code effectively owns the library code, which via callbacks owns the user code. 
 I have no idea what the best point the cycle is to break it, and not surprise 
someone down the road.  The only idea I have is to redesign the library a 
little, and make anything that accepts a callback actually be a subclass of 
collections.abc.Container, or even collections.abc.MutableSet.  That makes it 
very obvious that the object owns the callback, and that you will need to 
remove your object to unregister it.  The only problem is how to handle 
closures; since they are anonymous, how do you decide which one to remove?

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: pypandoc and restructured text

2015-02-21 Thread Chris Angelico
On Sun, Feb 22, 2015 at 12:39 AM, alb  wrote:
> Hi Wolfgang,
>
> Wolfgang Maier  wrote:
> []
>> I have pandoc 1.12.2.1 and it recognizes the figure directive just fine
>> (tested with html output so I cannot say anything about LaTeX).
>
> This reminds me that I need to move sooner or later from squeeze to
> wheezy...
>
> Do you know of anyway to install wheezy packages on squeeze? On
> backports there's nothing more than what comes with the squeeze sources.
>
> Thanks a lot for the hint. Maybe I should seriously think about
> upgrading the whole distro. It's just that Gnome3 really sucks to my
> taste and I'm not in the mood to look for another Desktop
> Environment...(maybe I should go back to CDE).
>

This is a bit off-topic for python-list, being more about Debian than
the language, but trying to install just one Wheezy package on Squeeze
is probably a bad idea; you'll probably find that it's built against a
different version of libc and such, so you'd need to upgrade pretty
much your whole system.

Your two best options are, IMO:

1) Install pandoc from source, which will then build it against your
existing libraries. This may not work (if it depends on a newer
version of something), but it should tell you if that's the case. You
won't disturb your system in any way.

2) Upgrade the entire OS to either Wheezy or Jessie. Personally, I'm
quite happy with Jessie, even though she's not officially stable yet
(and have you _seen_ her? Definitely mentally unstable); but whether
you go Wheezy or Jessie, grab Xfce rather than Gnome 3. The latest
Jessie installers are giving an option on installation as to which
desktop environment(s) to install, but if you do it as an upgrade, you
may want to explicitly uninstall Gnome and install Xfce, either before
or after the upgrade.

Happily using a mix of Debian versions here, but all with Xfce.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: pypandoc and restructured text

2015-02-21 Thread Fabien

On 21.02.2015 14:39, alb wrote:

Do you know of anyway to install wheezy packages on squeeze?


No need for a new distro. Use virtualenv, this is simple and great:

http://simononsoftware.com/virtualenv-tutorial-part-2/

Cheers,

Fabien
--
https://mail.python.org/mailman/listinfo/python-list


Re: pypandoc and restructured text

2015-02-21 Thread alb
Hi Wolfgang,

Wolfgang Maier  wrote:
[]
> I have pandoc 1.12.2.1 and it recognizes the figure directive just fine 
> (tested with html output so I cannot say anything about LaTeX).

This reminds me that I need to move sooner or later from squeeze to 
wheezy...

Do you know of anyway to install wheezy packages on squeeze? On 
backports there's nothing more than what comes with the squeeze sources.

> I've found an old forum post from 2011:
> 
> https://groups.google.com/forum/#!topic/pandoc-discuss/OmGYDycaMjs
> 
> confirming that the figure directive was not supported at that time.
> So, yes, I think upgrading pandoc could help.

Thanks a lot for the hint. Maybe I should seriously think about 
upgrading the whole distro. It's just that Gnome3 really sucks to my 
taste and I'm not in the mood to look for another Desktop 
Environment...(maybe I should go back to CDE).

Al
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Mark Lawrence

On 21/02/2015 05:41, Frank Millman wrote:


"Cem Karan"  wrote in message
news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com...

Hi all, I'm working on a project that will involve the use of callbacks,
and I want to bounce an idea I had off of everyone to make sure I'm not
developing a bad idea.  Note that this is for python 3.4 code; I don't
need to worry about any version of python earlier than that.

In order to inform users that certain bits of state have changed, I
require them to register a callback with my code.  The problem is that
when I store these callbacks, it naturally creates a strong reference to
the objects, which means that if they are deleted without unregistering
themselves first, my code will keep the callbacks alive.  Since this could
lead to really weird and nasty situations, I would like to store all the
callbacks in a WeakSet
(https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That
way, my code isn't the reason why the objects are kept alive, and if they
are no longer alive, they are automatically removed from the WeakSet,
preventing me from accidentally calling them when they are dead.  My
question is simple; is this a good design?  If not, why not?
   Are there any potential 'gotchas' I should be worried about?



I tried something similar a while ago, and I did find a gotcha.

The problem lies in this phrase - "if they are no longer alive, they are
automatically removed from the WeakSet, preventing me from accidentally
calling them when they are dead."

I found that the reference was not removed immediately, but was waiting to
be garbage collected. During that window, I could call the callback, which
resulted in an error.

There may have been a simple workaround. Perhaps someone else can comment.

Frank Millman



https://docs.python.org/3/library/gc.html has a collect function.  That 
seems like a simple workaround, but whether or not it classifies as a 
good solution I'll leave to others, I'm not qualified to say.



--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Cem Karan

On Feb 21, 2015, at 12:41 AM, Frank Millman  wrote:

> 
> "Cem Karan"  wrote in message 
> news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com...
>> Hi all, I'm working on a project that will involve the use of callbacks, 
>> and I want to bounce an idea I had off of everyone to make sure I'm not 
>> developing a bad idea.  Note that this is for python 3.4 code; I don't 
>> need to worry about any version of python earlier than that.
>> 
>> In order to inform users that certain bits of state have changed, I 
>> require them to register a callback with my code.  The problem is that 
>> when I store these callbacks, it naturally creates a strong reference to 
>> the objects, which means that if they are deleted without unregistering 
>> themselves first, my code will keep the callbacks alive.  Since this could 
>> lead to really weird and nasty situations, I would like to store all the 
>> callbacks in a WeakSet 
>> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That 
>> way, my code isn't the reason why the objects are kept alive, and if they 
>> are no longer alive, they are automatically removed from the WeakSet, 
>> preventing me from accidentally calling them when they are dead.  My 
>> question is simple; is this a good design?  If not, why not?
>>  Are there any potential 'gotchas' I should be worried about?
>> 
> 
> I tried something similar a while ago, and I did find a gotcha.
> 
> The problem lies in this phrase - "if they are no longer alive, they are 
> automatically removed from the WeakSet, preventing me from accidentally 
> calling them when they are dead."
> 
> I found that the reference was not removed immediately, but was waiting to 
> be garbage collected. During that window, I could call the callback, which 
> resulted in an error.
> 
> There may have been a simple workaround. Perhaps someone else can comment.

THAT would be one heck of a gotcha!  Must have been fun debugging that one!

Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Chris Angelico
On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan  wrote:
> OK, so it would violate the principle of least surprise for you.  
> Interesting.  Is this a general pattern in python?  That is, callbacks are 
> owned by what they are registered with?
>
> In the end, I want to make a library that offers as few surprises to the user 
> as possible, and no matter how I think about callbacks, they are surprising 
> to me.  If callbacks are strongly-held, then calling 'del foo' on a callable 
> object may not make it go away, which can lead to weird and nasty situations. 
>  Weakly-held callbacks mean that I (as the programmer), know that objects 
> will go away after the next garbage collection (see Frank's earlier message), 
> so I don't get 'dead' callbacks coming back from the grave to haunt me.
>
> So, what's the consensus on the list, strongly-held callbacks, or weakly-held 
> ones?

I don't know about Python specifically, but it's certainly a general
pattern in other languages. They most definitely are owned, and it's
the only model that makes sense when you use closures (which won't
have any other references anywhere).

If you're expecting 'del foo' to destroy the object, then you have a
bigger problem than callbacks, because that's simply not how Python
works. You can't _ever_ assume that deleting something from your local
namespace will destroy the object, because there can always be more
references. So maybe you need a more clear way of saying "I'm done
with this, get rid of it".

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Design thought for callbacks

2015-02-21 Thread Cem Karan

On Feb 21, 2015, at 12:42 AM, Chris Angelico  wrote:

> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan  wrote:
>> In order to inform users that certain bits of state have changed, I require 
>> them to register a callback with my code.  The problem is that when I store 
>> these callbacks, it naturally creates a strong reference to the objects, 
>> which means that if they are deleted without unregistering themselves first, 
>> my code will keep the callbacks alive.  Since this could lead to really 
>> weird and nasty situations, I would like to store all the callbacks in a 
>> WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  
>> That way, my code isn't the reason why the objects are kept alive, and if 
>> they are no longer alive, they are automatically removed from the WeakSet, 
>> preventing me from accidentally calling them when they are dead.  My 
>> question is simple; is this a good design?  If not, why not?  Are there any 
>> potential 'gotchas' I should be worried about?
>> 
> 
> No, it's not. I would advise using strong references - if the callback
> is a closure, for instance, you need to hang onto it, because there
> are unlikely to be any other references to it. If I register a
> callback with you, I expect it to be called; I expect, in fact, that
> that *will* keep my object alive.

OK, so it would violate the principle of least surprise for you.  Interesting.  
Is this a general pattern in python?  That is, callbacks are owned by what they 
are registered with?

In the end, I want to make a library that offers as few surprises to the user 
as possible, and no matter how I think about callbacks, they are surprising to 
me.  If callbacks are strongly-held, then calling 'del foo' on a callable 
object may not make it go away, which can lead to weird and nasty situations.  
Weakly-held callbacks mean that I (as the programmer), know that objects will 
go away after the next garbage collection (see Frank's earlier message), so I 
don't get 'dead' callbacks coming back from the grave to haunt me.

So, what's the consensus on the list, strongly-held callbacks, or weakly-held 
ones?

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python path on windows

2015-02-21 Thread Mark Lawrence

On 21/02/2015 11:05, Gisle Vanem wrote:

Dave Angel wrote:


Finally, when py.exe starts, it reads that first (shebang) line, and
decides which python interpreter to actually use.


py.exe? Do you mean python.exe?



py.exe or pyw.exe come with the Python launcher on Windows and work out 
which version of the interpreter to use.


https://www.python.org/dev/peps/pep-0397/
https://docs.python.org/3/using/windows.html#python-launcher-for-windows

I think it was first installed with Python 3.3.  It's available here 
https://bitbucket.org/vinay.sajip/pylauncher if you're on an older version.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list


Best practice: Sharing object between different objects

2015-02-21 Thread pfranken85
Hello!

I have a best-practice question: Imagine I have several hardware devices that I 
work with on the same I2C bus and I am using the python smbus module for that 
purpose. The individual devices are sensors, ADC, DAC components. As I said, I 
would like to derive the corresponding classes from one common class, let's say 
I2CDevice, so that they can share the same bus connection (I don't want to do a 
import smbus, ..., self.bus = smbus.SMBus(1) all the time). Of course, I could 
just pass the the bus reference to each instance, but I am pretty sure that 
there must be a nicer way to do this.

In particular, I imagine the following: It should be possible that I have two 
classes, ADC_I2C, DAC_I2C which share the same base class. Once I create an 
instance of ADC_I2C or DAC_I2C it should check whether a bus object exists, if 
not, it should create one and the other class should be able to use this bus 
reference as well. Do you get my point? I am pretty sure that such a design 
pattern should exist, maybe also in the reference of DB connections? 
Unfortunately I did not find something like this.

Any hints are highly appreciated!

Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: try pattern for database connection with the close method

2015-02-21 Thread Mark Lawrence

On 21/02/2015 02:42, Mario Figueiredo wrote:

Hello all,

I'm using the following pattern for db access  that requires me to
close the connection as soon as it is not needed:

 import sqlite3 as lite

 try:
 db = lite.connect('data.db')
 except lite.DatabaseError:
 raise OSError('database file corrupt or not found.')
 else:
 try:
 with db:
 db.execute(sql, parms)
 except lite.IntegrityError:
 raise ValueError('invalid data')
 finally:
 db.close()

Since it's a bit verbose, is there a better way?

Note: The user of this API has the whole database functionality
abstracted away. Hence the exception channeling in the except clauses.



Use your context manager at the outer level.

import sqlite3 as lite

try:
with lite.connect('data.db') as db:
try:
db.execute(sql, parms)
except lite.IntegrityError:
raise ValueError('invalid data')
except lite.DatabaseError:
raise OSError('database file corrupt or not found.')

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list


Re: Python path on windows

2015-02-21 Thread Gisle Vanem

Dave Angel wrote:


Finally, when py.exe starts, it reads that first (shebang) line, and decides 
which python interpreter to actually use.


py.exe? Do you mean python.exe?

Is there a way to make python.exe ignore all Shebang lines
in all scripts? I had many generated .py-files with:
  #!g:\ProgramFiler\Python27\python.EXE

After transferring my Python tree to a new Win-8.1 PC, my Python
tree was installed under "f:\ProgramFiler\" and my CD-ROM on "g:\"

This caused those scripts to access my CD-ROM or pop-up a Win-GUI
error dialogue if no CD-ROM was in the drive. Irritating.


--
--gv
--
https://mail.python.org/mailman/listinfo/python-list


Re: 'Lite' Databases (Re: sqlite3 and dates)

2015-02-21 Thread Sibylle Koczian

Am 20.02.2015 um 15:16 schrieb Dennis Lee Bieber:


The middle-ground is probably something like the embedded version of
Firebird (pity there has been no updated book -- "The Firebird Book" came
out in 2004, v1.5 while 2.5 is current [Whoops, looks like there /is/ an
update, print-on-demand in three overpriced volumes])

The second edition as PDF EBook is part of the IBPhoenix DVD (Developer 
Edition $100). But I can't find out from the website what that DVD 
contains in its current edition. It used to have Firebird itself and 
quite a lot of additional software and documentation.


Firebird embedded has no network access and multi-user capabilities at 
all if I read the documentation correctly 
(http://www.firebirdsql.org/manual/ufb-cs-embedded.html). But "A 
Firebird embedded server DLL can also be used as a network client" to a 
regular Firebird server (from the same page). On the other hand triggers 
and stored procedures should work exactly like the client/server 
versions (and the procedure language is pretty similar to PostgreSQL, so 
migration between these two isn't too difficult).


So this is a middle ground with other pros and cons than SQLite versus 
PostgreSQL.



--
https://mail.python.org/mailman/listinfo/python-list


Re: 'Lite' Databases (Re: sqlite3 and dates)

2015-02-21 Thread Paul Rubin
Ned Deily  writes:
>> Same reason lots of people have forked Postgres.  Or you might just want
>> to customize it.
> Well, for whatever reason one might have, one can: it's public domain 
> software.

Yes, but unlike with most FOSS software, your version has much lower
quality assurance than the "official" version because you don't have the
big test suite.  Even if you don't fork or change the code at all, but
you just port it to a new platform or compiler, you really should run
the full set of tests, but you can't.

> It seems like [the SQLite consortium] was an approach Richard Hipp
> and major users of SQLite took to ensure a sustaining funding model
> for the project while ensuring its independence.

The main benefit of the consortium seems to be very close an intense
support from the SQLite core developers, including customization and
porting services.  Consortium membership apparently starts at $75K a
year so I doubt anyone joins just to get the test suite.  There is some
mention of separate licenses just for the test suite but I didn't see a
price tag and I wonder if that generates significant revenue compared to
the consortium.

> given the immense good that the SQLite project has done for so many
> other projects over the years.

Maybe I should look into it more.  I've never particularly felt the need
for it since I've either used client/server databases or else simpler
embedded databases like bsddb or even flat files.  I actually think
Macid/Happstack-state is brilliant but I haven't used it yet.

> Many other less important projects have foundered for lack of
> sustained funding.

Other FOSS databases like Postgres, MySQL, MongoDB, Riak, Cassandra,
etc. all seem to be doing fine.  GNAT Ada (GCC-based Ada compiler and
surrounding ecosystem) is split into free and proprietary components and
you have to pay pretty big to get the proprietary parts, but the free
parts aren't hobbled in any way (like missing tests) AFAIK.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: 'Lite' Databases (Re: sqlite3 and dates)

2015-02-21 Thread Ned Deily
In article <871tljepea@jester.gateway.pace.com>,
 Paul Rubin  wrote:

> Ned Deily  writes:
> > (though I don't know why anyone would want to fork it).  
> 
> Same reason lots of people have forked Postgres.  Or you might just want
> to customize it.

Well, for whatever reason one might have, one can: it's public domain 
software.

> > I imagine that is done as an incentive to help
> > finance the on-going development and maintenance of SQLite. 
> It's a pretty unusual and annoying trick that other projects have not
> felt they had to resort to.
> 
> > https://www.sqlite.org/testing.html
> 
> Thanks, that's the page I remember.  The TH3 test suite is the
> interesting one and you can't get it without paying a lot of $.  I
> guess there is some semantic quibble possible about whether you pay for
> the test suite, or (as they put it) pay for Consortium membership and
> then (as a member) get the test suite for free.

I don't have any special knowledge of the history or current status of 
the Consortium but it's not difficult to find information and blog posts 
about it.  It seems like this was an approach Richard Hipp and major 
users of SQLite took to ensure a sustaining funding model for the 
project while ensuring its independence.  It strikes me as a very 
reasonable and modest constraint given the immense good that the SQLite 
project has done for so many other projects over the years.  Many other 
less important projects have foundered for lack of sustained funding.

https://www.sqlite.org/consortium.html
https://blog.lizardwrangler.com/2008/02/27/the-sqlite-consortium/
 
> Do you know the situation with the SQL Logic Test (SLT) also mentioned
> on that page?

No

-- 
 Ned Deily,
 n...@acm.org

-- 
https://mail.python.org/mailman/listinfo/python-list