Re: testing code

[Jim Lee]

On 07/07/18 21:21, Sharan Basappa wrote:


sorry. there was a copy paste error when i posted. I pasted test_2.py for both 
the files:

here are the files again. The issue remains.
[...]

output:
%run "D:/Projects/Initiatives/machine learning/programs/test_2_test.py"
30


[11:24 PM jlee@kerndev ~] $cat test_2.py
x = 10
y = 20

c = x-y

print c

def func1():
    return x+y

[11:24 PM jlee@kerndev ~] $cat test_2_test.py
import test_2

x = test_2.func1()
print x
[11:24 PM jlee@kerndev ~] $python test_2_test.py
-10
30
[11:24 PM jlee@kerndev ~] $


As you can see, the  code from the import is indeed executed by the 
python interpreter.


I'm not familiar with the "%run" prefix in your command - some sort of 
windows-ism?


-Jim


--
https://mail.python.org/mailman/listinfo/python-list

Re: testing code

[Sharan Basappa]

On Saturday, 7 July 2018 18:22:23 UTC+5:30, Chris Angelico  wrote:
> On Sat, Jul 7, 2018 at 10:02 PM, Sharan Basappa
>  wrote:
> > On Friday, 6 July 2018 09:22:31 UTC+5:30, Chris Angelico  wrote:
> >> On Fri, Jul 6, 2018 at 12:56 PM, Sharan Basappa
> >>  wrote:
> >> > Please let me know if the following understanding of mine is correct.
> >> > I need to put the program code in a separate file and organize every 
> >> > executable code in some form of function. If any code exists outside of 
> >> > function then it is not executable by importing.
> >> >
> >>
> >> Kinda. It's actually the other way around: if any code exists outside
> >> of functions, it will be executed immediately when you import. So
> >> you're correct in that it would be hard (maybe impossible) to
> >> unit-test that; and yes, the normal way to do it is to put all your
> >> important code into functions.
> >>
> >> ChrisA
> >
> > Chris,
> >
> > Things do work as per expected with one exception.
> > You mentioned that as soon as a file is imported, it executes immediately.
> >
> > Please see the example below:
> >
> > file: test_2.py
> >
> > x = 10
> > y = 20
> >
> > c = x-y
> >
> > print c
> >
> > def func1():
> > return x+y
> >
> > test_2_test.py
> >
> > x = 10
> > y = 20
> >
> > c = x-y
> >
> > print c
> >
> > def func1():
> > return x+y
> >
> > this is the output:
> > %run "D:/Projects/Initiatives/machine learning/programs/test_2_test.py"
> > 30
> >
> > As you can see, print c in test_2 file was not executed upon import as 
> > there is no corresponding output
> 
> I don't think you import your other module anywhere.
> 
> ChrisA

sorry. there was a copy paste error when i posted. I pasted test_2.py for both 
the files:

here are the files again. The issue remains.

test_2.py:
x = 10
y = 20

c = x-y

print c

def func1():
return x+y 

test_2_test.py:
import test_2

x = test_2.func1()
print x

output:
%run "D:/Projects/Initiatives/machine learning/programs/test_2_test.py"
30
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Chris Angelico]

On Sun, Jul 8, 2018 at 1:58 PM, Steven D'Aprano
 wrote:
> On Sun, 08 Jul 2018 12:23:41 +1000, Chris Angelico wrote:
>
>>> Some people, when confronted with a problem, say, "I know, I'll use
>>> threads". Nothhtwo probw ey ave lems.
>>
>> Right. Now they have to deal with interleaving, but that's all. And
>> honestly, MOST CODE wouldn't notice interleaving; it's only when you
>> change (either by rebinding or by mutating) something that can be seen
>> by multiple threads. Which basically means "mutable globals are a risk,
>> pretty much everything else is safe".
>
> Its not just globals though. Any mutable object shared between two
> threads is potentially a risk. Globals are just the most obvious example.
>

Right; by "basically" I mean "yes there are others but this is what
matters". The technically-accurate part is in the previous sentence:
"can be seen by multiple threads".

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Steven D'Aprano]

On Sun, 08 Jul 2018 12:23:41 +1000, Chris Angelico wrote:

>> Some people, when confronted with a problem, say, "I know, I'll use
>> threads". Nothhtwo probw ey ave lems.
> 
> Right. Now they have to deal with interleaving, but that's all. And
> honestly, MOST CODE wouldn't notice interleaving; it's only when you
> change (either by rebinding or by mutating) something that can be seen
> by multiple threads. Which basically means "mutable globals are a risk,
> pretty much everything else is safe".

Its not just globals though. Any mutable object shared between two 
threads is potentially a risk. Globals are just the most obvious example.



-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generate a static back-of-a-book style index?

[Cameron Simpson]

On 07Jul2018 21:57, Tim Chase  wrote:

On 2018-07-08 12:12, Cameron Simpson wrote:

On 07Jul2018 20:11, Skip Montanaro  wrote:
>> Have you looked at the ptx command? Might be called "gptx"
>
>Thanks, Cameron. I was unaware of it. Will check it out.

BTW, it well predates the GNU coreutils; I used it on V7 UNIX.


Interesting.  Despite your V7-provenance claim, it doesn't seem to
have persisted into either the FreeBSD boxes or the OpenBSD boxes I
have at hand.  Bringing in anything that involves GNU coreutils adds
it to the machine in question, but just kinda odd that those with a
tighter tie to "real" Unix would have dropped it.


It's associated with the troff stuff generally. Maybe that's not installed?


Off to go read `man gptx` now...


Or "man ptx", depends on your system and its install.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Matplotlib 3D limitations, please recommend alternative

[John Ladasky]

On Saturday, July 7, 2018 at 6:36:16 AM UTC-7, Rick Johnson wrote:
> John Ladasky wrote:
> 
> > Back then I wrote:
> > 
> > "I have concluded that Qt, PyQt, and OpenGL are all
> > rapidly-evolving, and huge, software packages.  There may
> > be compatibility problems, and relevant examples with the
> > right software combination may be hard to find.  Two weeks
> > of searching web pages hasn't turned up a single example
> > which demonstrates PyQt5 doing something simple in 3D with
> > OpenGL that I can study."
> 
> PyQT is not the only GUI toolkit with OpenGL support, you
> know. I would suggest you have a look at the alternatives.

I'm fully aware of that.  I have tried a few Python GUIs, and I have the most 
experience with PyQt5.  It is not necessary for me to integrate my current 3D 
graphics project with PyQt5, but it would be nice.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generate a static back-of-a-book style index?

[Tim Chase]

On 2018-07-08 12:12, Cameron Simpson wrote:
> On 07Jul2018 20:11, Skip Montanaro  wrote:
> >> Have you looked at the ptx command? Might be called "gptx"  
> >
> >Thanks, Cameron. I was unaware of it. Will check it out.  
> 
> BTW, it well predates the GNU coreutils; I used it on V7 UNIX.

Interesting.  Despite your V7-provenance claim, it doesn't seem to
have persisted into either the FreeBSD boxes or the OpenBSD boxes I
have at hand.  Bringing in anything that involves GNU coreutils adds
it to the machine in question, but just kinda odd that those with a
tighter tie to "real" Unix would have dropped it.

Off to go read `man gptx` now...

-tkc



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Chris Angelico]

On Sun, Jul 8, 2018 at 12:12 PM, Steven D'Aprano
 wrote:
> On Sun, 08 Jul 2018 11:15:17 +1000, Chris Angelico wrote:
>
> [...]
>> Python threads don't switch only between lines of code,
>
> As I understand it, there could be a switch between any two byte codes,
> or maybe only between certain bytes codes. But certain more fine grained
> than just between lines of code.
>
>
>> so the actual
>> interaction is a bit more complicated than you say. In CPython, the
>> increment operation is:
>>
>>   3   0 LOAD_GLOBAL  0 (i)
>>   2 LOAD_CONST   1 (1)
>>   4 INPLACE_ADD
>>   6 STORE_GLOBAL 0 (i)
>>
>> A context switch could happen between any pair of statements.
>
> If you actually mean *statements* as opposed to byte codes, then the only
> place there could be a switch would be either before the LOAD_GLOBAL or
> after the STORE_GLOBAL (given that i is a built-in int and cannot have a
> custom __iadd__ method).
>
> Is that what you mean?

I may be wrong, but I always assume that a context switch could happen
between any two bytecode operations - or, if you're reading the
disassembly, between any two lines *of disassembly*. So there could be
a switch before LOAD_GLOBAL, a switch between that and LOAD_CONST,
another switch before the ADD, another before the STORE, and another
right at the end. Well, there won't be *all* of those, but there could
be any of them.

This might not be entirely correct - there might be pairs that are
functionally atomic - but it's the safe assumption.

>> For instance, if you replace "i
>> += 1" with "i += i", to double the value, you'll get this:
>>
>>   3   0 LOAD_GLOBAL  0 (i)
>>   2 LOAD_GLOBAL  0 (i)
>>   4 INPLACE_ADD
>>   6 STORE_GLOBAL 0 (i)
>>
>> and that could potentially have both of them load the initial value,
>> then one of them runs to completion, and then the other loads the result
>> - so it'll add 1 and 2 and have a result of 3, rather than 2 or 4.
>
> Some people, when confronted with a problem, say, "I know, I'll use
> threads". Nothhtwo probw ey ave lems.

Right. Now they have to deal with interleaving, but that's all. And
honestly, MOST CODE wouldn't notice interleaving; it's only when you
change (either by rebinding or by mutating) something that can be seen
by multiple threads. Which basically means "mutable globals are a
risk, pretty much everything else is safe".

>> But you're absolutely right that there are only a small handful of
>> plausible results, even with threading involved.
>
> Indeed. Even though threading is non-deterministic, it isn't *entirely*
> unconstrained.
>

Yeah. Quite far from it, in fact. Python threading is well-defined and
fairly easy to work with. Only in a handful of operations do you need
to worry about atomicity - like the one that started this thread.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Steven D'Aprano]

On Sun, 08 Jul 2018 11:15:17 +1000, Chris Angelico wrote:

[...]
> Python threads don't switch only between lines of code, 

As I understand it, there could be a switch between any two byte codes, 
or maybe only between certain bytes codes. But certain more fine grained 
than just between lines of code.


> so the actual
> interaction is a bit more complicated than you say. In CPython, the
> increment operation is:
> 
>   3   0 LOAD_GLOBAL  0 (i)
>   2 LOAD_CONST   1 (1)
>   4 INPLACE_ADD
>   6 STORE_GLOBAL 0 (i)
> 
> A context switch could happen between any pair of statements.

If you actually mean *statements* as opposed to byte codes, then the only 
place there could be a switch would be either before the LOAD_GLOBAL or 
after the STORE_GLOBAL (given that i is a built-in int and cannot have a 
custom __iadd__ method).

Is that what you mean?


> In this
> particular example, the end result doesn't change - coherent results are
> 2 and 3, nothing else - but in other situations, there may be times when
> the separate steps might be significant.

Fortunately I wasn't talking about other code snippets, only the one 
shown :-)


> For instance, if you replace "i
> += 1" with "i += i", to double the value, you'll get this:
> 
>   3   0 LOAD_GLOBAL  0 (i)
>   2 LOAD_GLOBAL  0 (i)
>   4 INPLACE_ADD
>   6 STORE_GLOBAL 0 (i)
> 
> and that could potentially have both of them load the initial value,
> then one of them runs to completion, and then the other loads the result
> - so it'll add 1 and 2 and have a result of 3, rather than 2 or 4.

Some people, when confronted with a problem, say, "I know, I'll use 
threads". Nothhtwo probw ey ave lems.



> But you're absolutely right that there are only a small handful of
> plausible results, even with threading involved.

Indeed. Even though threading is non-deterministic, it isn't *entirely* 
unconstrained.



-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generate a static back-of-a-book style index?

[Cameron Simpson]

On 07Jul2018 20:11, Skip Montanaro  wrote:

Have you looked at the ptx command? Might be called "gptx" on a system with the
GNU coreutils installed with "g" prefixes.


Thanks, Cameron. I was unaware of it. Will check it out.


BTW, it well predates the GNU coreutils; I used it on V7 UNIX. Nicely, it 
includes the surrounding words in the index output for context, makes for a far 
more usable index listing. You may need to postprocess the output to fit your 
needs.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Chris Angelico]

On Sun, Jul 8, 2018 at 11:04 AM, Steven D'Aprano
 wrote:
>> The only thing Python should guarantee is that the data structures stay
>> "coherent" under race conditions. In other words, there cannot be a
>> segmentation fault. For example, if two threads executed this code in
>> parallel:
>>
>> global i
>> i = 1
>> i += 1
>>
>> a legal end result could be that i contains the string "impossible".
>
> That wouldn't be coherent. The only coherent results are that i could
> equal either 2 or 3:
>
> Possibility 1:
>
> i = 1   # thread 1
> i += 1  # thread 1
> i = 1   # thread 2
> i += 1  # thread 2
> assert i == 2
>
> Possibility 2:
>
> i = 1   # thread 1
> i = 1   # thread 2
> i += 1  # thread 1
> i += 1  # thread 2
> assert i == 3
>
>
> Executing statements out of order is impossible, even in threaded code.
> Redefining the meaning of the integer literal 1 is impossible (ignoring
> unsupported shenanigans with ctypes). Monkey-patching the int __iadd__
> method is impossible. So there is no coherent way to get a result of
> "impossible" from just adding 1 to 1 in any coherent implementation of
> Python.

Python threads don't switch only between lines of code, so the actual
interaction is a bit more complicated than you say. In CPython, the
increment operation is:

  3   0 LOAD_GLOBAL  0 (i)
  2 LOAD_CONST   1 (1)
  4 INPLACE_ADD
  6 STORE_GLOBAL 0 (i)

A context switch could happen between any pair of statements. In this
particular example, the end result doesn't change - coherent results
are 2 and 3, nothing else - but in other situations, there may be
times when the separate steps might be significant. For instance, if
you replace "i += 1" with "i += i", to double the value, you'll get
this:

  3   0 LOAD_GLOBAL  0 (i)
  2 LOAD_GLOBAL  0 (i)
  4 INPLACE_ADD
  6 STORE_GLOBAL 0 (i)

and that could potentially have both of them load the initial value,
then one of them runs to completion, and then the other loads the
result - so it'll add 1 and 2 and have a result of 3, rather than 2 or
4.

But you're absolutely right that there are only a small handful of
plausible results, even with threading involved.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generate a static back-of-a-book style index?

[Skip Montanaro]

> Have you looked at the ptx command? Might be called "gptx" on a system with 
> the
> GNU coreutils installed with "g" prefixes.

Thanks, Cameron. I was unaware of it. Will check it out.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Steven D'Aprano]

On Sun, 08 Jul 2018 00:00:26 +0300, Marko Rauhamaa wrote:

> Ian Kelly :
>> the leaning of the devs seems to be to refrain from documenting it and
>> instead document that *no* operations are guaranteed atomic.
> 
> I believe that to be wise. Otherwise, Python would limit its future
> implementation options.

o_O

By that logic, one should say we shouldn't document the semantics of 
operations, so we don't limit the implementation.

"What does list.sort do?"

"It currently sorts the list using a stable comparison sort, but you 
can't rely on that it, because we didn't want to limit the 
implementation."

Sorting is guaranteed to be stable. Does that limit the implementation 
options? Yes, of course it does. So does the fact that it *sorts* -- 
we're limited to implementations which *sort*, and specifically 
comparison sorts (not, for example, radix sorts).

Changing the implementation to a radix sort that only works on ints would 
break things. So would changing the implementation to something which 
empties the list of all elements. ("The empty list is always sorted.")

Stable semantics are more important than freedom to change implementation.

Whether or not an operation is thread-safe, or atomic, is part of the 
semantics of the operation. It's not merely a performance issue, and we 
ought to treat it will a bit more respect.

I'm not saying that everything needs to be documented as thread-safe or 
not (for at least one release, Python's sorting actually was stable 
before it was documented as such) but the gung-ho attitude that safety is 
just an implementation detail should be resisted.

Changing implementations from one which is thread safe to one which is 
not can break people's code, and shouldn't be done on a whim. Especially 
since such breakage could be subtle, hard to notice, harder to track 
down, and even harder still to fix.


> The only thing Python should guarantee is that the data structures stay
> "coherent" under race conditions. In other words, there cannot be a
> segmentation fault. For example, if two threads executed this code in
> parallel:
> 
> global i
> i = 1
> i += 1
> 
> a legal end result could be that i contains the string "impossible".

That wouldn't be coherent. The only coherent results are that i could 
equal either 2 or 3:

Possibility 1:

i = 1   # thread 1
i += 1  # thread 1
i = 1   # thread 2
i += 1  # thread 2
assert i == 2

Possibility 2:

i = 1   # thread 1
i = 1   # thread 2
i += 1  # thread 1
i += 1  # thread 2
assert i == 3


Executing statements out of order is impossible, even in threaded code. 
Redefining the meaning of the integer literal 1 is impossible (ignoring 
unsupported shenanigans with ctypes). Monkey-patching the int __iadd__ 
method is impossible. So there is no coherent way to get a result of 
"impossible" from just adding 1 to 1 in any coherent implementation of 
Python.


-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generate a static back-of-a-book style index?

[Cameron Simpson]

On 07Jul2018 18:24, Skip Montanaro  wrote:

I just generated a static website which is nothing more than around
ten years of email messages from a defunct vintage bike mailing list
(170k messages, one per file). Though I've submitted the front page
URL to both Google and Bing, I was thinking that it would be handy to
have something like a static index such as you might find in the back
of a book. I poked around a bit, but the prevalence of JS-based static
websites is hampering the search. I found this on PyPI:

https://pypi.org/project/static_site_index/

but it looks fairly unmaintained and contains no documentation.
(Fortunately, it appears small, so I will at least skim the code to
see if it looks useful.)

Surely this is a solved problem, hopefully in Python?


Have you looked at the ptx command? Might be called "gptx" on a system with the 
GNU coreutils installed with "g" prefixes.


It does exactly this, and has done for decades :-)

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Generate a static back-of-a-book style index?

[Skip Montanaro]

I just generated a static website which is nothing more than around
ten years of email messages from a defunct vintage bike mailing list
(170k messages, one per file). Though I've submitted the front page
URL to both Google and Bing, I was thinking that it would be handy to
have something like a static index such as you might find in the back
of a book. I poked around a bit, but the prevalence of JS-based static
websites is hampering the search. I found this on PyPI:

https://pypi.org/project/static_site_index/

but it looks fairly unmaintained and contains no documentation.
(Fortunately, it appears small, so I will at least skim the code to
see if it looks useful.)

Surely this is a solved problem, hopefully in Python?

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Marko Rauhamaa]

Ian Kelly :
> the leaning of the devs seems to be to refrain from documenting it and
> instead document that *no* operations are guaranteed atomic.

I believe that to be wise. Otherwise, Python would limit its future
implementation options.

The only thing Python should guarantee is that the data structures stay
"coherent" under race conditions. In other words, there cannot be a
segmentation fault. For example, if two threads executed this code in
parallel:

global i
i = 1
i += 1

a legal end result could be that i contains the string "impossible".


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Ethan Furman]

On 07/06/2018 10:51 AM, INADA Naoki wrote:
> On Sat, Jul 7, 2018 at 2:49 AM Steven D'Aprano wrote:

>> I have a dict with string keys:
>>
>> --> D = {'a': None, 'b': None}

>> How do I do a thread-safe insertion if, and only if, the key isn't
>> already there?
>

D.setdefault('c', None)


This is how Enum avoids race conditions when setting up the re.RegexType 
enumeration.

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Fwd: SSLContext.load_verify_locations

[Joseph Netti (RIT Student)]

Hi,

I think I found either a bug in the documentation or a bug in the
load_verify_locations function (or I am just using the function wrong).

https://docs.python.org/3/library/ssl.html#ssl.SSLContext.
load_verify_locations

I am trying to use crls with the ssl.py library which according to the
documentation should be done by with the load_verify_locations function.
Here is my stackoverflow post about it: https://stackoverflow.com/ques
tions/51196492/how-to-use-crls-in-pyopenssl?noredirect=1#
comment89404681_51196492

Thanks,

Joe
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Ian Kelly]

On Fri, Jul 6, 2018 at 11:56 AM INADA Naoki  wrote:
>
> D.setdefault('c', None)

Not guaranteed to be atomic.

I think the only safe way to do it is with a Lock.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: about main()

[Gene Heskett]

On Friday 06 July 2018 14:27:16 Grant Edwards wrote:

> On 2018-07-06, Gene Heskett  wrote:
> > In that case, I hate to say it, but your education is sorely lacking
> > in the fundamentals. Smelting for instance was discussed at length
> > in the high school physics books I was reading by the time I was in
> > the 3rd grade. Don't they teach anything in school anymore? Tanning
> > leather for instance involves a long soaking in strong tea, and
> > doesn't name the brand or genus of the tea, the important part was
> > the tannic acid content.
>
> IIRC, oak leaves/chips work well also.

Yes, and fairly well too from what I've read when I had a shed full of 
deer skins one winter about 50 years ago.
> --
> Grant Edwards   grant.b.edwardsYow! I'm shaving!!
>   at   I'M SHAVING!!
>   gmail.com
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Devin Jeanpierre]

On Sat, Jul 7, 2018 at 6:49 AM Marko Rauhamaa  wrote:
> Is that guaranteed to be thread-safe? The documentation ( s://docs.python.org/3/library/stdtypes.html#dict.setdefault>) makes no
> such promise.

It's guaranteed to be thread-safe because all of Python's core
containers are thread safe (in as far as they document
behaviors/invariants, which implicitly also hold in multithreaded code
-- Python does not take the approach other languages do of
"thread-compatible" containers that have undefined behavior if mutated
from multiple threads simultaneously). It isn't guaranteed to be
_atomic_ by the documentation, but I bet no Python implementation
would make dict.setdefault non-atomic.

There's no good description of the threading rules for Python data
structures anywhere. ISTR there was a proposal to give Python some
defined rules around thread-safety a couple of years ago (to help with
things like GIL-less python projects), but I guess nothing ever came
of it.

-- Devin
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Ian Kelly]

On Sat, Jul 7, 2018 at 8:03 AM Stefan Behnel  wrote:
>
> Marko Rauhamaa schrieb am 07.07.2018 um 15:41:
> > Steven D'Aprano :
> >> On Sat, 07 Jul 2018 02:51:41 +0900, INADA Naoki wrote:
> >>> D.setdefault('c', None)
> >>
> >> Oh that's clever!
> >
> > Is that guaranteed to be thread-safe? The documentation ( > s://docs.python.org/3/library/stdtypes.html#dict.setdefault>) makes no
> > such promise.
>
> It's implemented in C and it's at least designed to avoid multiple lookups
> and hash value calculations, which suggests that it's also thread-safe by
> design (or by a side-effect of the design). Whether that's guaranteed, I
> cannot say, but a change that makes it non-thread-safe would probably be
> very controversial.

It's only implemented in C if you're using CPython (and if it's the
builtin dict type and not a subclass). If there's any chance that your
code might run under any other interpreter than CPython, then you
can't rely on the GIL for thread-safety. I would also point out
https://bugs.python.org/issue25343. While some operations are known to
be atomic (and therefore thread-safe), the leaning of the devs seems
to be to refrain from documenting it and instead document that *no*
operations are guaranteed atomic.

> > At least __collectios_abc.py
> > contains this method definition for MutableMapping:
> >
> > def setdefault(self, key, default=None):
> > 'D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D'
> > try:
> > return self[key]
> > except KeyError:
> > self[key] = default
> > return default
> >
> > There are more such non-thread-safe definitions.
>
> That's a different beast, because Python code can always be interrupted by
> thread switches (between each byte code execution). C code cannot, unless
> it starts executing byte code (e.g. for calculating a key's hash value) or
> explicitly allows a thread switch at a given point.

dict.setdefault does potentially call __hash__ and __eq__ on the key.
Since this is part of the lookup I don't know whether it affects
thread-safety as long as the key is properly hashable, but it does
make it more difficult to reason about. I don't *think* that
setdefault calls Py_DECREF, but if it did then that is another
potential point of thread interruption.

By contrast, using a mutex to guard accesses is definitely safe, and
it's also self-documenting of the fact that thread-safety is a
concern.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Stefan Behnel]

Marko Rauhamaa schrieb am 07.07.2018 um 15:41:
> Steven D'Aprano :
>> On Sat, 07 Jul 2018 02:51:41 +0900, INADA Naoki wrote:
>>> D.setdefault('c', None)
>>
>> Oh that's clever!
> 
> Is that guaranteed to be thread-safe? The documentation ( s://docs.python.org/3/library/stdtypes.html#dict.setdefault>) makes no
> such promise.

It's implemented in C and it's at least designed to avoid multiple lookups
and hash value calculations, which suggests that it's also thread-safe by
design (or by a side-effect of the design). Whether that's guaranteed, I
cannot say, but a change that makes it non-thread-safe would probably be
very controversial.


> At least __collectios_abc.py
> contains this method definition for MutableMapping:
> 
> def setdefault(self, key, default=None):
> 'D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D'
> try:
> return self[key]
> except KeyError:
> self[key] = default
> return default
> 
> There are more such non-thread-safe definitions.

That's a different beast, because Python code can always be interrupted by
thread switches (between each byte code execution). C code cannot, unless
it starts executing byte code (e.g. for calculating a key's hash value) or
explicitly allows a thread switch at a given point.

Stefan

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

[Marko Rauhamaa]

Steven D'Aprano :
> On Sat, 07 Jul 2018 02:51:41 +0900, INADA Naoki wrote:
>> D.setdefault('c', None)
>
> Oh that's clever!

Is that guaranteed to be thread-safe? The documentation () makes no
such promise.

At least __collectios_abc.py
contains this method definition for MutableMapping:

def setdefault(self, key, default=None):
'D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D'
try:
return self[key]
except KeyError:
self[key] = default
return default

There are more such non-thread-safe definitions.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: testing code

[Chris Angelico]

On Sat, Jul 7, 2018 at 10:02 PM, Sharan Basappa
 wrote:
> On Friday, 6 July 2018 09:22:31 UTC+5:30, Chris Angelico  wrote:
>> On Fri, Jul 6, 2018 at 12:56 PM, Sharan Basappa
>>  wrote:
>> > Please let me know if the following understanding of mine is correct.
>> > I need to put the program code in a separate file and organize every 
>> > executable code in some form of function. If any code exists outside of 
>> > function then it is not executable by importing.
>> >
>>
>> Kinda. It's actually the other way around: if any code exists outside
>> of functions, it will be executed immediately when you import. So
>> you're correct in that it would be hard (maybe impossible) to
>> unit-test that; and yes, the normal way to do it is to put all your
>> important code into functions.
>>
>> ChrisA
>
> Chris,
>
> Things do work as per expected with one exception.
> You mentioned that as soon as a file is imported, it executes immediately.
>
> Please see the example below:
>
> file: test_2.py
>
> x = 10
> y = 20
>
> c = x-y
>
> print c
>
> def func1():
> return x+y
>
> test_2_test.py
>
> x = 10
> y = 20
>
> c = x-y
>
> print c
>
> def func1():
> return x+y
>
> this is the output:
> %run "D:/Projects/Initiatives/machine learning/programs/test_2_test.py"
> 30
>
> As you can see, print c in test_2 file was not executed upon import as there 
> is no corresponding output

I don't think you import your other module anywhere.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: testing code

[Sharan Basappa]

On Friday, 6 July 2018 09:22:31 UTC+5:30, Chris Angelico  wrote:
> On Fri, Jul 6, 2018 at 12:56 PM, Sharan Basappa
>  wrote:
> > Please let me know if the following understanding of mine is correct.
> > I need to put the program code in a separate file and organize every 
> > executable code in some form of function. If any code exists outside of 
> > function then it is not executable by importing.
> >
> 
> Kinda. It's actually the other way around: if any code exists outside
> of functions, it will be executed immediately when you import. So
> you're correct in that it would be hard (maybe impossible) to
> unit-test that; and yes, the normal way to do it is to put all your
> important code into functions.
> 
> ChrisA

Chris,

Things do work as per expected with one exception.
You mentioned that as soon as a file is imported, it executes immediately.

Please see the example below:

file: test_2.py

x = 10
y = 20

c = x-y

print c

def func1():
return x+y  

test_2_test.py

x = 10
y = 20

c = x-y

print c

def func1():
return x+y  

this is the output:
%run "D:/Projects/Initiatives/machine learning/programs/test_2_test.py"
30

As you can see, print c in test_2 file was not executed upon import as there is 
no corresponding output
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SWIG+Distutils - no harmony?

[ruifernqq]

Em terça-feira, 15 de janeiro de 2002 21:26:05 UTC+1, Martin Bless  escreveu:
> Getting courageous I try building 'example.pyd' 
> from the SWIG 'Simple Example' using SWIG and the Distutils.
> Looks promising but doesn't succeed. I didn't
> modify 'example.c' and 'example.i'.
> 
> Problems/questions:
> 
> (1) Is 'setup.py' ok?
> (2) BEWARE: What's really shocking:
> After see (unsuccessful)
>   python setup.py build -cmingw32
> the original 'example.c' is GONE AND AWAY -
> replaced by the wrapper.
> (3) What's going wrong?
> 
> Martin
> 
> Sources und logs following:
> 
> >--<>--<>--<
> # setup.py
> # mb: Is this ok?
> import distutils
> from distutils.core import setup, Extension
> #from distutils.extension import Extension
> 
> setup(name = "Example 'Simple' from the SWIG examples",
>   version = "2.2",
>   ext_modules = [Extension("example", ["example.i"])])
> 
> >--<>--<>--<
> /* File : example.i */
> %module example
> 
> extern intgcd(int x, int y);
> extern double Foo;
>   
> >--<>--<>--<
> /* File : example.c */
> 
> /* A global variable */
> double Foo = 3.0;
> 
> /* Compute the greatest common divisor of positive integers */
> int gcd(int x, int y) {
>   int g;
>   g = y;
>   while (x > 0) {
> g = x;
> x = y % x;
> y = g;
>   }
>   return g;
> }
> 
> >--<>--<>--<
> C:\python
> Python 2.2 (#28, Dec 21 2001, 12:21:22) [MSC 32 bit (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>>
> 
> >--<>--<>--<
> swig -version
> 
> SWIG Version 1.3.10u-20011211-2004
> Copyright (c) 1995-1998
> University of Utah and the Regents of the University of California
> Copyright (c) 1998-2001
> University of Chicago
> 
> Compiled with CC
> 
> >--<>--<>--<
> >python setup.py build -cmingw32
> 
> running build
> running build_ext
> building 'example' extension
> swigging example.i to example.c
> c:\pyext\swig\swig.exe -python -dnone 
>   -ISWIG -o example.c example.i
> c:\mingw\bin\gcc.exe -mno-cygwin -mdll 
>   -O -Wall -IC:\PYTHON22\include 
>   -c example.c -o build\temp.win32-2.2\Release\example.o
> writing build\temp.win32-2.2\Release\example.def
> 
> c:\mingw\bin\gcc.exe -mno-cygwin -mdll -static 
>   -s build\temp.win32-2.2\Release\example.o 
>   build\temp.win32-2.2\Release\example.def 
>   -LC:\PYTHON22\libs -lpython22 
>   -o build\lib.win32-2.2\example.pyd
> 
> swig: Warning. -dnone option deprecated.
> 
> example.c: In function `SWIG_UnpackData':
> example.c:363: warning: `uu' might be used uninitialized in this
> function
> example.c: In function `SWIG_ConvertPtr':
> example.c:429: warning: suggest explicit braces to avoid ambiguous
> `else'
> example.c:432: warning: suggest explicit braces to avoid ambiguous
> `else'
> example.c:438: warning: suggest explicit braces to avoid ambiguous
> `else'
> example.c: At top level:
> example.c:154: warning: `SWIG_TypeQuery' defined but not used
> example.c:385: warning: `SWIG_ConvertPtr' defined but not used
> example.c:468: warning: `SWIG_ConvertPacked' defined but not used
> example.c:537: warning: `SWIG_MakeShadow' defined but not used
> build\temp.win32-2.2\Release\example.o(.text+0xbda):example.c:
> undefined reference to `gcd'
> build\temp.win32-2.2\Release\example.o(.text+0xc35):example.c:
> undefined reference to `Foo'
> build\temp.win32-2.2\Release\example.o(.text+0xc6b):example.c:
> undefined reference to `Foo'
> build\temp.win32-2.2\Release\example.o(.text+0xc71):example.c:
> undefined reference to `Foo'
> error: command 'gcc' failed with exit status 1
> 
> >--<>--<>--<

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looking for a recent quote about dynamic typing, possibly on this list

[Stefan Behnel]

Ben Finney schrieb am 07.07.2018 um 03:38:
> Steven D'Aprano  writes:
> 
>> Somebody gave a quote about dynamic typing, along the lines of
>>
>> "Just because a language allows a lot of dynamic features, doesn't mean 
>> people's code uses a lot of dynamism."
> 
> You did refer us to http://lambda-the-ultimate.org/node/1519> on
> this forum. That contains a quote attributed to John Aycock with the
> meaning you paraphrase above.

And since the link in that article seems broken, here's the cited paper:

https://legacy.python.org/workshops/2000-01/proceedings/papers/aycock/aycock.html

Stefan


PS: the link could probably be fixed with a redirect from python.org...

-- 
https://mail.python.org/mailman/listinfo/python-list