Re: Best strategy for testing class and subclasses in pytest?

[dieter]

"C.D. Reimer"  writes:

> I'm writing a chess engine to learn about Python classes and inheritance, and 
> using pytest for the unit test. I've created a Piece class, which has 99% of 
> the functionality for a chess piece, and subclass the other pieces -- Bishop, 
> King, Knight, Pawn, Queen, Rook -- 
> that will implement the move-specific functionality. I'm not sure
> what's the best strategy is for testing the class and subclasses.

I tend to give the tests a structure similar to the implementation, i.e.
the test units correspond to the functional units.

In your case, I would have tests verifying the correct functioning
of the base class and tests verifying that of the subclasses.

When an integration module uses the (already tested) services of
another module (as your subclasses use the services of the base class),
I try to avoid retesting the other modules functionality - but asume
in my new tests that the other (used) modules behave as expected.
This often significantly reduces the test complexity.

I your setup with base class and subclasses this implies that I
usually test only new and overridden methods in the subclasses.


In your case, you could also inherit your subclasses' test case
classes from that of your base class. This would lead to a reexecution
of the base class tests as part of the tests for each subclass.

I follow this route when a subclass has overridden base class methods
in a potentially sensible way.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Canopy editor-Variables browser and help

[Chris Angelico]

On Mon, Aug 24, 2015 at 8:09 AM, lbertolotti via Python-list
 wrote:
> Anyone?

When you post a follow-up like this, it helps to provide some context.
Fortunately for you, I have a threaded email client, but not everyone
does - and not everyone was subscribed to the list when you first
posted.

If you're having difficulties with Canopy, you may do better on a
specific Canopy mailing list. I, for one, have no experience with it,
and can't advise.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Canopy editor-Variables browser and help

[breamoreboy]

On Saturday, May 2, 2015 at 8:50:22 PM UTC+1, lbertolotti wrote:
> Can I:
> 1.Enable a variable browser in Canopy editor similar to the Spyder editor?
> 2.Writing a function say log( gives me the function help, but I can't read 
> the whole documentation
> 3.Eclipse had a auto-complete, can I enable this in Canopy?
> 4.Perhaps I should try using another editor? I was having some problems with 
> Spyder pythonpath...

What do the Canopy docs at http://docs.enthought.com/canopy/ tell you?

If you've used Eclipse why not give PyDev a try?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sometimes bottle takes a lot of time

[Ben Finney]

Johannes Bauer  writes:

> You're entirely right that this kind of personal feud and immature
> mockery is inappropriate for a mailing list and you're also right that
> it does create a toxic atmosphere. Since Python is the lanauge I'm
> most passionate about a detrimental effect on the Python community is
> something that is surely the exact opposite of what I want.
[…]

> I'll follow your advice and thank you for your honest words.

Thank you for publicly acknowledging this, and for taking responsibility
for positive change. This is essential to keeping our community healthy.

-- 
 \“Technology is neither good nor bad; nor is it neutral.” |
  `\   —Melvin Kranzberg's First Law of Technology |
_o__)  |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: asyncio, coroutines, etc. and simultaneous execution

[Akira Li]

Charles Hixson  writes:

> If I understand correctly asyncio, coroutines, etc. (and, of course,
> Threads) are not simultaneously executed, and that if one wants that
> one must still use multiprocessing.  But I'm not sure.  The note is
> still there at the start of threading, so I'm pretty sure about that
> one.

Due to GIL (used by CPython, Pypy, not used by Jython, IronPython,
pypy-stm) only one thread executes Python bytecode at a time. GIL can be
released on I/O or in a C extension such as numpy, lxml, regex. It is
true for any Python module or concept you use.

Unrelated: use concurrent and parallel execution instead of
"simultaneously executed."

Parallelism might make a program faster (it implies that hardware
supports it).

Concurrency is a way to structure the code. The same concurrent program
can run in parallel and without parallelism.

Recommended: "Concurrency is not Parallelism
(it's better!)" talk by Rob Pike's talk
http://stackoverflow.com/questions/1050222/concurrency-vs-parallelism-what-is-the-difference


> The requirement that coroutines always be awaited seems to confirm
> this, but doesn't really say so explicitly. And the concurrent.futures
> can clearly be either, depending on your choices, but the chart in
> 18.5.3.1.3 Example: Chain coroutines is of a kind that I am more
> familiar with in the context of multiprocessing. (E.g., the only gap
> in the chart, which extends across all headings is when a result is
> being waited for during a sleep.)  For threaded execution I would
> expect there to be a gap whenever processing is shifted from one
> column to another.
>
> If someone has authority to edit the documentation a comment like:

Anybody can suggest a patch
https://docs.python.org/devguide/docquality.html

> If you want your application to make better use of the computational
> resources of multi-core machines, you are advised to use
> multiprocessing
> 
> or concurrent.futures.ProcessPoolExecutor
> .
>  However,
> threading is still an appropriate model if you want to run multiple
> I/O-bound tasks simultaneously.
>
> (to quote from the threading documentation) would be helpful at or
> very near the top of each of the appropriate modules.  It would also
> be useful if the modules that were definitely intended to result in
> simultaneous execution, when feasible, were so marked quite near the
> top.

It is not necessary that multiprocessing will make your code faster on a
multi-core machine. It may make it slower depending on the task.

Performance optimization is a vast topic. Short remarks such as you've
suggested are likely misleading.

> OTOH, I may be mistaken about coroutines.  I haven't been able to tell.

Many cooperative multitasking implementations use a *single*
thread. There is a way to offload blocking code e.g.,
loop.run_in_executor() in asyncio.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Canopy editor-Variables browser and help

[lbertolotti via Python-list]

Anyone?
-- 
https://mail.python.org/mailman/listinfo/python-list

asyncio, coroutines, etc. and simultaneous execution

[Charles Hixson]

If I understand correctly asyncio, coroutines, etc. (and, of course, 
Threads) are not simultaneously executed, and that if one wants that one 
must still use multiprocessing.  But I'm not sure.  The note is still 
there at the start of threading, so I'm pretty sure about that one.


The requirement that coroutines always be awaited seems to confirm this, 
but doesn't really say so explicitly. And the concurrent.futures can 
clearly be either, depending on your choices, but the chart in 
18.5.3.1.3 Example: Chain coroutines is of a kind that I am more 
familiar with in the context of multiprocessing. (E.g., the only gap in 
the chart, which extends across all headings is when a result is being 
waited for during a sleep.)  For threaded execution I would expect there 
to be a gap whenever processing is shifted from one column to another.


If someone has authority to edit the documentation a comment like:

If you want your application to make better use of the computational 
resources of multi-core machines, you are advised to use multiprocessing 
 
or concurrent.futures.ProcessPoolExecutor 
. 
However, threading is still an appropriate model if you want to run 
multiple I/O-bound tasks simultaneously.


(to quote from the threading documentation) would be helpful at or very 
near the top of each of the appropriate modules.  It would also be 
useful if the modules that were definitely intended to result in 
simultaneous execution, when feasible, were so marked quite near the top.


OTOH, I may be mistaken about coroutines.  I haven't been able to tell.

P.S.:  I do note that the threading comment was a "*CPython 
implementation detail:"*, and not a part of the Python specifications.  
But CPython is, I believe, a sufficiently dominant implementation that 
such details are quite important.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sometimes bottle takes a lot of time

[Johannes Bauer]

On 23.08.2015 18:47, Michael Torrie wrote:

> Since this is an ajax thing, I can entirely
> understand that Firefox introduces random delays.  Practically all
> ajax-heavy sites I've ever used has had random slowdowns in Firefox.

This would imply that random six-second delays have somehow passed the
Firefox QA effortlessly. It's something that is entirely possible, but
also something that I would consider magnitudes less likely than other
explanations. Six seconds is *huge* for regular web applications.

> Name resolution could be an issue, but the script he wrote to simulate
> the browser requests does not show the slowdown at all.  Firefox could
> be doing name resolution differently than the rest of the system and
> Chrome of course, which wouldn't surprise me as Firefox seems to more
> and more stupid stuff.

Proxy settings are another thing that could influence behavior. Maybe
the proxy of his Chrome is differently configured than Firefox and this
is causing issues. SOCKS proxies can emulate DNS as well. So there is a
plethora of possible causes; no need to shift blame before evidence is
presented, no need to jump to conclusions.

> But your bashing on him is inappropriate and
> unhelpful.
[...]
> What is annoying to me is how you have done nothing but jump all over
> him this whole thread, and several other times.  
[...]
> In fact I can find very few of your posts to this list where you
> aren't bashing Cecil in recent months.  This does not reflect well on
> the list community and drives away people who would otherwise want to
> learn Python.

I think you're right about this. I've had some run-in with Cecil some
months ago - don't even remember what it was about. The thing I did
remember was that I was particularly annoyed by the whole discussion
back then. This probably led to me being a lot more agressive in my
choice of tone than I should have been.

You're entirely right that this kind of personal feud and immature
mockery is inappropriate for a mailing list and you're also right that
it does create a toxic atmosphere. Since Python is the lanauge I'm most
passionate about a detrimental effect on the Python community is
something that is surely the exact opposite of what I want.

> If Cecil's posts annoy you, please ignore them (I wouldn't even respond
> to this post of yours, but I feel like something has to be said).

I'll follow your advice and thank you for your honest words.

Cheers,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sometimes bottle takes a lot of time

[Michael Torrie]

On 08/23/2015 08:05 AM, Johannes Bauer wrote:
> By git bisect he can find out where
> he introduced the bug.

Like Cecil said, this is of little help.  There was no code changed from
when he didn't notice the behavior until he did.

>> Note that this says nothing about the location of the bug, in can still
>> be either in the OPs code or in the framework.
> 
> Yup. Note that he has now shifted from blaming bottle to blaming
> Firefox. Same thing with that claim. If somehow website delivery was
> delayed 6 seconds reproducibly, people would have noticed.

Well it does look like the problem is indeed inside firefox.  Chrome
does exhibit the problem.  Just fetching the urls in a script does not
have the problem either.  Since this is an ajax thing, I can entirely
understand that Firefox introduces random delays.  Practically all
ajax-heavy sites I've ever used has had random slowdowns in Firefox.

> I suspect that either the OPs program is at fault or the OP's setup
> (name resolution or some other weird stuff going on). 

Name resolution could be an issue, but the script he wrote to simulate
the browser requests does not show the slowdown at all.  Firefox could
be doing name resolution differently than the rest of the system and
Chrome of course, which wouldn't surprise me as Firefox seems to more
and more stupid stuff.

> But instead of
> tackling this problem systematically, like a Software Engineer would
> (Wireshark, debugger, profiler) he just blames other people's software.
> This, in my humble opinion, is annoying

He is tackling the problem systematically, though perhaps not in the
same way you would. Sure there are ways he can improve his process, and
he is doing that slowly.  But your bashing on him is inappropriate and
unhelpful.  And sometimes things that are obvious to you and others
(such as strings being iterable in his sql binding problem) are not
obvious to new users of python, even ones with a lot of experience in
other languages.

What is annoying to me is how you have done nothing but jump all over
him this whole thread, and several other times.  You seem to have made
it your mission to bash him continually on this list, mocking him and
saying if he were a wise Senior Software Engineer he would know such and
such.  In fact I can find very few of your posts to this list where you
aren't bashing Cecil in recent months.  This does not reflect well on
the list community and drives away people who would otherwise want to
learn Python.

If Cecil's posts annoy you, please ignore them (I wouldn't even respond
to this post of yours, but I feel like something has to be said).

I for one am happy to help out, and I'm very glad to see a person come
and learn python and be enthusiastic about it.  Unlike many people
learning Python, Cecil has made a strong attempt to learn the idiomatic
ways of programming in Python, and seems to be really enjoying it.  He
hasn't been turned off by the sometimes toxic atmosphere of the list.
He hasn't run off saying Python sux because of whitespace.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sometimes bottle takes a lot of time

[Cecil Westerhof]

On Sunday 23 Aug 2015 17:44 CEST, MRAB wrote:

>> I never blamed bottle, I was asking if it could be a problem with
>> bottle.
>>
> The subject says otherwise. :-)

Yeah, my communication skills can take some improvement. I meant: I
have this problem. I think it could have to do something with bottle.
Can anybody shed some light on it?

I hope to be more precise next time.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is this the way to go with SQLite

[Chris Angelico]

On Mon, Aug 24, 2015 at 2:17 AM, Dennis Lee Bieber
 wrote:
> SQLite3 supports the non-standard
>
> INSERT OR REPLACE ...
>
> (or one can do INSERT OR IGNORE; the OR XXX has a number of values that are
> allowed to control behavior... BUT the OR clause only applies if a UNIQUE
> constraint would be violated by the INSERT action... So if the field is not
> a unique index, no foul is detected)

Sure. But that's still the same as MySQL's (equally non-standard) "ON
DUPLICATE KEY UPDATE", and various others. It's a way of shoving down
a level the common idiom of "do this insert, only if that doesn't
work, do this update instead". But that doesn't seem to be what the
original code was doing - it was more like "do this insert, but if
it's a duplicate based on URL, check if the other fields are the same,
and if not... uhh, print something out?", which is far from common.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sometimes bottle takes a lot of time

[MRAB]

On 2015-08-23 16:20, Cecil Westerhof wrote:

On Sunday 23 Aug 2015 16:05 CEST, Johannes Bauer wrote:


On 22.08.2015 16:15, Christian Gollwitzer wrote:


Probably yes. You should take a look at the OP again and compare
the time stamps. It says that in between two consecutive calls of
the same program, the request was served once in a second, and once
with serious delays. Despite that the server is localhost. In
between both trials there are 20 seconds. I do not see, how git
bisect would help here.


I do completely understand that in two consecutive runs one time the
problem occurs and the other time it doesn't.

It's highly unlikely that such a bug would ever have passed the
bottle QA and if it did it would affect thousands of users (who
would report this issue, since it's very severe). It is much more
likely the bug is somewhere within the OP's program. By git bisect
he can find out where he introduced the bug.


You have to explain something to me: how can I introduce a bug without
changing anything? Maybe by having wrong thoughts?



Note that this says nothing about the location of the bug, in can
still be either in the OPs code or in the framework.


Yup. Note that he has now shifted from blaming bottle to blaming
Firefox. Same thing with that claim. If somehow website delivery was
delayed 6 seconds reproducibly, people would have noticed.


I never blamed bottle, I was asking if it could be a problem with
bottle.


The subject says otherwise. :-)

[snip]

--
https://mail.python.org/mailman/listinfo/python-list

Re: Sometimes bottle takes a lot of time

[Cecil Westerhof]

On Sunday 23 Aug 2015 16:05 CEST, Johannes Bauer wrote:

> On 22.08.2015 16:15, Christian Gollwitzer wrote:
>
>> Probably yes. You should take a look at the OP again and compare
>> the time stamps. It says that in between two consecutive calls of
>> the same program, the request was served once in a second, and once
>> with serious delays. Despite that the server is localhost. In
>> between both trials there are 20 seconds. I do not see, how git
>> bisect would help here.
>
> I do completely understand that in two consecutive runs one time the
> problem occurs and the other time it doesn't.
>
> It's highly unlikely that such a bug would ever have passed the
> bottle QA and if it did it would affect thousands of users (who
> would report this issue, since it's very severe). It is much more
> likely the bug is somewhere within the OP's program. By git bisect
> he can find out where he introduced the bug.

You have to explain something to me: how can I introduce a bug without
changing anything? Maybe by having wrong thoughts?


>> Note that this says nothing about the location of the bug, in can
>> still be either in the OPs code or in the framework.
>
> Yup. Note that he has now shifted from blaming bottle to blaming
> Firefox. Same thing with that claim. If somehow website delivery was
> delayed 6 seconds reproducibly, people would have noticed.

I never blamed bottle, I was asking if it could be a problem with
bottle.

And it is (now) clear that it has to do something with Firefox. Just
fetching the URL's does not give a problem. Chromium does not have a
problem. Only Firefox has a problem. Even in safemode. And with an
older version of Firefox there is (almost) no problem.


> I suspect that either the OPs program is at fault or the OP's setup
> (name resolution or some other weird stuff going on). But instead of
> tackling this problem systematically, like a Software Engineer would
> (Wireshark, debugger, profiler) he just blames other people's
> software. This, in my humble opinion, is annoying as fuck.

Do you know what I find annoying? That you need to keep bashing me,
instead of thinking: was I maybe wrong in bashing? Especially because
you are the only one that thinks you were right. Being the only one
who thinks something, does not necessarily make you wrong, but should
make you contemplate.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Good resource for pythonanywhere and WSGI

[Cecil Westerhof]

I want to host my web application on pythonanywhere so that the people
of Mozilla can investigate the problem it has with this application. I
do not find documentation that is useful for me. Has anyone a good
resource?

The application I want to deploy on pythonanywhere:
https://github.com/CecilWesterhof/PublishedPhotos

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sometimes bottle takes a lot of time

[Johannes Bauer]

On 22.08.2015 16:15, Christian Gollwitzer wrote:

> Probably yes. You should take a look at the OP again and compare the
> time stamps. It says that in between two consecutive calls of the same
> program, the request was served once in a second, and once with serious
> delays. Despite that the server is localhost. In between both trials
> there are 20 seconds. I do not see, how git bisect would help here.

I do completely understand that in two consecutive runs one time the
problem occurs and the other time it doesn't.

It's highly unlikely that such a bug would ever have passed the bottle
QA and if it did it would affect thousands of users (who would report
this issue, since it's very severe). It is much more likely the bug is
somewhere within the OP's program. By git bisect he can find out where
he introduced the bug.

> Note that this says nothing about the location of the bug, in can still
> be either in the OPs code or in the framework.

Yup. Note that he has now shifted from blaming bottle to blaming
Firefox. Same thing with that claim. If somehow website delivery was
delayed 6 seconds reproducibly, people would have noticed.

I suspect that either the OPs program is at fault or the OP's setup
(name resolution or some other weird stuff going on). But instead of
tackling this problem systematically, like a Software Engineer would
(Wireshark, debugger, profiler) he just blames other people's software.
This, in my humble opinion, is annoying as fuck.

Cheers,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is this the way to go with SQLite

[Chris Angelico]

On Sun, Aug 23, 2015 at 11:18 PM, Cecil Westerhof  wrote:
> Also an URL is unique, so I need to check that if it is found, the
> values are the same as the ones I wanted to insert.

And if they aren't? Currently, all you do is print out a message and
continue on; what happens if you get the same URL coming up more than
once?

> select_url  = '''SELECT year
>  ,  month
>  ,  description
>  FROM   LINKS
>  WHERE  URL = ?'''
> year= 2015
> month   = 8

PEP 8 has a word or two to say about this, but carry on. Incidentally,
I'd be inclined to put the SELECT query down below, same as the INSERT
query is; it's not in any way different from just using a string
literal there, and this separates two pieces of code (IMO)
unnecessarily.

> for link in links:
> description = link[0]
> url = link[1]

for description, url in links:

> url_values  = c.execute(select_url, [url]).fetchall()
> if len(url_values) == 0:

if not url_values:

> print('Adding {0}'.format(link))
> c.execute('''INSERT INTO links
>  (year, month, description, URL)
>  VALUES
>  (?, ?, ?, ?)
>   ''',
>   [year, month, description, url])
> else:
> to_insert   = (year, month, description)
> found   = url_values[0]
> if found != to_insert:
> print('For {0} found {1} instead of {2}'.format(url, found, 
> to_insert))

Otherwise, looks reasonable. I'm normally expecting to see this kind
of "query, and if it isn't there, insert" code to have an UPDATE in
its other branch (which makes it into a classic upsert or merge
operation - what MySQL calls "INSERT... ON DUPLICATE KEY UPDATE"), or
else throw an error (in which case the cleanest way is to put a unique
key on the column in question and let the database throw the error).
The risk normally is of a race condition; you could execute your
SELECT query, find no results, and then have someone else insert one
just a moment before you do. But with SQLite, you're probably assuming
no other writers anyway - an assumption which (I think) you can
mandate simply by opening a transaction and holding it through the
full update procedure - which would make this safe.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Is this the way to go with SQLite

[Cecil Westerhof]

I understood that with sqlite3 in Python you can not use prepared
statements. Below the way I solved this.

Also an URL is unique, so I need to check that if it is found, the
values are the same as the ones I wanted to insert.

This is my code.

select_url  = '''SELECT year
 ,  month
 ,  description
 FROM   LINKS
 WHERE  URL = ?'''
year= 2015
month   = 8
for link in links:
description = link[0]
url = link[1]
url_values  = c.execute(select_url, [url]).fetchall()
if len(url_values) == 0:
print('Adding {0}'.format(link))
c.execute('''INSERT INTO links
 (year, month, description, URL)
 VALUES
 (?, ?, ?, ?)
  ''',
  [year, month, description, url])
else:
to_insert   = (year, month, description)
found   = url_values[0]
if found != to_insert:
print('For {0} found {1} instead of {2}'.format(url, found, 
to_insert))


-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sometimes bottle takes a lot of time

[Cecil Westerhof]

On Sunday 23 Aug 2015 03:05 CEST, Chris Angelico wrote:

>> But in principal I have found the problem. (Not the reason.) The
>> problem is Firefox. (So it is not bottle and also not AngularJS.)
>> When using Chrome there is no problem. Not even when I do 15 times
>> a refresh. With Firefox there is this problem. Even when I restart
>> it.
>
> Huh, interesting. I can't help there, but I wish you the very best
> of luck in finding it.

It looks like it is a recent problem. I also installed it on an old
AcerAspire One. The fetch takes there five times as long, but fetching
through Firefox is a lot faster. Not as fast as Chromium, but the
difference is a lot less. But there Ice-wasel 38.0 is used instead of
Firefox 40.

Mozilla wants a life version. I am trying to install it on
PythonAnywhere.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to handle cpu cache in python ( or fastest way to call a function once)

[Steven D'Aprano]

On Sun, 23 Aug 2015 10:07 pm, Vladimir Ignatov wrote:

> Hi,
> 
>>> for i in range(100): #just to create a time interval, seems this
>>> disturb cpu cache?
>>> pass
> 
> Python interpreter consumes memory quite extensively because
> "everything is object".  So constructions like:
> 
> range(100):
> 
> _take_ memory.  Additionally it will trigger garbage collecting code
> on deallocation time so expect even more delay.

Normally you would be correct, but as my timing results show, using xrange
instead of range does not make any difference.

Whatever is going on here, it isn't as simple as "range(100) builds a
giant list".



-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to handle cpu cache in python ( or fastest way to call a function once)

[Vladimir Ignatov]

Hi,

>> for i in range(100): #just to create a time interval, seems this disturb 
>> cpu cache?
>> pass

Python interpreter consumes memory quite extensively because
"everything is object".  So constructions like:

range(100):

_take_ memory.  Additionally it will trigger garbage collecting code
on deallocation time so expect even more delay.
To get most out of Python - all "numbers crushing" / "pixel pushing" /
"store gigabytes" code should go to low-level compiled binary
libraries.


Vladimir

https://itunes.apple.com/us/app/python-code-samples/id1025613117
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to handle cpu cache in python ( or fastest way to call a function once)

[Steven D'Aprano]

On Sun, 23 Aug 2015 04:10 pm, Yuzhi Xu wrote:

> I find out that python's VM seems to be very unfriendly with CPU-Cache.

Possibly. More comments below.


> for example:
> ***
> import time
> a = range(500)
> 
> sum(a)
> 
> for i in range(100): #just to create a time interval, seems this
> disturb cpu cache?
> pass
> 
> 
> st = time.time()
> sum(a)
> print (time.time() - st)*1e6
> 
> *
> time:> 100us

On my machine, I get about 20-25 μs for this:

(a.py contains your code above)

[steve@ando ~]$ python2.7 a.py
21.9345092773
[steve@ando ~]$ python2.7 a.py
21.9345092773
[steve@ando ~]$ python2.7 a.py
24.0802764893
[steve@ando ~]$ python2.7 a.py
23.8418579102



> another case:
> *
> import time
> a = range(500)
> 
> for i in range(10):
> st = time.time()
> sum(a)
> print (time.time() - st)*1e6
> 
> *
> time:~ 20us


Running this as b.py, I get times of around 15μs, a bit faster than the
first version, but not a factor of five times faster as you get.

[steve@ando ~]$ python2.7 b.py
[...]
15.0203704834
15.0203704834
25.0339508057
16.9277191162
20.0271606445
94.8905944824
15.9740447998
15.0203704834
15.0203704834
15.0203704834
15.0203704834
14.066696167
13.8282775879
15.0203704834
Traceback (most recent call last):
  File "b.py", line 6, in 
sum(a)
KeyboardInterrupt


Above, you say:

> for i in range(100): #just to create a time interval, seems this
> disturb cpu cache?
> pass

But remember that range() is a function, and so, yes, it may disturb the CPU
cache. What did you expect?

But I'm not sure how the CPU cache will interact with code in a high-level
language like Python. I suspect that more likely, it simply has something
to do with range(100) building an enormous list of integers.

Here's another version:

[steve@ando ~]$ cat c.py
import time
a = range(500)
sum(a)
for i in range(100):
pass
sum(a)
st = time.time()
sum(a)
print (time.time() - st)*1e6

[steve@ando ~]$ python2.7 c.py
15.9740447998



And one more:


[steve@ando ~]$ cat d.py
import time
a = range(500)
sum(a)
for i in xrange(100): # Use xrange instead of range
pass
st = time.time()
sum(a)
print (time.time() - st)*1e6

[steve@ando ~]$ python2.7 d.py
22.1729278564
[steve@ando ~]$ python2.7 d.py
23.1266021729



So... on my machine, the difference between xrange and range makes no
difference: in both cases, calling sum() takes about 22μs.

But calling sum() twice speeds up the second call to about 16μs, or about
25% faster. (Not 80% faster, as you find.)


One last test:


[steve@ando ~]$ cat e.py
import time
a = range(500)
# Without warm-up.
st = time.time()
sum(a)
print (time.time() - st)*1e6
# Second time, with warm-up.
st = time.time()
sum(a)
print (time.time() - st)*1e6
# Add a delay.
for i in xrange(1000):
pass
st = time.time()
sum(a)
print (time.time() - st)*1e6
st = time.time()
sum(a)
print (time.time() - st)*1e6


[steve@ando ~]$ python2.7 e.py
15.0203704834
15.0203704834
10.9672546387
10.9672546387
[steve@ando ~]$ python2.7 e.py
15.9740447998
12.8746032715
12.1593475342
10.9672546387
[steve@ando ~]$ python2.7 e.py
15.9740447998
20.0271606445
15.0203704834
15.9740447998




-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to handle cpu cache in python ( or fastest way to call a function once)

[Stefan Behnel]

Yuzhi Xu schrieb am 23.08.2015 um 08:10:
> I find out that python's VM seems to be very unfriendly with CPU-Cache.
> see:
> http://stackoverflow.com/questions/32163585/how-to-handle-cpu-cache-in-python-or-fastest-way-to-call-a-function-once
> http://stackoverflow.com/questions/32153178/python-functionor-a-code-block-runs-much-slower-with-a-time-interval-in-a-loop
> 
> for example:
> ***
> import time
> a = range(500)
> 
> sum(a)
> 
> for i in range(100): #just to create a time interval, seems this disturb 
> cpu cache?
> pass
> 
> 
> st = time.time()
> sum(a)
> print (time.time() - st)*1e6
> 
> *
> time:> 100us
> 
> 
> another case:
> *
> import time
> a = range(500)
> 
> for i in range(10):
> st = time.time()
> sum(a)
> print (time.time() - st)*1e6
> 
> *
> time:~ 20us
> 
> 
> we can see when running frequently, the code becomes much faster.

That does not seem like a straight forward deduction. Especially the
interpretation that the CPU caching behaviour is to blame here seems rather
far fetched.

My guess is that it rather has to do with CPython's internal object caching
or something at that level. However, given the absolute timings above, I
wouldn't bother too much finding it out. It's unlikely to hurt real-world
code. (And in fact, the more interesting case where things are happing
several times in a row rather than being a negligible constant one-time
effort seems to be substantially faster in your timings. Congratulations!)


> is there a solution?

Is there a problem?

Stefan


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sandboxing Python

[Akira Li]

Mark Lawrence  writes:

> I was always led to believe that the subject was a difficult thing to
> do, but here
> https://www.reddit.com/r/learnpython/comments/3huz4x/how_to_do_math_inside_raw_input/
> is a safe solution in only 23 characters, or are there any discernable
> flaws in it?

Related: 
http://stackoverflow.com/questions/2371436/evaluating-a-mathematical-expression-in-a-string

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Qestion

[Cameron Simpson]

On 23Aug2015 09:28, Laura Creighton  wrote:

In a message of Sat, 22 Aug 2015 06:53:21 -, ali ranjbar writes:

I have python version 2.4.3
Which version of PIL is appropriate for me and how can I add it to my systems?


If you really have python 2.4.3 then you badly need a newer Python.


If he's on RedHat Linux 5 (or CentOS 5) the system Python is 2.4.3.

Cheers,
Cameron Simpson 

"How do you know I'm Mad?" asked Alice.
"You must be," said the Cat, "or you wouldn't have come here."
--
https://mail.python.org/mailman/listinfo/python-list

Re: Qestion

[Laura Creighton]

In a message of Sat, 22 Aug 2015 06:53:21 -, ali ranjbar writes:
>hi dear friend
>
>I have python version 2.4.3
>
>Which version of PIL is appropriate for me and how can I add it to my systems?
>
>Regards
>
>-- 
>https://mail.python.org/mailman/listinfo/python-list

If you really have python 2.4.3 then you badly need a newer Python.
However Pillow, with a version number <2.0 is promised to work for
you.  https://pillow.readthedocs.org/installation.html
The latest 1.x version of Pillow is https://pypi.python.org/pypi/Pillow/1.7.8
so you can get this.

Or did you mean that you have python 3.4.3 ?
In which case you still want Pillow
but this one.  https://pypi.python.org/pypi/Pillow/2.9.0
And this one may already come as a package for your system, if you
don't want to use pip.

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list