Re: =- and -= snag

[aapost]

On 3/14/23 18:50, Rob Cliffe wrote:
On 14/03/2023 21:28, avi.e.gr...@gmail.com wrote:



Type hints are actually situationally quite useful (though yes, kind of 
hard to understand when you first come across their use, largely because 
of things like Union). And I wouldn't recommend their use in all 
situations because maintaining them is frustrating.


xmlschema uses them extensively though and I appreciate them. The module 
does what I need where all the other offerings failed, and parsing the 
source because of the subject matter would have been harder to quickly 
grok without them.


Alternatively, I spent quite a while on the subject of pep 232 (before 
finding the pep) trying to understand it because intuitively I felt like 
there was something I must be missing. Turns out there isn't. And what 
my thoughts were assuming was the intent, are discussed under "Future 
Directions", but when seeing it laid out like that, subjectively the 
dissenting conclusion of "It opens the way to mind abuse." is more 
correct. "What if I launch a virtual machine inside a virtual machine 
inside a virtual machine inside a virtual machine, what happens?"...


If I want to have a grouping of minor functionality or attributes on a 
callable, I can get that with creating a class that defines __call__.


I would say 232 is fine for what it is (I don't have to use it, and in 
many cases it probably is), but the only wide use I know of that I have 
come across is in the standard library ElementTree:

# For tests and troubleshooting
register_namespace._namespace_map = _namespace_map

Which I hate that variable and it's design with a passion. lol.

And exactly, the more a language starts policing my ability to be 
flexible in my trying to find a creative solution to a problem, the less 
I want to use it. Sometimes to have something useably beautiful to 
interact with requires something painfully complex under the surface. 
Hiding and avoiding all complexity away so you never have to see it or 
make mistakes prevents you from growing.


Anyway, the 20th rule to The Zen of Python is:
Don't believe your own bullshit *shrug*


--
https://mail.python.org/mailman/listinfo/python-list

Re: =- and -= snag

[rbowman]

On Tue, 14 Mar 2023 22:15:34 GMT, Gilmeh Serda wrote:

> On Mon, 13 Mar 2023 22:26:20 +0100, Morten W. Petersen wrote:
> 
>> numbers for calculations didn't add up.  It went into negative numbers,
>> when that shouldn't have been possible.
> 
> We have all written code that makes us wonder why the compiler even
> bothered with it and didn't give up on the first line.

gcc does have a flag so it will stop on the first error instead of 
wandering around aimlessly spitting out 100 errors because you missed a 
curly bracket. Sometimes it's useful.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: =- and -= snag

[Keith Thompson]

"Morten W. Petersen"  writes:
> I was working in Python today, and sat there scratching my head as the
> numbers for calculations didn't add up.  It went into negative numbers,
> when that shouldn't have been possible.
>
> Turns out I had a very small typo, I had =- instead of -=.
>
> Isn't it unpythonic to be able to make a mistake like that?

Very early versions of C (around 1975 or so, before K was
published) actually used "op=" for compound assignment operators, so
`x =- 2` would subtract 2 from x.  It was changed to "=op" (`x -= 2`)
precisely to avoid this ambiguity that programmers kept running
into (people were less generous with whitespace back then).

As late as the late 1990s, I used a compiler (VAXC) that still
recognized the old-style compound assignment operators, though I
think it warned about them.

I thought "Pythonic" was more about how you write code than about the
design of the language.  But designing a language syntax so typos are
likely to be syntax errors rather than valid code with different
semantics is an interesting challenge.

-- 
Keith Thompson (The_Other_Keith) keith.s.thompso...@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tkinter and cv2: "not responding" popup when imshow launched from tk app

[aapost]

On 3/14/23 06:54, John O'Hagan wrote:

Hi list

I'm trying to use cv2 to display images created as numpy arrays, from
within a tkinter app (which does other things with the arrays before
they are displayed as images). The arrays are colour-coded
visualisations of genomes and can be over a billion elements in size,
and I've found the PIL methods to display images in tkinter are too
slow and memory-heavy.

Here is minimal code that demonstrates the problem in the subject line:

import cv2
from tkinter import *

images=['a.jpg', 'b.jpg', 'c.jpg'] #change to image paths

cv2.namedWindow('W', cv2.WND_PROP_FULLSCREEN)
cv2.setWindowProperty('W', cv2.WND_PROP_FULLSCREEN,
cv2.WINDOW_FULLSCREEN)
counter=[0]
def show():
cv2.imshow('W', cv2.imread(images[counter[0] % len(images)]))
cv2.waitKey(1)
counter[0] += 1

root=Tk()
root.wm_attributes("-topmost", 1)
Button(root, text=' Show ', command=show).pack()
mainloop()

It works up to a point - I can cycle through the images by clicking the
button - but if I mouse-click on the displayed image (e.g. to use the
zooming and panning features of cv2), nothing happens, and a few
seconds later the image greys out and a popup appears saying "'Unknown'
is not responding" and giving the option of waiting or forcing close
(but sometimes these options are greyed out too). Clicking "wait", if
available, closes the popup but it comes back a few seconds later. If I
then click on the tkinter window titlebar, the popup changes to "'Tk'
is not responding". Clicking on the button still works and after a few
clicks the popup closes.

This happens under both x11 and wayland, but under wayland, I  also get
this error:

"QSocketNotifier: Can only be used with threads started with QThread
qt.qpa.wayland: Wayland does not support QWindow::requestActivate()"

and only every second button press displays a new image ,with only
every second image displayed.

I think this particular popup is a Gnome thing, but AIUI most DEs have
something similar to detect stuck apps. But the app is not stuck.

I suspect this is some kind of interaction between the call to
cv2.waitKey (which is necessary but I've never understood why!) and the
tkinter event loop, but it's beyond my knowledge.

Any suggestions about causes or workarounds?

Thanks

--

John





I don't get any of the zoom/panning behavior with waitKey(1). But a 
couple notes from the web:



https://docs.opencv.org/2.4/modules/highgui/doc/user_interface.html

Note

This function should be followed by waitKey function which displays the 
image for specified milliseconds. Otherwise, it won’t display the image. 
For example, waitKey(0) will display the window infinitely until any 
keypress (it is suitable for image display). waitKey(25) will display a 
frame for 25 ms, after which display will be automatically closed. (If 
you put it in a loop to read videos, it will display the video 
frame-by-frame)


https://pythonexamples.org/python-opencv-imshow/

cv2.waitKey(0) is important for holding the execution of the python 
program at this statement, so that the image window stays visible. If 
you do not provide this statement, cv2.imshow() executes in fraction of 
a second and the program closes all the windows it opened, which makes 
it almost impossible to see the image on the window.





if I change waitKey to 0, I get the ability to zoom/pan, but it places 
the tk window in a blocked state because it is waiting on cv2 to return. 
If I hit the ESC key, it releases the wait and gives control back to the 
tk window, allowing me to press show again to continue to the next image.


I can try to change waitKey to a high ms like 1000 and have zoom/pan 
for that amount of time before it gives control back to tk (not a 
sensical approach).


The fact that you can still see the image after tk takes back control is 
I believe just a matter of design, some examples show 
cv2.destroyAllWindows() after waitKey(0) to clean that up, but of course 
if you are reusing the same window, that destroys the target.


You can resolve that if you move
cv2.namedWindow('W', cv2.WND_PROP_FULLSCREEN)
cv2.setWindowProperty('W', cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN)

to inside the show, and destroy it after the wait, and make waitKey 0, 
this allows creation/cleanup of that window per image


Hitting ESC when done zooming/panning on each image to get back to tk.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Tkinter and cv2: "not responding" popup when imshow launched from tk app

[John O'Hagan]

On Tue, 2023-03-14 at 13:52 +, Weatherby,Gerard wrote:
> Assuming you’re using opencv-python, I’d post query at
> https://github.com/opencv/opencv-python/issues.


Thanks Gerard

I'm using the python3-opencv package from Debian testing. Is that
github the appropriate place for this query?

Thanks

--

John

> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tkinter and cv2: "not responding" popup when imshow launched from tk app

[John O'Hagan]

On Tue, 2023-03-14 at 08:07 -0400, Thomas Passin wrote:
> On 3/14/2023 6:54 AM, John O'Hagan wrote:
> > Hi list
> > 
> > I'm trying to use cv2 to display images created as numpy arrays,
> > from
> > within a tkinter app (which does other things with the arrays
> > before
> > they are displayed as images). The arrays are colour-coded
> > visualisations of genomes and can be over a billion elements in
> > size,
> > and I've found the PIL methods to display images in tkinter are too
> > slow and memory-heavy.
> > 
> > Here is minimal code that demonstrates the problem in the subject
> > line:
> > 
> > import cv2
> > from tkinter import *
> > 
> > images=['a.jpg', 'b.jpg', 'c.jpg'] #change to image paths
> > 
> > cv2.namedWindow('W', cv2.WND_PROP_FULLSCREEN)
> > cv2.setWindowProperty('W', cv2.WND_PROP_FULLSCREEN,
> > cv2.WINDOW_FULLSCREEN)
> > counter=[0]
> > def show():
> >     cv2.imshow('W', cv2.imread(images[counter[0] % len(images)]))
> >     cv2.waitKey(1)
> >     counter[0] += 1
> > 
> > root=Tk()
> > root.wm_attributes("-topmost", 1)
> > Button(root, text=' Show ', command=show).pack()
> > mainloop()
> > 
> > It works up to a point - I can cycle through the images by clicking
> > the
> > button - but if I mouse-click on the displayed image (e.g. to use
> > the
> > zooming and panning features of cv2), nothing happens, and a few
> > seconds later the image greys out and a popup appears saying
> > "'Unknown'
> > is not responding" and giving the option of waiting or forcing
> > close
> > (but sometimes these options are greyed out too). Clicking "wait",
> > if
> > available, closes the popup but it comes back a few seconds later.
> > If I
> > then click on the tkinter window titlebar, the popup changes
> > to "'Tk'
> > is not responding". Clicking on the button still works and after a
> > few
> > clicks the popup closes.

[...]

> I don't know anything about the specifics here, but with billions of 
> elements you will not be able to show more than a small fraction on
> the 
> screen.  

Hi Thomas

Thanks for your reply.

In the real app I use interpolating methods to fit the whole image on
the screen. In cv2:

cv2.imshow('W', cv2.resize(array, (width, height))

It's very quick, fractions of a second even for a billion+ sized array.
The PIL/Tk equivalent:

im = ImageTk.PhotoImage(Image.fromarray(array).resize((width, height)))
canvas.create_image(width/2, height/2, image=im)

did the same thing but was very, very slow for such large arrays (15
minutes or more per image, with memory heavily swapped out).

Having said all that, the specific problem I'm having isn't related to
the size of the arrays. The code I posted above triggers the problem
even with small images.

> So I think that a progressive disclosure approach would pay 
> off.  Sample the arrays down to a more workable size before creating
> the 
> screen images.  If you want to zoom in, resample them and recreate
> new 
> images that only cover the zoomed in region in more detail.
> 
> It would also be useful to cache the generated images so they can be 
> re-displayed without needing to be regenerated each time.

This is exactly the approach I took in the first draft (except the
caching idea)! I wasn't using interpolating methods and was limited to
a minimum of one pixel per array element, and therefore limited in how
much of the array could be displayed. Even this got pretty slow in
tkinter if fully zoomed out, although the caching would have helped
with that. But the project brief requires the whole genome to be
visible by default.

Thanks

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Distributing program for Linux

[Weatherby,Gerard]

It’s really going to depend on the distribution and whether you have root 
access.

If you have Ubuntu and root access, you can add the deadsnakes repo, 
https://launchpad.net/~deadsnakes, and install whatever Python you want.

The default ‘python3’ remains but you can called a specific Python, (e.g. 
python3.10).

A typical shebang line would be:

#!/usr/bin/env python3.10

From: Python-list  on 
behalf of Loris Bennett 
Date: Tuesday, March 14, 2023 at 12:27 PM
To: python-list@python.org 
Subject: Distributing program for Linux
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

Hi,

If I write a system program which has Python >= 3.y as a dependency,
what are the options for someone whose Linux distribution provides
Python 3.x, where x < y?

I am aware that an individual user could use (mini)conda to install a
more recent version of Python in his/her home directory, but I am
interested in how root would install such a program.

Cheers,

Loris

--
This signature is currently under constuction.
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l02A4qczH46l1ScA8yisiIwlDKh96sy16woPSOSABWqym4b6dBtHzExfFwZsnPDezDwDqaM0fdCMs3080WQQZ-b5OghOOpI$
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Debugging reason for python running unreasonably slow when adding numbers

[Oscar Benjamin]

On Tue, 14 Mar 2023 at 16:27, Alexander Nestorov  wrote:
>
> I'm working on an NLP and I got bitten by an unreasonably slow behaviour in 
> Python while operating with small amounts of numbers.
>
> I have the following code:
>
> ```python
> import random, time
> from functools import reduce
>
> def trainPerceptron(perceptron, data):
>   learningRate = 0.002
>   weights = perceptron['weights']
>   error = 0
>   for chunk in data:
>   input = chunk['input']
>   output = chunk['output']
>
>   # 12x slower than equivalent JS
>   sum_ = 0
>   for key in input:
>   v = weights[key]
>   sum_ += v

In Python something like your task here would usually use something
along the lines of NumPy. Your two innermost loops involve adding up a
subset of numbers from a list chosen using a list of indices. This is
something that numpy can do much more efficiently with its fancy
indexing e.g.:

In [3]: a = np.array([1, 2, 3, 4, 5, 6, 7])

In [4]: b = np.array([0, 3, 5])

In [5]: a[b]
Out[5]: array([1, 4, 6])

In [6]: a[b].sum()
Out[6]: 11

This a[b].sum() operation in your code would be weights[input].sum()
and would be much faster than the loop shown (the speed difference
will be larger if you increase the size of the input array).

--
Oscar
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: =- and -= snag

[Rob Cliffe via Python-list]

On 14/03/2023 21:28, avi.e.gr...@gmail.com wrote:

TThere are people now trying to in some ways ruin the usability by putting in 
type hints that are ignored and although potentially helpful as in a linter 
evaluating it, instead often make it harder to read and write code if required 
to use it.

+1
Whenever I see code with type hints, I have to edit them out, either 
mentally, or physically, to understand what the code is actually doing.  
It's adding new syntax which I'm not used to and don't want to be forced 
to learn.

Rob Cliffe
--
https://mail.python.org/mailman/listinfo/python-list

Re: Debugging reason for python running unreasonably slow when adding numbers

[Chris Angelico]

On Wed, 15 Mar 2023 at 08:53, Peter J. Holzer  wrote:
>
> On 2023-03-14 16:48:24 +0900, Alexander Nestorov wrote:
> > I'm working on an NLP and I got bitten by an unreasonably slow
> > behaviour in Python while operating with small amounts of numbers.
> >
> > I have the following code:
> [...]
> >   # 12x slower than equivalent JS
> >   sum_ = 0
> >   for key in input:
> >   v = weights[key]
> >   sum_ += v
> >
> >   # 20x slower than equivalent JS
> >   #sum_ = reduce(lambda acc, key: acc + weights[key], input)
>
> Not surprising. Modern JavaScript implementations have a JIT compiler.
> CPython doesn't.
>
> You may want to try PyPy if your code uses tight loops like that.
>
> Or alternatively it may be possible to use numpy to do these operations.
>

Or use the sum() builtin rather than reduce(), which was
*deliberately* removed from the builtins. The fact that you can get
sum() without importing, but have to go and reach for functools to get
reduce(), is a hint that you probably shouldn't use reduce when sum
will work.

Naive code is almost always going to be slower than smart code, and
comparing "equivalent" code across languages is almost always an
unfair comparison to one of them.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Debugging reason for python running unreasonably slow when adding numbers

[Peter J. Holzer]

On 2023-03-14 16:48:24 +0900, Alexander Nestorov wrote:
> I'm working on an NLP and I got bitten by an unreasonably slow
> behaviour in Python while operating with small amounts of numbers.
> 
> I have the following code:
[...]
>       # 12x slower than equivalent JS
>       sum_ = 0
>       for key in input:
>           v = weights[key]
>           sum_ += v
> 
>       # 20x slower than equivalent JS
>       #sum_ = reduce(lambda acc, key: acc + weights[key], input)

Not surprising. Modern JavaScript implementations have a JIT compiler.
CPython doesn't.

You may want to try PyPy if your code uses tight loops like that.

Or alternatively it may be possible to use numpy to do these operations.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: =- and -= snag

[avi.e.gross]

There seem to be a fundamental disconnect here based on people not 
understanding what can happen when spaces are optional. Yes, I have had my 
share of times I found what I programmed was not quite right and been unhappy 
but the result was mostly learning how to not not not not do that next time and 
follow the rules.

When I write:

Var = -1

The result of five minus signs in a row is not a new long operator called 
"-". It is five instances of "-" and looks more like:

Var=-(-(-(-(-1

You can even throw in some plus signs and they are effectively ignored.

It is no different than some other operators like:

Var = not not not not True

So the "=-" case is not a single operator even as "-=" is a single operator.

If I add another negative symbol to the above, I have two operators with a 
binary operator of "-=" that may be implemented more efficiently or use a 
dunder method that handles it and then a unary "-" operator. 

>>> Var = 1
>>> Var -=-1
>>> Var
2

Yes, I can sympathize with humans who think the computer should do what they 
meant. They may also not like scenarios where they mess up the indentation and 
assume the computer should guess what they meant.

But life is generally not like that. Whenever you are in doubt as to how 
something will be parsed especially given how many precedence levels python has 
and so on, USE PARENTHESES and sometimes spaces.

If you type "Var - = 5" you get a syntax error because it sees two different 
operators that do not mesh well. If you type "Var = - 5" you get a result that 
should make sense as the minus binds to the 5 and negates it and then the 
assignment is done. If you leave the space between symbols out, then "-=" 
evaluates to a new operator and "=-" evaluates to two operators as described.

There are lots of trivial bugs people find including truly simple ones like 
subtracting instead of adding or using a floating point number like 5.0 when 
you meant to use an integer or forgetting that 5/3 and 5//3 do somewhat 
different things. People often substitute things like bitwise operators and the 
computer does what you tell it. Many languages have doubled operators like "&" 
versus "&&" that do different things. And if you want to make a list and 
instead of using square brackets use curly brackets, in python, you get a set! 
There are so many places you can mess up.

ALL languages have such features where people can and do make mistakes and 
sometimes cannot easily find them. Add in mistakes where different parts of a 
program use the same variable name and one changes it out and the other gets a 
confusing result. Simply put, lots of stuff is not only legal but often useful 
and even when a linter or other program sees what might be a mistake, it will 
often be wrong and something you wanted done.

Consider another such arena in the lowly spelling Checker that keeps telling me 
things are spelled wrong because it does not know Schwarzenegger is a name or 
that Grosz and Groß are valid variations on the spelling of my name.  Imagine 
what it does when I write in any language other than English. One solution has 
been to have it add such words to a dictionary but that backfires when it 
allows words as valid even though in the current context, it is NOT VALID. So 
some such programs allow you to designate what dictionary/language to use to 
check a region such as a paragraph. Some may transition to being closer to 
grammar checkers that try to parse your sentences and make sure not only that a 
word is a valid spelling but valid given what role it plays in the sentence!

Computer languages are both far simpler and yet weirder. You need to use them 
in ways the documentation says. But when you consider various ideas about 
scope, you can end up with it needing to know which of perhaps many copies of 
the same variable name is being referenced. So if you wrote a program where you 
had a local variable inside a function that you changed and you ASS U ME d you 
could use that variable outside the scope later and another variable already 
exists there with the same name, how is it supposed to know you made a mistake? 
How does it know you wanted a deep copy rather than a reference or shallow copy?

There are so many other examples that the short answer to many questions is 
something like THAT IS THE WAY IT IS. Don't do that!

I doubt anyone would like it if computer programs were written with multiple 
layers of redundancy so that many errors could be detected when all the parts 
do not align along with a checksum. Human languages that do something similar 
are, frankly, a royal pain. I mean once you have a gender and a tense and a  
singular/plural, for example, everything else nearby, such as adjectives, must 
be adjusted to have the right form or endings to match it. That may have been 
great when it was hard to hear a speaker from the back of the theater so the 
redundancy helped offer possible corrections but these days just makes the 

Re: =- and -= snag

[Morten W. Petersen]

Hoi.

After reading the replies, I think some script / linter etc. is the right
thing to do. What's the best linter for Python out there that covers this
as well?

At first glance I thought =- was a new assignment operator, and this is
what seemed unpythonic to me, to have two operators that were so similar.

-Morten

On Tue, Mar 14, 2023 at 5:31 PM Jon Ribbens via Python-list <
python-list@python.org> wrote:

> On 2023-03-13, Morten W. Petersen  wrote:
> > I was working in Python today, and sat there scratching my head as the
> > numbers for calculations didn't add up.  It went into negative numbers,
> > when that shouldn't have been possible.
> >
> > Turns out I had a very small typo, I had =- instead of -=.
> >
> > Isn't it unpythonic to be able to make a mistake like that?
>
> Why would it be? How could it be? Mandating white-space between
> operators would be unpythonic.
>
> That's nothing anyway - yesterday I had an issue in TypeScript which
> confused me for a while which turned out to be because 1 + 1 = 11.
> (I thought the whole point of TypeScript was to prevent things like
> that...)
> --
> https://mail.python.org/mailman/listinfo/python-list
>


-- 
I am https://leavingnorway.info
Videos at https://www.youtube.com/user/TheBlogologue
Twittering at http://twitter.com/blogologue
Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen
Also playing music and podcasting here:
http://www.mixcloud.com/morten-w-petersen/
On Google+ here https://plus.google.com/107781930037068750156
On Instagram at https://instagram.com/morphexx/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Debugging reason for python running unreasonably slow when adding numbers

[Thomas Passin]

On 3/14/2023 3:48 AM, Alexander Nestorov wrote:

I'm working on an NLP and I got bitten by an unreasonably slow behaviour in 
Python while operating with small amounts of numbers.

I have the following code:

```python
import random, time
from functools import reduce

def trainPerceptron(perceptron, data):
   learningRate = 0.002
   weights = perceptron['weights']
   error = 0
   for chunk in data:
       input = chunk['input']
       output = chunk['output']

       # 12x slower than equivalent JS
       sum_ = 0
       for key in input:
           v = weights[key]
           sum_ += v

       # 20x slower than equivalent JS
       #sum_ = reduce(lambda acc, key: acc + weights[key], input)

       actualOutput = sum_ if sum_ > 0 else 0

       expectedOutput = 1 if output == perceptron['id'] else 0
       currentError = expectedOutput - actualOutput
       if currentError:
           error += currentError ** 2
           change = currentError * learningRate
           for key in input:
               weights[key] += change


[snip]
Just a speculation, but the difference with the javascript behavior 
might be because the JS JIT compiler kicked in for these loops.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Distributing program for Linux

[Simon Ward]

On Tue, Mar 14, 2023 at 04:43:14PM +0100, Loris Bennett wrote:

If I write a system program which has Python >= 3.y as a dependency,
what are the options for someone whose Linux distribution provides
Python 3.x, where x < y?


The docs suggest creating your own package or building and installing 
from source:

https://docs.python.org/3/using/unix.html

To install from source, use ‘make altinstall’ (instead of ‘make 
install’) to avoid shadowing your system Python version. The alternative 
interpreter should be qualified with . version, e.g. 
python3.11


Depending on the package manager used by the distribution, 
‘checkinstall’ could be used to build from source and install as a 
package without rolling your own.

https://wiki.debian.org/CheckInstall

On Ubuntu check out the deadsnakes PPA:
https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa

Or use python-build from pyenv to install to a custom location:
https://github.com/pyenv/pyenv/wiki/Common-build-problems#installing-a-system-wide-python

Simon
--
https://mail.python.org/mailman/listinfo/python-list

Re: Running asyncio.run() more than once

[Clint Olsen]

On Monday, March 13, 2023 at 11:55:22 PM UTC-7, gst wrote:
> Le mardi 14 mars 2023 à 02:32:23 UTC-4, Clint Olsen a écrit : 
> I'm not asyncio expert or even not advanced user, but using a simple list to 
> hold the jobs to execute and fill it as necessary after results gathering is 
> not good ? 
> 
> ``` 
> @async 
> def execute_jobs(jobs: List["Job"]): 
> while len(jobs) > 0: 
> # launch_job(s) 
> # gather_job(s)_result(s) 
> # append_jobs_if_desired 
> ```

The problem with this implementation is that most/all of the code calling this 
app is not async code. So, we'd need a method (and thread) to communicate 
between the sync and async worlds.

A possible implementation is here: 
https://stackoverflow.com/questions/59650243/communication-between-async-tasks-and-synchronous-threads-in-python

So, while this is certainly possible, it would be much more straightforward to 
just call asyncio.run() more than once.

Thanks,

-Clint
-- 
https://mail.python.org/mailman/listinfo/python-list

Running asyncio.run() more than once

[Clint Olsen]

We have an application that involves submitting hundreds to thousands of jobs 
to a shared computing resource, and we're using asyncio to do so because it is 
far less overhead than threading or multiprocessing for the bookkeeping 
required to keep track of all these jobs. It makes extensive use of 
asyncio.create_subprocess_exec(). This was developed mostly in Python 3.9.7.

Normally we know ahead of time all the jobs that need to be run and this can be 
accommodated by a single call to asyncio.run(). However, in this new case we 
need to submit a few hundred jobs, review these results, and compose many more. 
That means a separate call to asyncio.run() is necessary.

I have tried to call our app twice, and during the second iteration things hang 
indefinitely. Processes get launched, but it eventually stops reporting job 
completions.

I have added debug=True to the asyncio.run() keyword args, but I'm not sure 
what I'm looking for that might tell me what's wrong. It may be something I'm 
doing, but based on the docs being ambiguous about this it could also be a 
fundamental limitation of asyncio.

Is what I'm trying going to be impossible to accomplish? I would hate to have 
to rig up some sort of crazy async/sync queue system to feed jobs dynamically 
all because of this problem with asyncio.run().

Thanks,

-Clint
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: =- and -= snag

[Jon Ribbens via Python-list]

On 2023-03-13, Morten W. Petersen  wrote:
> I was working in Python today, and sat there scratching my head as the
> numbers for calculations didn't add up.  It went into negative numbers,
> when that shouldn't have been possible.
>
> Turns out I had a very small typo, I had =- instead of -=.
>
> Isn't it unpythonic to be able to make a mistake like that?

Why would it be? How could it be? Mandating white-space between
operators would be unpythonic.

That's nothing anyway - yesterday I had an issue in TypeScript which
confused me for a while which turned out to be because 1 + 1 = 11.
(I thought the whole point of TypeScript was to prevent things like
that...)
-- 
https://mail.python.org/mailman/listinfo/python-list

Distributing program for Linux

[Loris Bennett]

Hi,

If I write a system program which has Python >= 3.y as a dependency,
what are the options for someone whose Linux distribution provides
Python 3.x, where x < y?

I am aware that an individual user could use (mini)conda to install a
more recent version of Python in his/her home directory, but I am
interested in how root would install such a program.

Cheers,

Loris

-- 
This signature is currently under constuction.
-- 
https://mail.python.org/mailman/listinfo/python-list

Debugging reason for python running unreasonably slow when adding numbers

[Alexander Nestorov]

I'm working on an NLP and I got bitten by an unreasonably slow behaviour in 
Python while operating with small amounts of numbers.

I have the following code:

```python
import random, time
from functools import reduce

def trainPerceptron(perceptron, data):
  learningRate = 0.002
  weights = perceptron['weights']
  error = 0
  for chunk in data:
      input = chunk['input']
      output = chunk['output']

      # 12x slower than equivalent JS
      sum_ = 0
      for key in input:
          v = weights[key]
          sum_ += v

      # 20x slower than equivalent JS
      #sum_ = reduce(lambda acc, key: acc + weights[key], input)

      actualOutput = sum_ if sum_ > 0 else 0

      expectedOutput = 1 if output == perceptron['id'] else 0
      currentError = expectedOutput - actualOutput
      if currentError:
          error += currentError ** 2
          change = currentError * learningRate
          for key in input:
              weights[key] += change

  return error

# Build mock data structure
data = [{
   'input': random.sample(range(0, 5146), 10),
   'output': 0
} for _ in range(11514)]
perceptrons = [{
   'id': i,
   'weights': [0.0] * 5147,
} for i in range(60)] # simulate 60 perceptrons

# Simulate NLU
for i in range(151): # 150 iterations
  hrstart = time.perf_counter()
  for perceptron in perceptrons:
    trainPerceptron(perceptron, data)
  hrend = time.perf_counter()
  print(f'Epoch {i} - Time for training: {int((hrend - hrstart) * 1000)}ms')
```

Running it on my M1 MBP I get the following numbers.

```
Epoch 0 - Time for training: 199ms
Epoch 1 - Time for training: 198ms
Epoch 2 - Time for training: 199ms
Epoch 3 - Time for training: 197ms
Epoch 4 - Time for training: 199ms
...
Epoch 146 - Time for training: 198ms
Epoch 147 - Time for training: 200ms
Epoch 148 - Time for training: 198ms
Epoch 149 - Time for training: 198ms
Epoch 150 - Time for training: 198ms
```

Each epoch is taking around 200ms, which is unreasonably slow given the small 
amount of numbers that are being processed. I profiled the code with `cProfile` 
in order to find out what is going on:

```
         655306 function calls (655274 primitive calls) in 59.972 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      3/1    0.000    0.000   59.972   59.972 {built-in method builtins.exec}
        1    0.005    0.005   59.972   59.972 poc-small.py:1()
     9060   59.850    0.007   59.850    0.007 poc-small.py:4(trainPerceptron)
        1    0.006    0.006    0.112    0.112 poc-small.py:34()
    11514    0.039    0.000    0.106    0.000 random.py:382(sample)
   115232    0.034    0.000    0.047    0.000 
random.py:235(_randbelow_with_getrandbits)
    11548    0.002    0.000    0.012    0.000 {built-in method 
builtins.isinstance}
    11514    0.002    0.000    0.010    0.000 :117(__instancecheck__)
   183616    0.010    0.000    0.010    0.000 {method 'getrandbits' of 
'_random.Random' objects}
    11514    0.002    0.000    0.008    0.000 {built-in method 
_abc._abc_instancecheck}
    11514    0.002    0.000    0.006    0.000 :121(__subclasscheck__)
   115140    0.005    0.000    0.005    0.000 {method 'add' of 'set' objects}
    11514    0.003    0.000    0.004    0.000 {built-in method 
_abc._abc_subclasscheck}
   115232    0.004    0.000    0.004    0.000 {method 'bit_length' of 'int' 
objects}
      151    0.003    0.000    0.003    0.000 {built-in method builtins.print}
```

This wasn't too helpful, so I tried with [line_profiler][1]:

```
Timer unit: 1e-06 s

Total time: 55.2079 s
File: poc-small.py
Function: trainPerceptron at line 4

Line #      Hits         Time  Per Hit   % Time  Line Contents
==
     4                                           @profile
     5                                           def 
trainPerceptron(perceptron, data):
     6      1214        301.0      0.2      0.0    learningRate = 0.002
     7      1214        255.0      0.2      0.0    weights = 
perceptron['weights']
     8      1214        114.0      0.1      0.0    error = 0
     9  13973840    1742427.0      0.1      3.2    for chunk in data:
    10  13973840    1655043.0      0.1      3.0        input = chunk['input']
    11  13973840    1487543.0      0.1      2.7        output = chunk['output']
    12
    13                                                 # 12x slower than 
equivalent JS
    14  13973840    1210755.0      0.1      2.2        sum_ = 0
    15 139738397   13821056.0      0.1     25.0        for key in input:
    16 139738397   13794656.0      0.1     25.0            v = weights[key]
    17 139738396   14942692.0      0.1     27.1            sum_ += v
    18
    19                                                 # 20x slower than 
equivalent JS
    20                                                 #sum_ = reduce(lambda 
acc, key: acc + weights[key], input)
    21
    22  13973839    1618273.0      

Re: Running asyncio.run() more than once

[gst]

Le mardi 14 mars 2023 à 02:32:23 UTC-4, Clint Olsen a écrit :
> We have an application that involves submitting hundreds to thousands of jobs 
> to a shared computing resource, and we're using asyncio to do so because it 
> is far less overhead than threading or multiprocessing for the bookkeeping 
> required to keep track of all these jobs. It makes extensive use of 
> asyncio.create_subprocess_exec(). This was developed mostly in Python 3.9.7. 
> 

I'm not asyncio expert or even not advanced user, but using a simple list to 
hold the jobs to execute and fill it as necessary after results gathering is 
not good ?

```
@async
def execute_jobs(jobs: List["Job"]):
while len(jobs) > 0:
# launch_job(s)
# gather_job(s)_result(s)
# append_jobs_if_desired
```

does not make the trick ?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: =- and -= snag

[aapost]

On 3/13/23 17:26, Morten W. Petersen wrote:


It went into negative numbers,
when that shouldn't have been possible.

Turns out I had a very small typo, I had =- instead of -=.

Isn't it unpythonic to be able to make a mistake like that?



That is why I tell Alice it is always best to stay positive and use 
x-=-1 to avoid situations like that. *shrugs*


--
https://mail.python.org/mailman/listinfo/python-list

Re: Tkinter and cv2: "not responding" popup when imshow launched from tk app

[Weatherby,Gerard]

Assuming you’re using opencv-python, I’d post query at 
https://github.com/opencv/opencv-python/issues.

From: Python-list  on 
behalf of John O'Hagan 
Date: Tuesday, March 14, 2023 at 6:56 AM
To: Python list 
Subject: Tkinter and cv2: "not responding" popup when imshow launched from tk 
app
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

Hi list

I'm trying to use cv2 to display images created as numpy arrays, from
within a tkinter app (which does other things with the arrays before
they are displayed as images). The arrays are colour-coded
visualisations of genomes and can be over a billion elements in size,
and I've found the PIL methods to display images in tkinter are too
slow and memory-heavy.

Here is minimal code that demonstrates the problem in the subject line:

import cv2
from tkinter import *

images=['a.jpg', 'b.jpg', 'c.jpg'] #change to image paths

cv2.namedWindow('W', cv2.WND_PROP_FULLSCREEN)
cv2.setWindowProperty('W', cv2.WND_PROP_FULLSCREEN,
cv2.WINDOW_FULLSCREEN)
counter=[0]
def show():
   cv2.imshow('W', cv2.imread(images[counter[0] % len(images)]))
   cv2.waitKey(1)
   counter[0] += 1

root=Tk()
root.wm_attributes("-topmost", 1)
Button(root, text=' Show ', command=show).pack()
mainloop()

It works up to a point - I can cycle through the images by clicking the
button - but if I mouse-click on the displayed image (e.g. to use the
zooming and panning features of cv2), nothing happens, and a few
seconds later the image greys out and a popup appears saying "'Unknown'
is not responding" and giving the option of waiting or forcing close
(but sometimes these options are greyed out too). Clicking "wait", if
available, closes the popup but it comes back a few seconds later. If I
then click on the tkinter window titlebar, the popup changes to "'Tk'
is not responding". Clicking on the button still works and after a few
clicks the popup closes.

This happens under both x11 and wayland, but under wayland, I  also get
this error:

"QSocketNotifier: Can only be used with threads started with QThread
qt.qpa.wayland: Wayland does not support QWindow::requestActivate()"

and only every second button press displays a new image ,with only
every second image displayed.

I think this particular popup is a Gnome thing, but AIUI most DEs have
something similar to detect stuck apps. But the app is not stuck.

I suspect this is some kind of interaction between the call to
cv2.waitKey (which is necessary but I've never understood why!) and the
tkinter event loop, but it's beyond my knowledge.

Any suggestions about causes or workarounds?

Thanks

--

John


--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iHFg1AtgcwsfEeHXaH_Nasebf9SGreVlDs-DevEIQbFiwUQThx-_rah1QkSHRJEotJFyd-d6OCQ3GuQa1MxvsnGA$
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tkinter and cv2: "not responding" popup when imshow launched from tk app

[Thomas Passin]

On 3/14/2023 6:54 AM, John O'Hagan wrote:

Hi list

I'm trying to use cv2 to display images created as numpy arrays, from
within a tkinter app (which does other things with the arrays before
they are displayed as images). The arrays are colour-coded
visualisations of genomes and can be over a billion elements in size,
and I've found the PIL methods to display images in tkinter are too
slow and memory-heavy.

Here is minimal code that demonstrates the problem in the subject line:

import cv2
from tkinter import *

images=['a.jpg', 'b.jpg', 'c.jpg'] #change to image paths

cv2.namedWindow('W', cv2.WND_PROP_FULLSCREEN)
cv2.setWindowProperty('W', cv2.WND_PROP_FULLSCREEN,
cv2.WINDOW_FULLSCREEN)
counter=[0]
def show():
cv2.imshow('W', cv2.imread(images[counter[0] % len(images)]))
cv2.waitKey(1)
counter[0] += 1

root=Tk()
root.wm_attributes("-topmost", 1)
Button(root, text=' Show ', command=show).pack()
mainloop()

It works up to a point - I can cycle through the images by clicking the
button - but if I mouse-click on the displayed image (e.g. to use the
zooming and panning features of cv2), nothing happens, and a few
seconds later the image greys out and a popup appears saying "'Unknown'
is not responding" and giving the option of waiting or forcing close
(but sometimes these options are greyed out too). Clicking "wait", if
available, closes the popup but it comes back a few seconds later. If I
then click on the tkinter window titlebar, the popup changes to "'Tk'
is not responding". Clicking on the button still works and after a few
clicks the popup closes.

This happens under both x11 and wayland, but under wayland, I  also get
this error:

"QSocketNotifier: Can only be used with threads started with QThread
qt.qpa.wayland: Wayland does not support QWindow::requestActivate()"

and only every second button press displays a new image ,with only
every second image displayed.

I think this particular popup is a Gnome thing, but AIUI most DEs have
something similar to detect stuck apps. But the app is not stuck.

I suspect this is some kind of interaction between the call to
cv2.waitKey (which is necessary but I've never understood why!) and the
tkinter event loop, but it's beyond my knowledge.

Any suggestions about causes or workarounds?



I don't know anything about the specifics here, but with billions of 
elements you will not be able to show more than a small fraction on the 
screen.  So I think that a progressive disclosure approach would pay 
off.  Sample the arrays down to a more workable size before creating the 
screen images.  If you want to zoom in, resample them and recreate new 
images that only cover the zoomed in region in more detail.


It would also be useful to cache the generated images so they can be 
re-displayed without needing to be regenerated each time.


--
https://mail.python.org/mailman/listinfo/python-list

Tkinter and cv2: "not responding" popup when imshow launched from tk app

[John O'Hagan]

Hi list

I'm trying to use cv2 to display images created as numpy arrays, from
within a tkinter app (which does other things with the arrays before
they are displayed as images). The arrays are colour-coded
visualisations of genomes and can be over a billion elements in size,
and I've found the PIL methods to display images in tkinter are too
slow and memory-heavy.

Here is minimal code that demonstrates the problem in the subject line:

import cv2
from tkinter import *

images=['a.jpg', 'b.jpg', 'c.jpg'] #change to image paths

cv2.namedWindow('W', cv2.WND_PROP_FULLSCREEN)
cv2.setWindowProperty('W', cv2.WND_PROP_FULLSCREEN, 
cv2.WINDOW_FULLSCREEN)
counter=[0]
def show():
   cv2.imshow('W', cv2.imread(images[counter[0] % len(images)]))
   cv2.waitKey(1)
   counter[0] += 1

root=Tk()
root.wm_attributes("-topmost", 1)
Button(root, text=' Show ', command=show).pack()
mainloop()

It works up to a point - I can cycle through the images by clicking the
button - but if I mouse-click on the displayed image (e.g. to use the
zooming and panning features of cv2), nothing happens, and a few
seconds later the image greys out and a popup appears saying "'Unknown'
is not responding" and giving the option of waiting or forcing close
(but sometimes these options are greyed out too). Clicking "wait", if
available, closes the popup but it comes back a few seconds later. If I
then click on the tkinter window titlebar, the popup changes to "'Tk'
is not responding". Clicking on the button still works and after a few
clicks the popup closes.

This happens under both x11 and wayland, but under wayland, I  also get
this error:

"QSocketNotifier: Can only be used with threads started with QThread
qt.qpa.wayland: Wayland does not support QWindow::requestActivate()"

and only every second button press displays a new image ,with only
every second image displayed.

I think this particular popup is a Gnome thing, but AIUI most DEs have
something similar to detect stuck apps. But the app is not stuck.

I suspect this is some kind of interaction between the call to
cv2.waitKey (which is necessary but I've never understood why!) and the
tkinter event loop, but it's beyond my knowledge.

Any suggestions about causes or workarounds?

Thanks

--

John


-- 
https://mail.python.org/mailman/listinfo/python-list