Re: Terminal Emulator (Posting On Python-List Prohibited)

[Skip Montanaro via Python-list]

Modern debian (ubuntu) and fedora block users installing using pip.
>

Even if you're telling it to install in ~/.local? I could see not allowing
to run it as root.

I honestly haven't tried. Maybe I should... 🤔 I have an old laptop running
XUbuntu 22.04 which I generally only use to compile the most recent
branches on GitHub (main, 3.12, & 3.13 at the moment).

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PyCon

[Skip Montanaro via Python-list]

>
> > I’m at PyCon in Pittsburgh and I’m haven’t an amazing time!
>
> s/haven’t/having/
>

No need to explain/correct. We understand you are excited. Many of us have
been in the same state before. ;-)

Enjoy,

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: xkcd.com/353 ( Flying with Python )

[Skip Montanaro via Python-list]

>
> > https://xkcd.com/1306/
> >   what does  SIGIL   mean?
>
> I think its' a Perl term, referring to the $/@/# symbols in front of
> identifiers.
>

I had a vague recollection of hearing it elsewhere (*Game of Thrones,* on
the armies' battle flags?), but didn't know what it meant. Google tells me:

*an inscribed or painted symbol considered to have magical power.*

So, they're more than just line noise. They confer power on their users...

Perhaps '@' in the context of decorators is the most prominent example in
Python, since decorators technically don't allow the programmer to do
something they couldn't before, but are now are used everywhere, a key
feature of many applications and modules.

Magical-ly, y'rs,

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Testing (sorry)

[Skip Montanaro via Python-list]

>
> Here is a typical bounce message that I get:
>
> : host mail.python.org[188.166.95.178] said:
> 450-4.3.2
>  Service currently unavailable 450 4.3.2
>
> Some time after I get one of these messages I re-send the post.  Usually
> it gets through then.
>

Looks kinda like greylisting to me. I'm pretty sure that's one of the tool
in the mail.python.org chain.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Testing (sorry)

[Skip Montanaro via Python-list]

I can't explain the delays, but will note that the gate-news program on the
server runs every 5 minutes via cron. There are multiple moving parts in
the overall system. You'll probably get a more useful answer from
postmas...@python.org.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: test-ignore

[Skip Montanaro via Python-list]

>
> True, but did the poster really need to send another one to say "yes,
> that worked"?
>

Maybe to test the bidirectionality of the gateway? 🤷 If the messages stop
I think we can let it die. It's not like this sort of activity is a regular
occurrence. (A bigger problem for me was always Usenet posters who used
fake email addresses.)

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: test-ignore

[Skip Montanaro via Python-list]

>
> > Test post to see if my Newsgroup post program is working.
>
> Aim your test messages at alt.test, please.
>

I agree that basic Usenet connectivity messages should go to alt.test. It's
not clear from the original post, but if the poster's aim was to see if
posts to comp.lang.python traverse the gateway and show up on this list,
then alt.test won't help.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Skip Montanaro via Python-list]

> I do have several circular references. My experience is that if I do not
> take some action to break the references when closing the session, the
> objects remain alive. Below is a very simple program to illustrate this.
>
> Am I missing something? All comments appreciated.

Python has normal reference counting, but also has a cyclic garbage
collector. Here's plenty of detail about how it works:

https://devguide.python.org/internals/garbage-collector/index.html

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.11.5 Pip Issue

[Skip Montanaro via Python-list]

> I downloaded Python 3.11.5, and there was nothing in the “Scripts” file,
> and there was no Pip. I would like to know why.
>

Can't help with the empty/missing Scripts folder. Does running

python -m ensurepip

get you a working pip? Which you should then run as

python -m pip ...

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What to use instead of nntplib?

[Skip Montanaro]

>
> My understanding is that nntplib isn't being erased from reality,
> it's merely being removed from the set of modules that are provided
> by default.
>
> I presume that once it's removed from the core, it will still be
> possible to install it via pip or some other mechanism.
>

It won't magically be available via pip unless someone steps up to maintain
it as a PyPI package

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What to use instead of nntplib?

[Skip Montanaro]

> I got a nice warning today from the inews utility I use daily:
>
> DeprecationWarning: 'nntplib' is deprecated and slated for removal in 
> Python 3.13
>
> What should I use in place of nntplib?

I'd recommend creating a PyPI project with the existing 3.12 code,
then using that from 3.13 onward.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacement for nntplib on Python3

[Skip Montanaro]

You could create a git repo with just nntplib and generate a package on
PyPI, then use that when running a version of Python which lacks nntplib.

Skip

On Wed, Apr 26, 2023, 8:42 PM Retrograde  wrote:

> I used to use a script that relied on nntplib, which is currently still
> available in Python3 but scheduled for deprecation in the near future.
> What would be a suitable replacement?
>
> --
> Retrograde 
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Small lament...

[Skip Montanaro]

Just wanted to throw this out there... I lament the loss of waking up on
April 1st to see a creative April Fool's Day joke on one or both of these
lists, often from our FLUFL... Maybe such frivolity still happens, just not in
the Python ecosystem? I know you can still import "this" or "antigravity",
but those are now old (both introduced before 2010). When was the last time
a clever easter egg was introduced or an April Fool's Day joke played?

¯\_(ツ)_/¯

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: JOB | Lead Linux Sysadmin (Edinburgh/London)

[Skip Montanaro]

There's a link at the bottom of each message to the list info pager. Follow
the directions on that page to unsubscribe.

Skip

On Wed, Mar 8, 2023, 5:38 PM Thomas Gregg  wrote:

> Is there any way to be removed from this list?
> Thank you,  Tom
>
> On Wed, Mar 8, 2023 at 3:51 PM Skip Montanaro 
> wrote:
>
> > > Hello, I'm working with an employer that is looking to hire someone in
> > > (Edinburgh or London) that can administer on-prem and vmware
> > > platforms.
> > >
> >
> > James,
> >
> > If you haven't already, please post to the Phone Jobs Board:
> >
> > https://www.python.org/jobs/
> >
> > Skip
> >
> > >
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: JOB | Lead Linux Sysadmin (Edinburgh/London)

[Skip Montanaro]

> Hello, I'm working with an employer that is looking to hire someone in
> (Edinburgh or London) that can administer on-prem and vmware
> platforms.
>

James,

If you haven't already, please post to the Phone Jobs Board:

https://www.python.org/jobs/

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.10 Fizzbuzz

[Skip Montanaro]

Dang auto-correct... Should read

... double quotes around "strings" and single quotes around 'c'haracters ...

On Sun, Feb 26, 2023, 6:28 PM Skip Montanaro 
wrote:

> is there any reason to prefer"over'   ?
>>
>
> Not really. As an old C programmer for many years I used double
> quotes"around "strings" and single word around 'c'haracters, because that's
> what I was used to. (This was long before triple quoted strings appeared in
> the language.)
>
> Aside: Given all the various ways to quote strings for display, it irks me
> a bit to see repr() still use single quotes in all cases, which requires
> display of single quotes to be escaped. (In similar fashion, it would be a
> minor improvement in my mind if the repr() code used raw strings where they
> would simplify the display.)
>
> Skip
>
>>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.10 Fizzbuzz

[Skip Montanaro]

>
> is there any reason to prefer"over'   ?
>

Not really. As an old C programmer for many years I used double
quotes"around "strings" and single word around 'c'haracters, because that's
what I was used to. (This was long before triple quoted strings appeared in
the language.)

Aside: Given all the various ways to quote strings for display, it irks me
a bit to see repr() still use single quotes in all cases, which requires
display of single quotes to be escaped. (In similar fashion, it would be a
minor improvement in my mind if the repr() code used raw strings where they
would simplify the display.)

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Skip Montanaro]

> And yet, it appears that *something* changed between Python 2 and Python
3 such that it *is* atomic:

I haven't looked, but something to check in the source is opcode
prediction. It's possible that after the BINARY_OP executes, opcode
prediction jumps straight to the STORE_FAST opcode, avoiding the transfer
to the top of the virtual machine loop. That would (I think) avoid checks
related to GIL release and thread switches.

I don't guarantee that's what's going on, and even if I'm correct, I don't
think you can rely on it.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Skip Montanaro]

Thanks for the responses.

Peter wrote:

> Which OS is this?

MacOS Ventura 13.1, M1 MacBook Pro (eight cores).

Thomas wrote:

> I'm no expert on locks, but you don't usually want to keep a lock while
> some long-running computation goes on.  You want the computation to be
> done by a separate thread, put its results somewhere, and then notify
> the choreographing thread that the result is ready.

In this case I'm extracting the noun phrases from the body of an email
message (returned as a list). I have a collection of email messages
organized by month (typically 1000 to 3000 messages per month). I'm using
concurrent.futures.ThreadPoolExecutor() with the default number of workers (
os.cpu_count() * 1.5, or 12 threads on my system) to process each month, so
12 active threads at a time. Given that the process is pretty much CPU
bound, maybe reducing the number of workers to the CPU count would make
sense. Processing of each email message enters that with block once. That's
about as minimal as I can make it. I thought for a bit about pushing the
textblob stuff into a separate worker thread, but it wasn't obvious how to
set up queues to handle the communication between the threads created by
ThreadPoolExecutor() and the worker thread. Maybe I'll think about it
harder. (I have a related problem with SQLite, since an open database can't
be manipulated from multiple threads. That makes much of the program's
end-of-run processing single-threaded.)

> This link may be helpful -
>
> https://anandology.com/blog/using-iterators-and-generators/

I don't think that's where my problem is. The lock protects the generation
of the noun phrases. My loop which does the yielding operates outside of
that lock's control. The version of the code is my latest, in which I
tossed out a bunch of phrase-processing code (effectively dead end ideas
for processing the phrases). Replacing the for loop with a simple return
seems not to have any effect. In any case, the caller which uses the
phrases does a fair amount of extra work with the phrases, populating a
SQLite database, so I don't think the amount of time it takes to process a
single email message is dominated by the phrase generation.

Here's timeit output for the noun_phrases code:

% python -m timeit -s 'text = """`python -m timeit --help`""" ; from
textblob import TextBlob ; from textblob.np_extractors import
ConllExtractor ; ext = ConllExtractor() ; phrases = TextBlob(text,
np_extractor=ext).noun_phrases' 'phrases = TextBlob(text,
np_extractor=ext).noun_phrases'
5000 loops, best of 5: 98.7 usec per loop

I process the output of timeit's help message which looks to be about the
same length as a typical email message, certainly the same order of
magnitude. Also, note that I call it once in the setup to eliminate the
initial training of the ConllExtractor instance. I don't know if ~100us
qualifies as long running or not.

I'll keep messing with it.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Is there a more efficient threading lock?

[Skip Montanaro]

I have a multi-threaded program which calls out to a non-thread-safe
library (not mine) in a couple places. I guard against multiple
threads executing code there using threading.Lock. The code is
straightforward:

from threading import Lock

# Something in textblob and/or nltk doesn't play nice with no-gil, so just
# serialize all blobby accesses.
BLOB_LOCK = Lock()

def get_terms(text):
with BLOB_LOCK:
phrases = TextBlob(text, np_extractor=EXTRACTOR).noun_phrases
for phrase in phrases:
yield phrase

When I monitor the application using py-spy, that with statement is
consuming huge amounts of CPU. Does threading.Lock.acquire() sleep
anywhere? I didn't see anything obvious poking around in the C code
which implements this stuff. I'm no expert though, so could easily
have missed something.

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Upgrading Python on Ubuntu 22.04.1 LTS

[Skip Montanaro]

>
>
> I have a server with Ubuntu 22.04.1 LTS and the Python version there
> is Python 3.10.6. Is there a safe way to upgrade to the latest version of
> Python 3.10 (3.10.9)? I tried with the OS update and upgrade but the Python
> version remained the same.
>
> I have an Ubuntu 22.04 system. The installed Python 3.10 version is 3.10.7:

$ apt search python3 | egrep '^python3.10.*installed'


WARNING: apt does not have a stable CLI interface. Use with caution in
scripts.


python3.10/now 3.10.7-1+focal1 amd64 [installed,local]
python3.10-distutils/now 3.10.7-1+focal1 all [installed,local]
python3.10-doc/jammy-updates,jammy-security,now 3.10.6-1~22.04.2 all
[installed]
python3.10-gdbm/now 3.10.7-1+focal1 amd64 [installed,local]
python3.10-lib2to3/now 3.10.7-1+focal1 all [installed,local]
python3.10-minimal/now 3.10.7-1+focal1 amd64 [installed,local]
python3.10-venv/now 3.10.7-1+focal1 amd64 [installed,local]


Off the top of my head, I can't recall if it's LTS or not. If you want to
go beyond 3.10.6, it should be possible. As Grant indicated though,
upgrading packages on an Ubuntu system (of any flavor) is the province of
the Ubuntu community.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Android APK

[Skip Montanaro]

>
> Is there a good python library for converting python3 to android APK
>

I'm not an app type of person, so don't watch that space. I've heard of
Kivy:

https://kivy.org/

I also something called python-for-android on PyPI with a recent release:

https://pypi.org/project/python-for-android/

I'd be interested to see what else turns up.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: argparse — adding a --version flag in the face of positional args

[Skip Montanaro]

Thanks. It occurs to me that instead of providing two special actions
("help" and "version"), it might be worthwhile to provide a standard way of
saying, "if present, process this option and exit before considering other
details of the command line." Matt's example action works well enough for
my needs, but it would be nice if more than one such option could be given.
For example:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("url")
parser.add_argument("--version", version="777", action="version")

args = parser.parse_args(["--help", "--version"])

Which option is processed depends on their order on the command line. I
don't believe it's possible to run the script and see them both processed.
That's probably a secondary consideration though. My script is working well
enough in this regard now.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: argparse — adding a --version flag in the face of positional args

[Skip Montanaro]

>
> ummm, hate to say this, but have you checked the documentation?  this
> case is supported using an action named 'version' without doing very much.
>

Thanks, Mats.

I actually searched all over the argparse docs. (There's a lot to digest.
Honestly, if I wasn't attempting to be sort of up-to-date I'd just continue
using getopt.) It never occurred to me that "version" would be special, so
I didn't specifically search for it, or realize there would be a specific
action devoted to it. I knew (the default) "help" was special, so I focused
my search for it. Obviously, "--help" is a pretty bad search term.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: argparse — adding a --version flag in the face of positional args

[Skip Montanaro]

> class VersionAction(argparse.Action):
> def __call__(self, parser, namespace, values, option_string):
> print(VERSION)
> exit()
...
> parser.add_argument("-v", "--version", nargs=0, action=VersionAction)

Thanks. An action class didn't occur to me. I looked briefly at the
code for argparse to see how it handled --help. The added argument
seemed normal, so gave up, figuring there was some special handling of
that option.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

argparse — adding a --version flag in the face of positional args

[Skip Montanaro]

I have a script to which I'd like to add a --version flag. It should print
the version number then exit, much in the same way --help prints the help
text then exits. I haven't been able to figure that out. I always get a
complaint about the required positional argument.

I think I could use something like nargs='*', but that would push off
detection of the presence of the positional arg to the application.
Shouldn't I be able to tell argparse I'm going to process --verbose, then
exit?

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ref-strings in logging messages (was: Performance issue with CPython 3.10 + Cython)

[Skip Montanaro]

Dang autocorrect. Subject first word was supposed to be "f-strings" not
"ref-strings." Sorry about that.

S

On Fri, Oct 7, 2022, 10:45 AM Skip Montanaro 
wrote:

>
>
> On Fri, Oct 7, 2022 at 9:42 AM Andreas Ames 
> wrote:
>
>> 1. The culprit was me. As lazy as I am, I have used f-strings all over the
>> place in calls to `logging.logger.debug()` and friends, evaluating all
>> arguments regardless of whether the logger was enabled or not.
>>
>
> I thought there was some discussion about whether and how to efficiently
> admit f-strings to the logging package. I'm guessing that's not gone
> anywhere (yet).
>
> Skip
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Ref-strings in logging messages (was: Performance issue with CPython 3.10 + Cython)

[Skip Montanaro]

On Fri, Oct 7, 2022 at 9:42 AM Andreas Ames 
wrote:

> 1. The culprit was me. As lazy as I am, I have used f-strings all over the
> place in calls to `logging.logger.debug()` and friends, evaluating all
> arguments regardless of whether the logger was enabled or not.
>

I thought there was some discussion about whether and how to efficiently
admit f-strings to the logging package. I'm guessing that's not gone
anywhere (yet).

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Python gets a mention on Reddit - in r/tifu

[Skip Montanaro]

Not exactly where I expected to see Python mentioned on Reddit, but I found
this amusing (yes, despite the subreddit, it should be safe for work):

https://www.reddit.com/r/tifu/comments/wibmkh/tifu_by_teaching_my_13yo_brother_python/

Just a little diversion for a Sunday afternoon. I was particularly amused,
as my son is in dev ops, writes Python from time-to-time, and my grandson
is 13. I could definitely see the pattern in the story transferring over to
Chris and Carmine.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dictionary order?

[Skip Montanaro]

>
> So I decided to write a little test program to run on a variety of
> CPythons, to confirm what I was thinking.
>
> And instead I got a surprise.
>
> On 1.4 through 2.1 I got descending key order.  I expected the keys to be
> scattered, but they weren't.
>
> On 2.2 through 3.5 I got ascending key order.  I expected the keys to be
> scattered, but they weren't.
>
> On 3.6 through 3.10 I got insertion order, as expected.
>
> But why are 1.4 through 3.5 ordering so much?
>

That's long in the past, but I seem to recall that key order was
unspecified. That would give the implementer (likely Tim Peters much of the
time) the freedom to do whatever worked best for performance or simplicity
of implementation.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Resolving Weekday Schedules to Dates

[Skip Montanaro]

> Do you know of a library that resolves schedules like every Wednesday
> at 3:00pm to absolute time, that is return the datetime of the next
> occurrence?

Take a look at the `rrule` module in the `dateutil` package:

https://dateutil.readthedocs.io/en/stable/rrule.html

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Now for something completely different...

[Skip Montanaro]

This isn't really Python-related, but I don't wander around any more
general software engineering/computer science places on the web, so I will
throw this out here.

I start a framebuilding class next week (lugged steel construction for
those into bikes). After about three weeks of work, I should have a
custom-made frame, which will also happen to have been built by me. It will
need paint or powder coat and graphics. I'm looking for ideas about the
graphics.

Though I spent most of my professional life noodling around with Python
(I'm now retired), I don't really want the graphics to be terribly
Python-centric. Consequently, "import this" is probably not going to work.
I have decided to go ahead and use my first name in lower case Courier as
the core piece of the downtube graphic:

skip

That's not too informative (other than its relationship to moi), and I have
room for probably four or five more characters. (I have a graphic artist in
mind, so the space need not strictly be text either.)

So, how would you fill the remaining space? I've considered "n++" or "n+1",
but not thought of anything more than that. For those who are not into
bikes, you may not have heard of the Velominati Rules
. I quote Rule #12:

*The correct number of bikes to own is n+1.*
While the minimum number of bikes one should own is three, the correct
number is n+1, where n is the number of bikes currently owned. This
equation may also be re-written as s-1, where s is the number of bikes
owned that would result in separation from your partner.

I currently have six to ten, depending on how demanding you are that they
be ready to ride with nothing more than air in the tires. This project,
while definitely n+1 might well also be approaching s-1. :-)

Ideas about head tube graphics are also welcome. I have no idea there at
all yet. Again, some subtle reference to software engineering would be nice.

So, if you'd like to play along, toss out some ideas. I'll consider
everything as long as it's not NSFW.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do projects exist to audit PyPI-hosted packages?

[Skip Montanaro]

>
> A related problem is that even if a package is maintained by somebody with
> good intentions, the account might be hijacked by a malicious actor and
> since PyPi is separate from source control, people might not be able to
> find out easily and malware could spread through PyPi.
>

I hadn't considered that. Some sort of authenticated connection between the
source code hosting service and the PyPI user posting the package would be
nice.



Some other (only tangentially related) stuff occurs to me as I search for
useful bits...

I'd kinda be curious what hosting services other than GitHub or GitLab are
in common use. GNU Savannah? SourceForge? PyPI relevance isn't a terrific
indicator (I assume it uses Libraries.io's SourceRank to get a relevance
score), but it's still some kind of indicator how useful a package is.
Perhaps the PyPI BigQuery stuff has hosting info. I've not dug into it.
(Thinking that obscure hosting service might be a small knock against a
package, but that's just a thought. I realize not everyone is happy with
corporate hosting services.)

Having a decent idea what functional alternatives are out there to a
particular package would be nice as well. Again, considering pynput, I hit
Google up for "python packages similar to pynput" which led me here:

https://www.libhunt.com/r/pynput

I was unaware of its existence before. I have no idea how useful it might
be for narrowly focused packages like pynput. Something with application to
a much wider community, like numpy, returns a bunch more:

https://www.libhunt.com/r/numpy



Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Do projects exist to audit PyPI-hosted packages?

[Skip Montanaro]

I woke with a start in what amounted to the middle of the night (I really
need to get about three more hours of sleep, but you'll understand why I
was awake to write this).

Many years ago, so as to preserve my wrists, I wrote a tool
 to
monitor mouse and keyboard activity. It tells me when to rest. I use it
when I have problems, then put it away until it's needed again. I have
resurrected it a few times over the years, most recently a month or two
ago. Having never been all that fond of how I tracked keyboard and mouse
activity, I was happy when I stumbled upon pynput
. "Yay!", I thought. My worries are over.

Then extremely early this morning I woke thinking, "Damn, this runs on my
computer and it can see my mouse and keyboard activity. How do I know it's
not stealing my keystrokes?" Not going back to sleep after that. So, I'm
going through the code (and the Xlib package on which it relies) to make
myself more comfortable that there are no issues. Note: I am *most
certainly not* accusing the pynput author of any mischief. In fact, I
suspect there's no problem with the package. It's got a bunch of stars and
plenty of forks on GitHub (for what that's worth). I suspect the code has
had plenty of eyeballs looking at it. Still, I don't really know how well
vetted it might be, so I have no assurances of that. I saw it mentioned
somewhere (discuss I think?), checked it out, and thought it would solve my
activity tracking in a cross-platform way. (I currently only use an Xorg
environment, so while I am looking at the code, I'm not paying attention to
the Windows or MacOS bits either.)

This got me thinking. If I'm curious about pynput, might other people be as
well? What about other packages? I'm actually not worried about Python
proper or vulnerabilities which have already been found
. PyPI currently advertises that
it hosts over 373k packages. With that many hosted packages, it is almost
certainly a haven for some undetected vulnerabilities. Knowing which
packages have been audited — at least in a cursory fashion — could be used
as a further criterion to use when deciding which packages to consider
using on a project.

So, does something already exist (pointers appreciated)? Thx...

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Receive a signal when waking or suspending?

[Skip Montanaro]

> https://duckduckgo.com/?t=ffab&q=python+up+time pointed-out the
> "uptime — Cross-platform uptime library"
> - TLDR; I'm not sure if "uptime" and "boot" relate only to a
> 'cold-start' or if they include bringing the machine out of 'stand-by'
> or 'hibernation'

Yeah, that won't help. uptime(1) gives the time since the last boot.
Wake/suspend doesn't change the system's notion of boot time.

> https://duckduckgo.com/?t=ffab&q=python+battery+status pointed-out
> several ideas which require MS-Windows, but apparently the psutil
> library could be bent to your will:
>
https://www.geeksforgeeks.org/python-script-to-shows-laptop-battery-percentage/

I actually already have psutil installed. It gives you details about
various system bits (like battery status, which is why I installed it). It
doesn't seem to have anything that will notify about state changes, just
the current state (or — for example — continuous values like the battery
state of charge). I didn't see anything regarding wake/suspend, but I might
have missed it. Since my tool can't run while the system is suspended, it
would always see the system as "awake" but not know for how long it was in
that state. (I'm sure I could track in my tick routine and pay attention to
unusually long gaps.)

I think Marco's response gives me a bit better handle on what I need(*). Maybe
just an easier way to do things. For example, just before suspending and
just after waking, all scripts in /usr/lib/systemd/system-sleep/ are
executed. The only difference is the arguments with which they are called.
A script to suit my needs can just write a stream of records with
timestamps. It took just a minute or so:

#!/bin/sh

echo "`date --iso-8601=seconds` $1 $2" >> /tmp/suspensions
chmod 0644 /tmp/suspensions

Here's what /tmp/suspensions looks like after a couple cycles:

root@montanaro:/usr/lib/systemd/system-sleep# cat /tmp/suspensions
2022-04-23T19:17:41-05:00 pre suspend
2022-04-23T19:17:54-05:00 post suspend
2022-04-23T19:18:48-05:00 pre suspend
2022-04-23T19:19:04-05:00 post suspend


My tool can monitor that file for changes and keep track of the times of
state changes. It can thus track rest intervals across system suspensions
and know if I've been resting for a long enough period of time to allow me
to subject my wrists to further agony
 with
the keyboard and mouse.

Skip

(*) man systemd-sleep contains this admonition:

Note that scripts or binaries dropped in /lib/systemd/system-sleep/ are
intended for local use only and *should be considered hacks*. If
applications want to react to system suspend/hibernation and resume,
they should
rather use the Inhibitor interface[1].


A hack should suit my needs just fine. I'll leave more elegant solutions to
others. :-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Receive a signal when waking or suspending?

[Skip Montanaro]

> I don't know in Python, but maybe you can create a script that writes
> on a named pipe and read it from Python?
> https://askubuntu.com/questions/226278/run-script-on-wakeup

Thanks, that gives me something to munch on.
-- 
https://mail.python.org/mailman/listinfo/python-list

Receive a signal when waking or suspending?

[Skip Montanaro]

It's not clear there is a straightforward way to catch a signal or get
an event notification when my computer (Dell running XUbuntu 20.04) is
about to sleep or when it's just awakened. The app uses tkinter. Is
there some more-or-less easy way to do this? Mac support would be nice
(I have my eye on a new MacBook one of these days). I suppose bonus
points for something which works on Windows, but that's not a platform
I actually care about.

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: asyncio+tkinter

[Skip Montanaro]

>
> > Given that both asyncio & tkinter are modules in the standard lib and
> both
> > have event loops, I would have expected to find some "best practice"
> > solution to mixing the two.
>
> Agreed. For GTK, you can use a dedicated loop policy like this:
>
> import asyncio_glib
> asyncio.set_event_loop_policy(asyncio_glib.GLibEventLoopPolicy())
>
> It should be possible to make it this easy for asyncio + tkinter.
>

I haven't used GTK in ages and ages, but I do still have a tkinter program
that I drag out every now and then when my wrists start acting up. After
opining about this on the PyIdeas discuss forum, I spent a few minutes and
came up with a simple AsyncTk subclass of tkinter.Tk. Here's its hello
world:

https://gist.github.com/smontanaro/5e12c557602a76c609e46ca59387ad1c

I modified my activity watcher to use it:

https://github.com/smontanaro/python-bits/blob/master/src/watch.py

A few warnings:

   1. copy/paste reuse of that AsyncTk class
   2. the code is a mess (it's probably 20-25 years old, I make no
   apologies for the mess)
   3. as of about an hour ago it now uses pynput to watch for mouse/kbd
   watching

So, Tk+asyncio turns out to be fairly easy to do, at least for simple stuff.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Set tkinter top-level window to "always on visible workspace"

[Skip Montanaro]

> Also, wmctrl accepts window names instead of ids if you omit the -i
> option. In case winfo_id() is not returning the id you need.

Nice. That worked. FWIW, Tk never gave me the very outermost window as the
parent of the tkinter.Tk instance. Here's the start of hierarchy:

% xwininfo -tree

xwininfo: Please select the window about which you
  would like information by clicking the
  mouse in that window.

xwininfo: Window id: 0x5a0001f "Typing Watcher"

  Root window id: 0x7a2 (the root window) (has no name)
  Parent window id: 0xc73880 (has no name)
 1 child:
 0x5a0001e (has no name): ()  178x126+0+0  +2+43
1 child:
0x5a00020 (has no name): ()  178x126+0+0  +2+43
   5 children:
...

Tk thought 0x5a0001e was the top, but as you can see 0x5a0001f is the named
window. Calling wmctrl using "Typing Watcher" as the target window did the
trick.

Thanks,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Set tkinter top-level window to "always on visible workspace"

[Skip Montanaro]

> Would you accept a solution that involves a subprocess call?
>
> wmctrl -ir {id} -b add,sticky

I'm already checking idle time with xprintidle(1), so what's one more
(one-time) subprocess call?

Small amount of history, I'm referring to this mouse/typing watcher:

https://github.com/smontanaro/python-bits/blob/master/src/watch.py

I wrote this years and years ago when I first started having RSI problems.
So long ago that I just deleted a commented out piece of code which was
there in case I ever ran it with Python 1.4(!).

Over the years I've needed it on Linux, Linux+WIndows, Mac, etc. I'd put it
away for a couple years, then suffer a flare-up and pull it back out. I
never thought I'd need it after I retired, but here I am again. I've tried
all sorts of things to monitor mouse and keyboard activity. At the moment
I'm just using a Linux laptop, so have tossed out any sort of attempted
cross-platform tracking functionality.

> Now, the only problem is... figuring out your window ID. Worst case,
> parse wmctrl -lG to get that info, but it might be possible to get the
> window ID from Tkinter itself.

I'm struggling to get the outermost window (not sure what's going on), but
I will keep messing around.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Set tkinter top-level window to "always on visible workspace"

[Skip Montanaro]

> I think you must be using xfce4, not fvwm4 (there's an fvwm3 in
> development). See https://xfce.org/

Both?

% pgrep -fla xfwm4
1803 xfwm4 --replace
539426 xfwm4-settings

% pgrep -fla xfce
1599 xfce4-session
1755 /usr/bin/ssh-agent /usr/bin/im-launch startxfce4
1782 /usr/lib/x86_64-linux-gnu/xfce4/xfconf/xfconfd
...

I kind of assume xfce4 is the session manager sort of thing, while
xfwm4 is the actual window manager.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Set tkinter top-level window to "always on visible workspace"

[Skip Montanaro]

> I might be misguided, but on modern desktops that should be possible
> with a few mouseclicks. E.g. in KDE, there is a little pin icon
> displayed in every title bar of a toplevel window on the left side.

Correct, and that's what I'm currently doing. I'm lazy though. I want
the program to start always visible on the active virtual desktop. As
Cameron showed, this is possible. I have vague recollection of doing
this with the same program probably 15-20 years ago in an age when
window managers weren't largely configured with the mouse.
Unfortunately, I no longer recall what window manager(s) I used at the
time (probably twm or fvwm). Now I use fvwm4 and can't find squat
online about configuration files. I do have a ~/.config/xfce4/xfwm4/
directory, but it is completely empty.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Set tkinter top-level window to "always on visible workspace"

[Skip Montanaro]

> So you might tell your window manager to keep that window on the main
workspace.

Thanks. I'd forgotten about the possibility of doing this sort of thing in
the window manager config. That would certainly be fine in this case. (It's
been ages since I messed with this sort of thing.)

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Set tkinter top-level window to "always on visible workspace"

[Skip Montanaro]

I have a tkinter app (Ubuntu/X11 env) whose main window should always be
displayed on the currently visible workspace. Is there some way to set that
attribute programmatically? I see that there is a tkinter.Wm class and that
Toplevel widgets have a wm_attributes method, but haven't found any
examples in the library doc or on the wider net which demonstrate control
of this particular window manager interaction. (I don't care about Windows
or Mac, at least for the time being.)

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to detect an undefined method?

[Skip Montanaro]

> Let's say I have a Python app and have used an undefined method somewhere.
> Let
> us further assume I have not detected it thru my tests.
>
> Is there a way to detect it before deploying the app? pylint doesn't
> notice it.
>

This is maybe not exactly what you're looking for, but writing a test to
exercise that function call (if possible) would be top of my list. Next on
my list would be to use coverage to get a good idea of what code is not
tested. Writing more test cases to reduce the number of uncovered lines
will also make it easier to manually examine the rest of your (untested) code
to find calls to missing functions or methods.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

asyncio+tkinter

[Skip Montanaro]

Given that both asyncio & tkinter are modules in the standard lib and both
have event loops, I would have expected to find some "best practice"
solution to mixing the two. I've not used asyncio, but might find it useful
with the pynput module in the context of a Tk app. I see a few solutions
out in the wild, but this seems like something which might best be
addressed in either the asyncio or tkinter documentation, or better yet,
implemented in one or the other.

Skip Montanaro
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Gunicorn - HTTP and HTTPS in the same instance?

[Skip Montanaro]

Thanks all. I was hoping to get away without something more
sophisticated like NGINX. This is just a piddly little archive of an
old mailing list running on a single-core Ubuntu VM somewhere on the
East Coast. Speed is not a real requirement. Load balancing seemed
like overkill to me. Still, I guess if it has to be, then it has to
be.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Gunicorn - HTTP and HTTPS in the same instance?

[Skip Montanaro]

Hopefully some Pythonistas are also Gunicornistas. I've had little success
finding help with a small dilemma in the docs or in other more specific
sources.

I'm testing out a new, small website. It is just Gunicorn+Flask. I'd like
to both listen for HTTP and HTTPS connections. Accordingly, in my config, I
have the Gunicorn process bind to both ports 80 and 443 if running as root:

if IAM_ROOT:
bind = [
'0.0.0.0:443',
'0.0.0.0:80',
]
else:
bind = [
'0.0.0.0:8080',
]

Gunicorn listens on both ports, but insists on SSL/TLS chit chat over port
80, not just port 443 (which seems to work okay). Is there some magic
incantation to get it to just talk HTTP on port 80, or will I need to spin
up two instances? (The non-root config works fine - plain old HTTP over
port 8080.)

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: email.message_from_file & quoted printable

[Skip Montanaro]

> That sometimes you want the raw content as it was in the file? That if
> you have that choice (decoded or raw) the default should be False, as it
> does less?

Yeah, thinking back later, the round-trip possibility seemed plausible.
I'll stop overthinking this now...

> That we my _first_ instinct - I've never used get_payload(), myself, and
> I've written my own complete mailfiler :-)

I'm only trying to display email through a web browser, possibly with a
link to the raw message.


> Really, if a function isn't doing what you want you should always read
> the docs for that function specificly, even if you don't want to read
> the entire module docs (which is a big ask).

Yeah, my mental model was simply off. I frequently pull up docs in my
browser. I was thinking the message contents would be normalized when
reading. It didn't occur to me the transformation would occur on the way
out.

I think another way I might have saved myself was if I was using a modern
IDE where I might have gotten some hints hovering over the method call.
But, I'm an End user. I'm sure there is some add-on package I could
install, but I haven't looked.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: email.message_from_file & quoted printable

[Skip Montanaro]

Cameron> Try decode=True.

Skip> :dopeslap: Thanks. Never been all that consistent reading documentation.

The more I think about it, the more I think maybe my lack of
documentation reading wasn't all that unreasonable. The content
transfer encoding and charset are properties of the message as a
whole. Iexpected the internal form of the message body to be already
decoded. Am I missing some key bit of perspective here?

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: email.message_from_file & quoted printable

[Skip Montanaro]

>
> From the docs:
>
>  get_payload(i=None, decode=False)


...

Try decode=True.


:dopeslap: Thanks. Never been all that consistent reading documentation.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

email.message_from_file & quoted printable

[Skip Montanaro]

I have a bunch of old email archives I'm messing around with, for example,
rendering them on-demand in HTML. Some of the files use quoted printable
content transfer encoding. Here's one (with a number of headers elided):

>From classicrendezvous-admin  Mon Dec  4 15:29:22 2000
Message-ID: <027801bfcdaa$fa175230$268cbcd4@stewart>
Reply-To: "Stewart Barrie" 
From: "Stewart Barrie" 
To: 
Date: Sat, 3 Jun 2000 23:28:00 +0100
MIME-Version: 1.0
X-StripMime: Non-text section removed by stripmime
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset="iso-8859-1"
Subject: {ClassicRend]Frame dent


Just inherited a nice 531 Claud Butler from the 60's I think. No =
eylelets and pencil stays. Unfortunately there is nice dent in the =
middle of the seat tube just above front mech clip height. Otherwise it = seems
fine.=20


Any thoughts on restoration? Are there any techniques for fixing dents =
in situ (I can hear the screams already) or are we definitely looking at =
a new seat tube? In which case, is it worth it?


Cheers



Stewart


--- StripMime Report -- processed MIME parts ---
multipart/alternative
  text/plain (text body -- kept)
  text/html
---


This message is stored in a file. I read it using email.message_from_file.
In the example above, the payload still retains the quoted printable bits.
Maybe it's not correctly encoded (no "=20" at the end of the second
paragraph, for example), but I would have thought the email package would
do what it could to decode things, but nope:

>>> print(msg.get_payload())
Just inherited a nice 531 Claud Butler from the 60's I think. No =
eylelets and pencil stays. Unfortunately there is nice dent in the =
middle of the seat tube just above front mech clip height. Otherwise it =
seems fine.=20


Any thoughts on restoration? Are there any techniques for fixing dents =
in situ (I can hear the screams already) or are we definitely looking at =
a new seat tube? In which case, is it worth it?


Cheers



Stewart


--- StripMime Report -- processed MIME parts ---
multipart/alternative

  text/plain (text body -- kept)
  text/html
---


Am I expecting too much from the email package when munching on crufty
20+yo archived email messages?

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sad news: Fredrik Lundh ("Effbot") has passed away

[Skip Montanaro]

Like many others, I'm saddened to hear of Fredrik Lundh's passing. I
vaguely recall meeting him just once, probably at a Python workshop,
before they grew big enough to be called conferences. Effbot.org was
my Tkinter, ElemenTree, and PIL reference and cheat sheet.

My attention to Python development has waxed and waned over the years.
Most of the time, the trip through the Python folder in my mail
program was generally pretty quick, hitting the 'd' key far more often
than I'd stop to read a message. There are only a few people whose
messages I'd always read. Effbot was one. In my opinion, Fredrik ranks
up there with Guido, Tim Peters and Barry Warsaw.

I went to effbot.org and saw the "on hiatus" message. Searching
through The Wayback Machine, it seems it went on hiatus in late
November, 2020. The 11 November 2020 snapshot appears to be the last
usable version:

https://web.archive.org/web/2020145627/http://effbot.org/

Probably worth a bookmark in your browser.

Rest easy /F ...

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Proliferation of Python packaging formats

[Skip Montanaro]

Is the proliferation of packaging formats in Python as nutzo as this author
believes?

https://drewdevault.com/2021/11/16/Python-stop-screwing-distros-over.html

Asking because I've never been in the business of releasing "retail" Python
applications or packages.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Package conflicts trying to install jpegdupes package

[Skip Montanaro]

Woo hoo! It's installed. The ultimate error was a missing turbojpeg.h
file. Thank goodness for the apt-file command. I was able to track
that down to the libturbojpeg0-dev package, install that, and after a
bit more fussing around now have jpegdupes installed.

Thanks for the help,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Package conflicts trying to install jpegdupes package

[Skip Montanaro]

>
> For the most part, a proper colorizing compiler can make the
> errors stand out among the spam of warnings, but it's hard when people
> copy and paste just the text.
>

In my defense, the entire traceback was red. :-) It's quite possible that
GCC colorized its error/warning messages, but colorizing might have been
suppressed by stderr being fed into a pipe, or by distutils tossing it out.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Package conflicts trying to install jpegdupes package

[Skip Montanaro]

>
> Those are all warnings. Are there any errors that follow them?
>

Maybe I just missed the actual errors, but the compiler exit status was 1,
so there must have been. I'll give it another try.


The package doesn't seem to have had a release since 2018, but the
> GitHub repository has had changes as recently as a month ago. You
> might have better luck cloning the repo and building that.
>

Thanks, good suggestion.

FWIW, I'm trying to whittle down at least 12,000 images to a more
manageable number for the sister of a good friend who recently passed away.
I've got a straightforward dedupe program, but need something which can
compare just the data chunk of JPEGs, ignoring the metadata. This program
apparently does that. Is like to avoid reinventing that wheel.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Package conflicts trying to install jpegdupes package

[Skip Montanaro]

I'd like to use the jpegdupes package  but
can't seem to get past compilation issued related to jpegtrans-cffi.
Attempts to install any available versions on my XUbuntu system crap out
with this C compiler error message:

 In file included from
jpegtran/__pycache__/_cffi__xd2d84bdexcdb1023.c:267:
src/epeg_private.h:71:17: warning: ‘_jpeg_init_source’ declared
‘static’ but never defined [-Wunused-function]
   71 | METHODDEF(void) _jpeg_init_source(j_decompress_ptr cinfo);
  | ^
src/epeg_private.h:72:20: warning: ‘_jpeg_fill_input_buffer’ declared
‘static’ but never defined [-Wunused-function]
   72 | METHODDEF(boolean) _jpeg_fill_input_buffer(j_decompress_ptr
cinfo);
  |^~~
src/epeg_private.h:73:17: warning: ‘_jpeg_skip_input_data’ declared
‘static’ but never defined [-Wunused-function]
   73 | METHODDEF(void) _jpeg_skip_input_data(j_decompress_ptr cinfo,
long num_bytes);
  | ^
src/epeg_private.h:74:17: warning: ‘_jpeg_term_source’ declared
‘static’ but never defined [-Wunused-function]
   74 | METHODDEF(void) _jpeg_term_source(j_decompress_ptr cinfo);
  | ^
src/epeg_private.h:76:17: warning: ‘_jpeg_init_destination’ declared
‘static’ but never defined [-Wunused-function]
   76 | METHODDEF(void) _jpeg_init_destination(j_compress_ptr cinfo);
  | ^~
src/epeg_private.h:77:20: warning: ‘_jpeg_empty_output_buffer’ declared
‘static’ but never defined [-Wunused-function]
   77 | METHODDEF(boolean) _jpeg_empty_output_buffer (j_compress_ptr
cinfo);
  |^
src/epeg_private.h:78:17: warning: ‘_jpeg_term_destination’ declared
‘static’ but never defined [-Wunused-function]
   78 | METHODDEF(void) _jpeg_term_destination (j_compress_ptr cinfo);
  | ^~

...
This seems like a rather odd error message for a presumably sort-of-stable
PyPI package. Looking at the version dependencies I see 2.6, 2.7, 3.3 or
PyPy. I thought nmybe I needed to try with Python2... Create a Conda
environment, attempt to install jpegdupes, only to discover it's Python 3
only.

So, I'm kind of stuck. Maybe I need to install Python 3.3 and try that? Any
other ideas?

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Track 3.9 instead of 3.10?

[Skip Montanaro]

I have a development branch in my fork of python/cpython, the
register2 branch of https://github.com/smontanaro/cpython. As I am
dealing with virtual machine internals I've found the changes to the
virtual machine between 3.9 and 3.10 too disruptive. I'd like to track
3.9 instead. How would I go about making the switch while minimizing
the number of (inevitable) conflicts? I was thinking of diffing my
current state against 3.10, then creating a new branch off 3.9 and
applying the diff to that. That makes most of the effort outside of
the view of git though, and won't necessarily minimize conflicts. Is
there some way to do this totally within the git infrastructure?

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: imaplib: is this really so unwieldy?

[Skip Montanaro]

> It's working (with Cyrus), but I have the feeling I'm doing it all wrong
> because it seems so unwieldy.

I have a program, Polly , which I
wrote to generate XKCD 936 passphrases. (I got the idea - and the name -
from Chris Angelico. See the README.) It builds its dictionary from emails
in my Gmail account which are tagged "polly" by a Gmail filter. I had put
it away for a few years, at which time it was still using Python 2. When I
came back to it, I wanted to update it to Python 3. As with so many 2-to-3
ports, the whole bytes/str problem was my stumbling block. Imaplib's API
(as you've discovered) is not the most Pythonic. I didn't spend much time
horsing around with it. Instead, I searched for higher-level packages,
eventually landing on IMAPClient .
Once I made the switch, things came together pretty quickly, due in large
part, I think, to its more sane API.

YMMV, but you're more than welcome to steal code from Polly.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bloody rubbish

[Skip Montanaro]

>
> Machine language is so much simpler, and you can code with just a hexpad.
>

Pshaa... All you need are front panel switches. ;-) (Yes, I had a professor
who required is to 'key' in our programs on the front panel, of a rack
mounted PDP-11 as I recall. Needless to say, we didn't use an assembler
either. We just wrote raw opcodes and their arguments on paper. This was in
the late 70s.)

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposal: Disconnect comp.lang.python from python-list

[Skip Montanaro]

> > Not that I support shutting down the Usenet/email gateway -- the
> > signal/noise ration seems fine to me.
> >
> On that I quite agree.  :-)

I remind people that the existing Usenet<->Mail gateway keeps the
mailing list software on mail.python.org from migrating to Mailman 3.
I'm sure Mark Sapiro and other Mailman maintainers would like to keep
moving away from Mailman 2. If the gateway (easier anonymity) and
Mailman 3 (maybe better list archives) are of interest to you, you
might check in with the Mailman dev list and see what would be
involved in porting the gateway code to MM3.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposal: Disconnect comp.lang.python from python-list

[Skip Montanaro]

> Are you saying that the messages that appear occasionally from people
> such as Ethan Furman claiming to be moderators and alleging that
> particular people have been banned or suspended are lies? And that the
> message I received once saying that my comp.lang.python post had been
> rejected from the list was also a lie?

I'm sorry, but I don't recall accusing you of lying, neither in my
words nor in my thoughts. If you think I was doing that, you must be
reading everything with a very jaundiced eye.

My issue was that we seem to disagree with the definition of
"moderation." Earlier in this thread it was suggested that
comp.lang.python.moderated be created and that some as-yet-unnamed
moderators would vet every single message to that group. I think of
moderation as things which humans do, not tool chains. I suppose the
python.org postmasters could disable all elements of that tool chain
and let the admins/moderators do it all manually. Still, I don't think
of the mailing list as moderated. Yes, Ethan is one of three listed
admins of the list. I wouldn't have mentioned him by name, but you
did. (I suppose his role common knowledge anyway.)

I see three held messages in the queue right now with received times
between 00:33 to 08:15 today (timezone isn't mentioned). I seriously
doubt Ethan or the other admins approved all the other messages which
have reached the list in that interval. Despite the existing tool
chain, a number of messages which shouldn't probably leak through
without being vetted by Ethan or the other admins.

I will offer one other bit of information. As I understand it, the
only reason python-list@python.org can't be migrated to Mailman 3 is
because it doesn't support the gateway between Usenet and mail. It's
worth considering for people thinking about whether or not to
disconnect the two. (I have no opinion on that subject. Clearly others
do.)

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposal: Disconnect comp.lang.python from python-list

[Skip Montanaro]

> Are you unaware that the group is already gatewayed to a moderated
> mailing list for which all of that work is already done?

What is this moderation of which you speak? I'm one of the Python
postmasters (I maintain the SpamBayes setup) and am aware of a
multi-part tool chain (black hole lists, SpamBayes, greylisting, etc),
but not of human moderation on a grand scale, just of the relatively
few posts which are held for the admins/moderators by SpamBayes. My
definition of "moderation" is that a human vets every post. That's
certainly not the case for python-list@python.org. Posts gated from
comp.lang.python to the mailing list only get passed through
SpamBayes. All other elements of the tool chain occur ahead of the
gateway.

If we are using two different definitions of "moderation" I think it
is important to be clear what we mean.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: About \033[m

[Skip Montanaro]

>Porque quando se usa formatac,ao de cores, o python nao consegue
>
   centralizar dentro da cadeia de 40 caracteres ?
>
>Tive que colocar 54 no parametro pois de alguma forma esta sendo
>considerado os caracteres de dentro do comando \033[m


Python doesn't know there is anything special about escape sequences (that
they take up no space on the screen). I think you will have better results
if you place the escape sequences in the format specifier:

 print("\033[7;30;43m{:^40}\033[m".format("Programac,ao Python"))

( haven't tried this, but I think it will work as you expect.)

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Formatter Questions

[Skip Montanaro]

> 1) Are there any major other formatters that I can/should look at? I see some 
> "online" pretty printers, but I'm after something I can run on whole 
> recursive directories of code.

I use Emacs, so I'm kind of biased, but I find its python-mode does a
good job, good enough that I wouldn't use a batch formatter to conform
to some other conventions, then wind up having a mixed set of
conventions after my next edit. I presume vim and all IDEs worth their
salt also do a suitable job of formatting code on-the-fly.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-ideas] Looking for people interested in a Python register virtual machine project

[Skip Montanaro]

> a) You're working with CPython bleeding edge.
> b) You find that (bleeding edge) adding extra chore.
> c) Nobody told you to work on bleeding (nor boss, nor a maintainer who
> said "I'll merge it once you've done"),
>
> Then: why do you complicate your task by working on bleeding edge?
> Could take not too old CPython version, e.g. 3.8/3.9, instead, and work
> with that.

I started this in the 3.9 alpha timeframe. Staying up-to-date wasn't
too difficult. It never occurred to me that the virtual machine would
undergo so much churn for 3.10, so I just stuck with main/master and
paid the price when the changes started to arrive in earnest. When the
3.10 branch is created I will take that off-ramp this time.

> Btw, from just "pep", it's unclear if, and how much, you reuse Victor's
> work. If not, why? (The answer is useful to contributors - you ask them
> to "reuse" your code - how it's regarding your reuse of code of folks
> who were before you?).

Both Victor's and my earlier work took place in the dim dark past.
There have been so many functional changes to the virtual machine that
directly reusing either old code base wasn't feasible. I do have a
copy of Victor's work though which I have referred to from
time-to-time. I just never tried to merge it with something recent,
like 3.9.

> > > explicitly. Would be interesting to read (in the following "pep"
> > > sections) what makes them "almost completely distinct".
> >
> > Well, sure. The main difference is the way two pairs of instructions
> > (say, BINARY_ADD vs BINARY_ADD_REG) get their operands and save their
> > result. You still have to be able to add two objects, call functions,
> > etc.
>
> I'd note that your reply skips answering question about calling
> convention for register-based VM, and that's again one of the most
> important questions (and one I'd be genuinely interested to hear).

I've not attempted to make any changes to calling conventions. It
occurred to me that the LOAD_METHOD/CALL_METHOD pair could perhaps be
merged into a single opcode, but I haven't really thought about that.
Perhaps there's a good reason the method is looked up before the
arguments are pushed onto the stack (call_function()?). In a
register-based VM there's no need to do things in that order.

> > The fact that I treat the current frame's stack space as registers
> > makes it pretty much impossible to execute both stack and register
> > instructions within the same frame.
>
> I don't see how that would be true (in general, I understand that you
> may have constraints re: that, but that's exactly why I bring that up -
> why do you have constraints like that?). Even existing Python VM allows
> to use both in the same frame, e.g. LOAD_FAST. It takes value of
> register and puts it on a stack.

Sure, but that's because it must. All operands must be on the stack.
My code does have a step where it tries to remove LOAD_FAST_REG and
STORE_FAST_REG opcodes. It's not very good though. Pesky implicit
references cause problems. Still, I am able to remove some of them.
This should get better over time. And, it is possible that at some
point I decide to add back in some stack space for stuff like calling
functions, constructing lists, etc.

> Do you mean details like need to translate stack-based instructions
> into 2 (or more) instructions of: a) actual register-register
> instruction and; b) stack pointer adjustment, so stack-based
> instructions still kept working?

Yes, but I see (I think) what you're getting at. If I continued to
maintain the stack pointer, in theory stack opcodes could exist along
with register opcodes.

> Yes, you would need to do that, until you fully switch to
> register-based ones. But then there's 2 separate tasks:
>
> 1. Make register VM work. (Should be medium complexity.)
> 2. Make it fast. (Likely will be hard.)

I'm not too worried about #2 yet. :-) And as demonstrated by the
current project's incompleteness, either I'm not up to medium
complexity tasks anymore or it's harder than you think. :-)

Some of the second step isn't too hard, I don't think. I already
mentioned eliding generated fast LOAD/STORE instructions, and in my
previous email mentioned copying constants from the code object to the
frame object on creation. I also think opcode prediction and fast
dispatch should be straightforward. I just haven't bothered with that
yet.

> If you want to achieve both right from the start - oh-oh, that may be
> double-hard.
>
> > Victor's implementation did things
> > differently in this regard. I believe he just allocated extra space
> > for 256 registers at the end of each frame, so (in theory, I suppose),
> > you could have instructions from both executed in the same frame.
>
> I hope you have a plan of how to deal with more than 256 registers,
> etc. Register VM adds a lot of accidental implementation complexity ;-).

One of the reasons I just reused the current stack space as my
register space harkens back to a thread with Tim Peters back in th

Re: [Python-ideas] Looking for people interested in a Python register virtual machine project

[Skip Montanaro]

Thanks for the response. I will try to address your comments inline.

> I guess it should be a good idea to answer what's the scope of this
> project - is it research one or "production" one? If it's research one,
> why be concerned with the churn of over-modern CPython versions?
> Wouldn't it be better to just use some scalable, incremental
> implementation which would allow to forward-port it to a newer version,
> if it ever comes to that?

The motivation for revisiting this idea was/is largely personal. As I
indicated, I first messed around with it over 20 years ago and it's
been in the back of my mind ever since. Somehow I never lost the code
despite I'm not sure how many computers came and went and that the
code was never uploaded to any sort of distributed version control
system. I decided to pick things up again as a way to mostly keep my
head in the game after I retired. So, neither "research" nor
"production" seems to be a correct descriptor. Still, if taken to
functional completion — functional enough for performance testing and
application to more than just toy scripts — I realized pretty quickly
that I'd need help.

> Otherwise, if it's "production", who's the "customer" and how they
> "compensate" you for doing work (chasing the moving target) which is
> clearly of little interest to you and conflicts with the goal of the
> project?

Nobody is compensating me. I have no desire to try and turn it into
something I do for hire. Maybe I misunderstood your question?

> > This PEP proposes the addition of register-based instructions to the
> > existing Python virtual machine, with the intent that they eventually
> > replace the existing stack-based opcodes.
>
> Sorry, what? The purpose of register-based instructions is to just
> replace stack-based instructions? That's not what's I'd like to hear as
> the intro phrase. You probably want to replace one with the other
> because register-based ones offer some benefit, faster execution
> perhaps? That's what I'd like to hear instead of "deciphering" that
> between the lines.

Replacing stack-based instructions would be a reasonable initial goal,
I think. Victor reported performance improvements in his
implementation (also a translator). As I indicated in the "PEP" (I use
that term rather loosely, as I have no plans at the moment to submit
it for consideration, certainly not in its current, incomplete state),
a better ultimate way to go would be to generate register instructions
directly from the AST. The current translation scheme allows me to
write simple test case functions, generate register instructions, then
compare that when called the two produce the same result.

> > They [2 instruction sets] are almost completely distinct.
>
> That doesn't correspond to the mental image I would have. In my list,
> the 2 sets would be exactly the same, except that stack-based encode
> argument locations implicitly, while register-based - explicitly. Would
> be interesting to read (in the following "pep" sections) what makes them
> "almost completely distinct".

Well, sure. The main difference is the way two pairs of instructions
(say, BINARY_ADD vs BINARY_ADD_REG) get their operands and save their
result. You still have to be able to add two objects, call functions,
etc.

> > Within a single function only one set of opcodes or the other will
> > be used at any one time.
>
> That would be the opposite of "scalable, incremental" development
> approach mentioned above. Why not allow 2 sets to freely co-exist, and
> migrate codegeneration/implement code translation gradually?

The fact that I treat the current frame's stack space as registers
makes it pretty much impossible to execute both stack and register
instructions within the same frame. Victor's implementation did things
differently in this regard. I believe he just allocated extra space
for 256 registers at the end of each frame, so (in theory, I suppose),
you could have instructions from both executed in the same frame.

> > ## Motivation
>
> I'm not sure the content of the section corresponds much to its title.
> It jumps from background survey of the different Python VM optimizations
> to (some) implementation details of register VM - leaving "motivation"
> somewhere "between the lines".
>
> > Despite all that effort, opcodes which do nothing more than move data
> > onto or off of the stack (LOAD_FAST, LOAD_GLOBAL, etc) still account
> > for nearly half of all opcodes executed.
>
> ... And - you intend to change that with a register VM? In which way and
> how? As an example, LOAD_GLOBAL isn't going anywhere - it loads a
> variable by *symbolic* name into a register.

Certainly, if you have data which isn't already on the stack, you are
going to have to move data. As the appendix shows though, a fairly
large chunk of the current virtual machine does nothing more than
manipulate the stack (LOAD_FAST, STORE_FAST, POP_TOP, etc).

> > Running Pyperformance using a development version of Python 3.9
> > showed t

Re: [Python-ideas] Looking for people interested in a Python register virtual machine project

[Skip Montanaro]

> Yeah, that is old writing, so is probably less clear (no pun intended)
> than it should be. In frame_dealloc, Py_CLEAR is called for
> stack/register slots instead of just Py_XDECREF. Might not be
> necessary.

Also, the intent is not to change any semantics here. The
implementation of RETURN_VALUE_REG still Py_INCREFs the to-be-returned
value. It's not like the data can get reclaimed before the caller
receives it.

S
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-ideas] Looking for people interested in a Python register virtual machine project

[Skip Montanaro]

> In the "Object Lifetime" section you say "registers should be cleared upon 
> last reference". That isn't safe, since there can be hidden dependencies on 
> side effects of __del__, e.g.:
>
> process_objects = create_pipeline()
> output_process = process_objects[-1]
> return output_process.wait()
>
> If the process class terminates the process in __del__ (PyQt5's QProcess 
> does), then implicitly deleting process_objects after the second line will 
> break the code.

Yeah, that is old writing, so is probably less clear (no pun intended)
than it should be. In frame_dealloc, Py_CLEAR is called for
stack/register slots instead of just Py_XDECREF. Might not be
necessary.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-ideas] Looking for people interested in a Python register virtual machine project

[Skip Montanaro]

Yes, I remember Parrot. As I understand it their original goal was a
language-agnostic virtual machine, which might have complicated things.

I will do a bit of reading and add some text to the "PEP."

Skip

On Sat, Mar 20, 2021, 11:36 AM David Mertz  wrote:

> The Parrot project was also intended to be the same thing, and for a while
> had a fair number of contributors. Unfortunately, it never obtained the
> performance wins that were good for.
>
> On Sat, Mar 20, 2021, 11:55 AM Skip Montanaro 
> wrote:
>
>> Back in the late 90s (!) I worked on a reimagining of the Python
>> virtual machine as a register-based VM based on 1.5.2. I got part of
>> the way with that, but never completed it. In the early 2010s, Victor
>> Stinner got much further using 3.4 as a base. The idea (and dormant
>> code) has been laying around in my mind (and computers) these past
>> couple decades, so I took another swing at it starting in late 2019
>> after retirement, mostly as a way to keep my head in the game. While I
>> got a fair bit of the way, it stalled. I've picked it up and put it
>> down a number of times in the past year, often needing to resolve
>> conflicts because of churn in the current Python virtual machine.
>> Though I kept getting things back in sync, I realize this is not a
>> one-person project, at least not this one person. There are several
>> huge chunks of Python I've ignored over the past 20 years, and not
>> just the internals. (I've never used async anything, for example.) If
>> it is ever to truly be a viable demonstration of the concept, I will
>> need help. I forked the CPython repo and have a branch (register2) of
>> said fork which is currently synced up with the 3.10 (currently
>> master) branch:
>>
>> https://github.com/smontanaro/cpython/tree/register2
>>
>> I started on what could only very generously be called a PEP which you
>> can read here. It includes some of the history of this work as well as
>> details about what I've managed to do so far:
>>
>> https://github.com/smontanaro/cpython/blob/register2/pep-.rst
>>
>> If you think any of this is remotely interesting (whether or not you
>> think you'd like to help), please have a look at the "PEP". Because
>> this covers a fair bit of the CPython implementation, chances to
>> contribute in a number of areas exist, even if you have never delved
>> into Python's internals. Questions/comments/pull requests welcome.
>>
>> Skip Montanaro
>> ___
>> Python-ideas mailing list -- python-id...@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-id...@python.org/message/IUKZPH4ZSZ22RZFKMITQ3Q6A22P4BXWX/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Looking for people interested in a Python register virtual machine project

[Skip Montanaro]

Back in the late 90s (!) I worked on a reimagining of the Python
virtual machine as a register-based VM based on 1.5.2. I got part of
the way with that, but never completed it. In the early 2010s, Victor
Stinner got much further using 3.4 as a base. The idea (and dormant
code) has been laying around in my mind (and computers) these past
couple decades, so I took another swing at it starting in late 2019
after retirement, mostly as a way to keep my head in the game. While I
got a fair bit of the way, it stalled. I've picked it up and put it
down a number of times in the past year, often needing to resolve
conflicts because of churn in the current Python virtual machine.
Though I kept getting things back in sync, I realize this is not a
one-person project, at least not this one person. There are several
huge chunks of Python I've ignored over the past 20 years, and not
just the internals. (I've never used async anything, for example.) If
it is ever to truly be a viable demonstration of the concept, I will
need help. I forked the CPython repo and have a branch (register2) of
said fork which is currently synced up with the 3.10 (currently
master) branch:

https://github.com/smontanaro/cpython/tree/register2

I started on what could only very generously be called a PEP which you
can read here. It includes some of the history of this work as well as
details about what I've managed to do so far:

https://github.com/smontanaro/cpython/blob/register2/pep-.rst

If you think any of this is remotely interesting (whether or not you
think you'd like to help), please have a look at the "PEP". Because
this covers a fair bit of the CPython implementation, chances to
contribute in a number of areas exist, even if you have never delved
into Python's internals. Questions/comments/pull requests welcome.

Skip Montanaro
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Skip Montanaro]

>
> That's annoying. You have to roll your own solution!
>

Certainly seems like a known issue:

https://bugs.python.org/issue12737

That issue was opened in 2011.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-Dev] Re: Python 0.9.1

[Skip Montanaro]

> Also mind
> http://www.dalkescientific.com/writings/diary/archive/2009/03/27/python_0_9_1p1.html
> for result comparison.

Thanks, Paul. I had lost track of Andrew. Good to know he's still out
there. I wonder why his tar file was never sucked up into the
historical releases page.

Whew! My stupid little extraction script did a reasonable job. I see
plenty of differences, but a cursory examination shows they are only
in leading whitespace. Where I translated "\t" to TAB, it seems Andrew
used a suitable number of spaces. Python modules/scripts seem more
plausibly indented, and the couple I tried worked, so I'm a bit more
confident I have things right:

% PYTHONPATH=lib ./src/python
>>> import string
>>> print string.upper('hello world!')
HELLO WORLD!
>>>
% ./src/python lib/fact.py
9
[3, 3, 41, 271]
4096
[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]

The tests don't pass though. 1 * 1 raises an integer overflow exception:

>>> 1 * 1
Unhandled exception: run-time error: integer overflow
Stack backtrace (innermost last):
  File "", line 1

I'll let someone figure that out. :-)

At any rate, the git repo has been updated.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-Dev] Re: Python 0.9.1

[Skip Montanaro]

> If someone knows how to get the original Usenet messages from what Google 
> published, let me know.

Seems the original shar is there buried in a Javascript string toward
the end of the file. I think I've got a handle on it, though it will
take a Python script to massage back into correct format.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-Dev] Re: Python 0.9.1

[Skip Montanaro]

>
> Wow. Was white-space not significant in this release of Python? I see the
>> lack of indentation in the first Python programs.
>>
>
> Indentation most certainly was significant from day 0. I suspect what
> happened is that these files got busted somehow by the extraction process
> used by Skip or Hiromi.
>

Yes, that's certainly possible. While it's nice that Google has archived
this stuff, their faithfulness to the original formats leaves a bit to be
desired (and gmane still doesn't work for me, eliminating that option). Guido's
messages are displayed as HTML, and I saw no way to get at the raw Usenet
messages. I just copied the shar data and saved the result. It seems clear
that tabs copied as spaces. The Makefile indentation was hosed up. It
should have dawned on me that the .py, .c and .h files would be messed up
as well. I was only concerned with building the interpreter.

If someone knows how to get the original Usenet messages from what Google
published, let me know.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Python 0.9.1

[Skip Montanaro]

A note to webmas...@python.org from an astute user named Hiromi in
Japan* referred
us to Guido's shell archives for the 0.9.1 release from 1991. As that
wasn't listed in the historical releases README file:

https://legacy.python.org/download/releases/src/README

I pulled the shar files (and a patch), then made a few tweaks to get it to
build:

% ./python
>>> print 'hello world!'
hello world!
>>> import sys
>>> dir(sys)
['argv', 'exit', 'modules', 'path', 'ps1', 'ps2', 'stderr', 'stdin',
'stdout']
>>> sys.modules
{'builtin': ; 'sys': ; '__main__': }
>>> sys.exit(0)

I then pushed the result to a Github repo:

https://github.com/smontanaro/python-0.9.1

There is a new directory named "shar" with the original files, a small
README file and a compile.patch file between the original code and the
runnable code.

It was a pleasant diversion for a couple hours. I was tired of shovelling
snow anyway... Thank you, Hiromi.

Skip

*  Hiromi is bcc'd on this note in case he cares to comment. I didn't want
to publish his email beyond the bounds of the webmaster alias without his
permission.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: UTF-16 or something else?

[Skip Montanaro]

>
> It's UTF-8 with a UTF-16 BOM prepended, which is not uncommon when you
> have a file that's been converted to UTF-8 from UTF-16 or has been
> produced by shitty Microsoft software. You can tell instantly at a
> glance that it's not UTF-16 because the ascii dump would l.o.o.k.
> .l.i.k.e. .t.h.i.s.
>

Ah, right. Been a long, long while (well before Unicode was a thing) since
I needed to use od(1) and don't remember dealing with UTF-16 before.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: UTF-16 or something else?

[Skip Montanaro]

>
> Try setting encoding to: "utf-8-sig".
>
> 'eb bb bf' is the byte order mark for UTF8 (most systems do not include
> this in UTF-8 encoded files)
>
> Python will correctly read UTF8 BOMs if you use the 'utf-8-sig' encoding
> when reading files
>

Excellent, thanks. That worked like a charm. Knowing what its called also
allowed me to look up more info.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

UTF-16 or something else?

[Skip Montanaro]

I downloaded US hospital ICU capacity data this morning from this page:

https://healthdata.gov/dataset/covid-19-reported-patient-impact-and-hospital-capacity-facility

(The download link is about halfway down the page.)

Trying to read it using my personal CSV tools without specifying an
encoding, it failed to understand the first column, hospital_pk. That is
apparently because the file isn't simply ASCII or UTF-8. There are a few
bytes ahead of the "h". However, if I open the file using "utf-16" as the
encoding, Python complains there is no BOM. od(1) suggests there is
*something* ahead of the first column name, but it's three bytes, not two:

% od -A x -t x1z -v <
reported_hospital_capacity_admissions_facility_level_weekly_average_timeseries_20210207.csv
| head
00 *ef bb bf* 68 6f 73 70 69 74 61 6c 5f 70 6b 2c 63  >...hospital_pk,c<
10 6f 6c 6c 65 63 74 69 6f 6e 5f 77 65 65 6b 2c 73  >ollection_week,s<
20 74 61 74 65 2c 63 63 6e 2c 68 6f 73 70 69 74 61  >tate,ccn,hospita<
...

I'm opening the file like so:

inf = open(args[0], "r", encoding=encoding)

where encoding is passed on the command line. I know I can simply edit out
those bytes and probably be good-to-go, but I'd prefer not to. What should
I be passing for the encoding?

Skip, who thought everybody had effectively settled on utf-8 at this point,
but apparently not...
-- 
https://mail.python.org/mailman/listinfo/python-list

Log exception so traceback contains timestamp and level?

[Skip Montanaro]

The logging package can log exceptions and call stacks, but it does
(in my opinion) a suboptimal job of it. Consider this simple example:

>>> import logging
>>> FORMAT = '%(asctime)-15s %(levelname)s %(message)s'
>>> logging.basicConfig(format=FORMAT, force=True)
>>> log.warning("msg", stack_info=True)
2021-02-06 20:46:52,399 WARNING msg
Stack (most recent call last):
  File "", line 1, in 

It formats the warning message just fine, but simply dumps the
traceback onto the stream with no timestamp or level. For my purposes,
this is more important for exceptions. (I'm just using the stack trace
feature as it's easier in a small example.) I would like to call
something like

log.exception("Some message...")

and find something like this in the output stream:

2021-02-06 20:46:52,399 ERROR Some message...
2021-02-06 20:46:52,400 ERROR Traceback (most recent call last):
2021-02-06 20:46:52,402 ERROR   File "", line 1, in 

That way I can more easily grep log files for errors and get the
entire detail, including the traceback. It seems I have to subclass
logging.Formatter to override formatStack or formatException, then
construct the individual lines of output, set handler and formatter,
blah blah blah.

I'm pretty sure I've done this in the past by writing my own little
log_exception method which formats the exception and calls log.error
for each row of the traceback. That seems easier than the "right" way.
That seems harder than it ought to be. Hopefully I'm just missing
something simple.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Best practice for handling exceptions raised at lower levels?

[Skip Montanaro]

I'm curious if there are best practices for handling exceptions raised
by lower level modules and packages. For example, I wrote an
application called polly[1] which constructs a personalized dictionary
from email messages using IMAP4. That dictionary is then used as input
to a password generator based on the famous XKCD 936 comic[2]. The
idea is based on a similar tool Chris Angelico mentioned here several
years ago[3], though as not a D&D player, I didn't have player chat
available as user-specific input from which to construct a dictionary
and chose email instead.

So much for background. After a long period of stasis, I decided to
update polly to Python 3. While the 2-to-3 process was
straightforward, I had problems after the update which looked like
IMAP4 issues. I really didn't want to get knee deep in a pond full of
IMAP alligators, so rather than bookmarking the IMAP4 RFC[4], I cast
about for something higher level than imaplib, eventually settling on
IMAPClient[5]. I thus have a collection of modules and packages:
imapclient, imaplib, ssl, socket, and many other non-network bits.
IMAPClient did what I wanted, for the most part abstracting away IMAP4
details. I still have to craft a couple IMAP4 parameter strings, but
the rest of the interface is pretty Pythonic.

However... Network applications being what they are, hiccups are going
to happen. In the time since I swapped in the imapclient package, I've
also had to catch exceptions raised by lower level modules/packages I
wasn't using directly, discovering them only as they occurred. (Think
of it as lazy exception discovery. I suppose if I was developing the
application for others, I'd have been more proactive about uncovering
the exceptions.) In addition to a couple imapclient exceptions, I also
now need to catch socket.gaierror, ConnectionError and
imaplib.IMAP4.error.

Here's the crux of the problem. Where does responsibility generally
fall for low level exception handling? I don't mean to pick on
IMAPClient. It's just a recent example and got me thinking about the
problem. Is it (generally) the responsibility of the application
author or the package author? This isn't an issue just for network
applications with a number of moving parts (socket, ssl, higher level
packages, etc). It happens in almost all applications or packages,
even if it's just to deal with exceptions raised by internal data
structures.

Skip

[1] https://github.com/smontanaro/polly
[2] https://xkcd.com/936/
[3] https://mail.python.org/pipermail/python-list/2014-August/827854.html
[4] https://tools.ietf.org/html/rfc3501
[5] https://pypi.org/project/IMAPClient/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: IDE tools to debug in Python?

[Skip Montanaro]

>
> > Python is an interactive language. You can develop a lot while working
> > on a Python console. Then copy and paste into a program.
>
> Absolutely, the humble interactive prompt is often overlooked
> as a development tool. It's not as good as the "evaluate
> expression" tool in the Smalltalk workspace, say, but it's close.
>

In an earlier reply I failed to mention Jupyter. It's a pretty awesome
environment. Sort of takes the whole concept of "just insert prints" to a
new level. I started using it a bit at my last job for financial stuff, and
still mostly use it for that, but it's another viable option.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.

[Skip Montanaro]

CW> How do you troubleshooting/debugging in Python?

GE> Mostly I read exception trace and read the code and think about it.
GE> If that doesn't work, I add some print() or syslog() calls.

CW> I know hardcore computer scientists would tell me about Python debugger.

GE> I've been writing Python for 20+ years. I've never tried a debugger.

Agree with Grant on these points. I certainly use gdb to debug C code
(like the interpreter), but for Python code, tracebacks and print
statements pretty much take care of things. I know pdb has been around
for a long while and many people like IDEs, but I've not generally
found them all that useful. I'm stuck in the 90s I guess.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why sqrt is not a built-in function?

[Skip Montanaro]

> I don't understand why sqrt is not a built-in function.
> Why do we have to first import the function from the math module?
> I use it ALL THE TIME!

For one, it's specific to numeric types. You might use it all the
time, but I (for example) almost never need to use it, or to import
the math module for my Python work/play. OTOH, I use stuff from the
sys and csv modules (and lately, pandas) all the time. I import those
modules when necessary. If you don't need them, I suspect you might
grumble that they are useless for you. Requiring explicit module
import gives programmers more control over the content of their module
namespaces if the builtins don't include the kitchen sink.

If you skim the output of

import builtins
dir(builtins)

you should see that the two largest classes of builtin identifiers are
exceptions (ZeroDivisionError, etc) and types (list, set, etc). Other
classes include singleton constants (True, False, None) and general
sequence operations (useful for loop control - reversed, iter,
zip...). math.sqrt doesn't fit into those object classes. The
remainder are a mixture of things, but generally aren't quite as
special purpose as that. Granted, there are a couple which might be
better left out (IMO), like round and pow, but I suspect their
presence might simply be more a matter of them being present since
Python's earliest days and reflect a tendency later to avoid
gratuitous breakage of existing code.

Finally, should have never considered it, I think you might want to
study the output of

import this

Think on the second and last lines in particular.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python3 change logs

[Skip Montanaro]

Dang... I'm having very incomplete thoughts. Apologies for the multiple
replies when one would have sufficed.

https://docs.python.org/3/whatsnew/3.9.html

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python3 change logs

[Skip Montanaro]

Also, check www.python.org for "What's New" pages. I believe one is
generated for every release. It will be less detailed than change logs in
GitHub, but more reader friendly.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python3 change logs

[Skip Montanaro]

Change logs are kept as part of the source, I believe. Try browsing the
cpython GitHub repo:

https://github.com/python/cpython

Skip

On Tue, Dec 15, 2020, 10:05 AM Rich Shepard 
wrote:

> I've upgraded from Python-3.7.x to Python-3.9.x and want to learn about
> differences (if any) in tkinter between the two versions.
>
> Looking on the python.org web site I did not find change logs with the
> sources or under the other menus I checked. Please point me to any
> available
> logs.
>
> Regards,
>
> Rich
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to run Jupyter notebook in command line and get full error message?

[Skip Montanaro]

>
> My VPN keeps dropping and can not run Jupyter Notebook as it is.
>

You don't provide a lot of detail, but this seems similar to the kind of
flaky networking we used to deal with in the Before Times. Simply
connecting directly to a host over the Internet was often plagued by
disconnects. For that we used screen(1). You don't mention your operating
system. It's available for most/all Linux systems and Macs. If you're on
Windows I imagine you could get it using WSL.

Of course, I might be way off-base focusing on just the above statement
from your post. If so, you might want to provide some more details about
your problem.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Cannot marshal objects

[Skip Montanaro]

> Because the marshaling is happening in the guts of xmlrpc.

Okay, I misunderstood your original message. You used the word
"marshal" (and the xmlrpc error did as well). I thought you were
referring to the marshal module. I didn't understand (and ignored) the
references to the xmlrpc module. In Python 3, the Marshaller class has
migrated into the xmlrpc.client module.

Your code worked for me in 3.8. Note, however, that this is not going
to round trip properly. You will stuff in a Decimal object at one end
and get out a double at the other end. Recent versions support
unmarshalling to Decimal objects in the Unmarshaller class. I don't
know why the Marshaller class doesn't accept Decimal objects for
serialization. You could try this:

def dump_decimal(self, value, write):
write("")
write(str(value))
write("\n")

xmlrpc.client.Marshaller.dispatch[decimal.Decimal] = dump_decimal

I used the "ex:" prefix based on this document:

http://ws.apache.org/xmlrpc/types.html

YMMV. The tag name understood by the Unmarshaller class doesn't
include that prefix.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Cannot marshal objects

[Skip Montanaro]

> I am getting this error.

I assume you mean the email subject. It doesn't work in 3.8 either:

>>> import decimal
>>> d = decimal.Decimal(3.5)
>>> d
Decimal('3.5')
>>> import marshal
>>> marshal.dumps(d)
Traceback (most recent call last):
  File "", line 1, in 
ValueError: unmarshallable object

but that's not surprising to me. The marshal module is more-or-less
meant to serialize Python byte code. Pickle is more generally used for
object serialization. Why not use it?

>>> import pickle, decimal
>>> d = decimal.Decimal(3.5)
>>> pickle.dumps(d)
b'\x80\x04\x95!\x00\x00\x00\x00\x00\x00\x00\x8c\x07decimal\x94\x8c\x07Decimal\x94\x93\x94\x8c\x033.5\x94\x85\x94R\x94.'

That's meant more for serialization of data objects.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: constant folding - why not more

[Skip Montanaro]

>
> On the contrary, comparison remains for runtime:
> >>> dis.dis(compile('1 < 2', filename='', mode='eval',
> >>> optimize=2))
>   1   0 LOAD_CONST   0 (1)
>   2 LOAD_CONST   1 (2)
>   4 COMPARE_OP   0 (<)
>   6 RETURN_VALUE
> In function fold_unaryop (though comparison is a binary operation) in
> Python/ast_opt.c is remark: /* Fold not into comparison */
>
> Is there a reason why comparison (== != < > <= >=) is not folded?
>

I can think of two reasons. One, this kind of comparison will almost never
appear in production code (maybe in unit tests?). Unlike the C family of
languages, Python doesn't have a macro processor which would give symbolic
names to numeric constants or string literals. Code generators might
conceivably generate constant comparisons, but they might be able to easily
do constant folding of comparisons themselves.

Two, given that this sort of construct will almost never be found in the
wild, folding constant comparisons in the compiler would increase the
maintenance burden of the compiler (just slightly, but still...) with no
clear benefit.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help

[Skip Montanaro]

>
> When I run this, my answer is 8 but it should be 336 can some help ._.
>

Looks like you are returning from inside the loop.

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Artie 3000 the coding robot

[Skip Montanaro]

> >   Sounds like an expensive way to do turtle graphics...
> >
> Really expensive -- From what I could make out, the "robot" has no
> independent functioning. It is tethered (WiFi, maybe Bluetooth) to the
> controlling computer. The programs run on the computer and send command
> packets to the robot...

Thanks Dennis. My initial thought was that it probably used the turtle
module, but I didn't see it mentioned anywhere. Seems the "value add"
is just the communication module. I guess I'll pass.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Artie 3000 the coding robot

[Skip Montanaro]

I'm looking for feedback on Artie 3000 the Coding Robot. It's recommended
by wirecutter.com for introductory programming for ages seven and up
(though stemfinity.com
 says 7-12). I was
thinking about getting it for my grandson who's turning 12 next month. I
worry the introductory lessons might be kind of boring for him (though he's
very into Minecraft, so who knows), and I've seen nothing online about
transitioning to programming Artie using Python. His dad's a techie, so I'm
not worried if he gets stuck. Here's the manufacturer's page
 for
the tool/toy.

Any feedback appreciated...

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Pandas split Date in day month year and hour

[Skip Montanaro]

> I would like know how can I get for this array the values for day, month
> and hour:
>
> 2017-11-01 03:00:00   year = 2017  month = 11day = 1and
> hour = 3

Pandas has a datetime type. You should probably be using it. It's been
awhile (a year at least), but if your datetime column is filled with
datetime types, you should be able to append columns to your DataFrame
which correspond to individual components of the datetime instances.
Don't be looking outside of Pandas to Python's time or datetime
module.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Syntax question

[Skip Montanaro]

> Typing is not required by
> Python. However, you may find the extra error-checking helpful...

I haven't used type hints much, if at all, but my understanding is
that the "extra error-checking" of which you speak is gotten through
other static checkers, correct? I know the syntax was developed with
the MyPy folks (http://mypy-lang.org/), but I'm not sure what other
tools currently make use of it. Any pointers?

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explicit is better than Implicit

[Skip Montanaro]

>
> When Excel reads a file, it looks for stuff and decides to upgrade its
> type. Eg dates etc (particularly pernicious with US-style dates versus
> the rest of the planet).  Mojibake for data ensues.
>
> As always, I am reminded of Heuer's Razor:
>
> If it can't be turned off, it's not a feature. - Karl Heuer
>

Good one. I always remember the start-up days (late 90s) when I developed
and maintained an online concert calendar (Musi-Cal) written in Python. The
technology got bought by another start-up (Mojam) who used Perl for their
web stuff. Both front end systems talked to my Python-based back end
(communication between both front ends and the one back end was via
XML-RPC). I was sometimes frustrated by the stuff Perl did. The one which
stuck with me all these years was its silent conversion of the band name "
311 " to the integer 311 on which
my Python backend obligingly barfed. I eventually had to put in data type
checks for all fields in my back end (my front end already had that sort of
input validation) as I could no longer assume a sentient front end was
handing it data.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Explicit is better than Implicit

[Skip Montanaro]

Hmmm... Rename genes, fix Excel, or dump Excel in favor of Python? I know
what my choice would have been. :-)

https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates


Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Strings: double versus single quotes

[Skip Montanaro]

>
>
> May I ask why not simply like this:
>
> stmt = """
>   select foo from bar
> where a = 'bag'
>   and c = 'dog'
>   """
>

Sorry, I don't recall. I wouldn't be at all surprised that it has something
to do with Emacs's Python mode behavior. Emacs wouldn't know what to do
with spaces in the string, but knows where to put string literals within
the open parens. I'm pretty sure I was doing this before triple quoted
strings existed.

Thankfully, I don't need to mess around with SQL anymore. :-)

Skip

>
-- 
https://mail.python.org/mailman/listinfo/python-list