Re: [Python-Dev] Python startup time

[Gregory Szorc]

On Wed, May 2, 2018 at 8:26 PM, Benjamin Peterson 
wrote:

>
>
> On Wed, May 2, 2018, at 09:42, Gregory Szorc wrote:
> > The direction Mercurial is going in is that `hg` will likely become a
> Rust
> > binary (instead of a #!python script) that will use an embedded Python
> > interpreter. So we will have low-level control over the interpreter via
> the
> > C API. I'd also like to see us distribute a copy of Python in our
> official
> > builds. This will allow us to take various shortcuts, such as not having
> to
> > probe various sys.path entries since certain packages can only exist in
> one
> > place. I'd love to get to the state Google is at where they have
> > self-contained binaries with ELF sections containing Python modules. But
> > that requires a bit of very low-level hacking. We'll likely have a Rust
> > binary (that possibly static links libpython) and a separate JAR/zip-like
> > file containing resources.
>
> I'm curious about the rust binary. I can see that would give you startup
> time benefits similar to the ones you could get hacking the interpreter
> directly; e.g., you can use a zipfile for everything and not have site.py.
> But it seems like the Python-side wins would stop there. Is this all a
> prelude to incrementally rewriting hg in rust? (Mercuric oxide?)
>

The plans are recorded at https://www.mercurial-scm.org/wiki/OxidationPlan.
tl;dr we want to write some low-level bits in Rust but we anticipate the
bulk of the application logic remaining in Python.

Nobody in the project is seriously talking about a complete rewrite in
Rust. Contributors to the project have varying opinions on how aggressively
Rust should be utilized. People who contribute to the C code, low-level
primitives (like storage, deltas, etc), and those who care about
performance tend to want more Rust. One thing we almost universally agree
on is that we want to rewrite all of Mercurial's C code in Rust. I
anticipate that figuring out the balance between Rust and Python in
Mercurial will be an ongoing conversation/process for the next few years.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Terry Reedy]

On 5/2/2018 12:42 PM, Gregory Szorc wrote:

I know this kinda/sorta exists with zipimporter. But zipimporter uses 
zlib (slow) and only allows .py/.pyc files. And I think some Python 
application distribution tools have also solved this problem. I'd 
*really* like to see a proper/robust solution in Python itself. Along 
that vein, it would be really nice if the "standalone Python 
application" story were a bit more formalized. From my perspective, it 
is insanely difficult to package and distribute an application that 
happens to use Python. It requires vastly different solutions for 
different platforms. I want to declare a minimal boilerplate somewhere 
(perhaps in setup.py) and run a command that produces an 
as-self-contained-as-possible application complete with platform-native 
installers.


I few years ago I helped my wife create a tutorial in the Renpy visual 
storytelling engine.  It is free and open source.

https://www.renpy.org
It is written in Python, while users write scripts in both Python and a 
custom scripting language.


When we were done, we pressed a button and it generated self-contained 
zip files for Windows, Linux, and Mac.  This can  be done from any of 
the three platforms.  After we tested all three files, she created a web 
page with links to the three files for download.  There have been no 
complaints so far. Perhaps the file generators could be adapted to 
packaging a project directory into a self-contained app.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[INADA Naoki]

Recently, I reported how stdlib slows down `import requests`.
https://github.com/requests/requests/issues/4315#issuecomment-385584974

For Python 3.8, my ideas for faster startup time are:

* Add lazy compiling API or flag in `re` module.  The pattern is compiled
when first used.
* Add IntEnum and IntFlag alternative in C, like PyStructSequence for
namedtuple.
   It will make importing `socket` and `ssl` module much faster.  (Both
module has huge enum/flag).
* Add special casing for UTF-8 and ASCII in TextIOWrapper.  When
application uses only
   UTF-8 or ASCII, we can skip importing codecs and encodings package
entirely.
* Add faster and simpler http.parser (maybe, based on h11 [1]) and avoid
using email module in http module.

[1]: https://h11.readthedocs.io/en/latest/

I don't have significant estimate how they can make `import requests`
faster, but I believe most of these ideas
are worth enough.

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Benjamin Peterson]

On Wed, May 2, 2018, at 09:42, Gregory Szorc wrote:
> The direction Mercurial is going in is that `hg` will likely become a Rust
> binary (instead of a #!python script) that will use an embedded Python
> interpreter. So we will have low-level control over the interpreter via the
> C API. I'd also like to see us distribute a copy of Python in our official
> builds. This will allow us to take various shortcuts, such as not having to
> probe various sys.path entries since certain packages can only exist in one
> place. I'd love to get to the state Google is at where they have
> self-contained binaries with ELF sections containing Python modules. But
> that requires a bit of very low-level hacking. We'll likely have a Rust
> binary (that possibly static links libpython) and a separate JAR/zip-like
> file containing resources.

I'm curious about the rust binary. I can see that would give you startup time 
benefits similar to the ones you could get hacking the interpreter directly; 
e.g., you can use a zipfile for everything and not have site.py. But it seems 
like the Python-side wins would stop there. Is this all a prelude to 
incrementally rewriting hg in rust? (Mercuric oxide?)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Steven D'Aprano]

On Thu, May 03, 2018 at 12:01:53AM +0200, Antoine Pitrou wrote:
> On Wed, 2 May 2018 22:54:04 +0100
> Paul Moore  wrote:

> > Nor do I think the tone of his message here is acceptable - regardless
> > of how annoyed he is, posting insults ("no-one gives a damn") about
> > volunteer contributors in a public mailing list isn't reasonable or
> > constructive. Call that "playing speech police" if you want, but I
> > think that being offended or annoyed and saying so is perfectly
> > reasonable.
> 
> Will all due respect, it's sometimes unpredictable what kind of wording
> Anglo-Saxons will take as an insult, as there's lot of obsequiosity
> there that doesn't exist in other cultures. To me, "not give a damn"
> reads like a familiar version of "not care about something", but
> apparently it can be offensive.

I'm Anglo-Saxon[1], and honestly I believe that it is thin-skinned to 
the point of ludicrousness to say that "no-one gives a damn" is an 
insult. This isn't 1939 when Clark Gable's famous line "Frankly my dear, 
I don't give a damn" was considered shocking. Its 2018 and to not give a 
damn is a more forceful way of saying that people don't care, that they 
are indifferent.

It is a truism on the internet that nobody gets to decide for anyone 
else what they do or don't find offensive, but I think that the 
respectful and kind response is to interpret Ivan's statement as a cry 
of anguish and pain, to read it with at least a modicum of sympathy, 
rather than to read it as an insult and offensive accusation of 
indifference.

(And why should being accused of indifference be offensive? The world is 
full of things I have neither the time nor inclination to give a damn 
about. I deny that I ought to feel guilty or ashamed by that fact.)

I think Guido's response was great: acknowledge Ivan's pain (apparently 
he lost a job or some income) without attacking him, neither dismissing 
Ivan's feelings nor validating them as a tactic for getting his way. 
Thank you Guido for leading by example.

With respect to Paul, I literally cannot imagine why he thinks that 
*anyone*, not even the tkinter maintainers or developers themselves, 
ought to feel *offended* by Ivan's words.

But I think a clue might be his subsequent use of the word *annoyed*. Is 
it annoying to be told that "no-one cares" when in fact you care? Of 
course it can be. It is a perfectly reasonable to feel annoyed. But it 
isn't reasonable to lash out at every little annoyance.

All interpersonal interactions can involve annoyances. And none of us 
are purely on the receiving end, we all also cause them. None of us are 
so perfect that we can afford to lash out each time somebody causes some 
tiny little annoyance. We ought to gloss over the little ones, just as 
we hope others will swallow *their* annoyance at the things we do.

If we're going to be open, respectful and considerate, we have a duty 
not to have a hair-trigger "I'm offended" response at tiny annoyances.

"That's offensive!", in this day and age, is the nuclear weapon of 
interpersonal conflict, and nothing Ivan said was so terrible that it 
deserved such an attack. Not if we are to be open, considerate and 
respectful. We ought to start by respecting the clear emotional pain in 
his email and not responding by going on the attack. "A soft answer 
turns away wrath".



[1] By culture, not genetics.

-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Terry Reedy]

On 5/2/2018 6:01 PM, Antoine Pitrou wrote:

On Wed, 2 May 2018 22:54:04 +0100
Paul Moore  wrote:



His *other* email seems reasonable, and warrants a response, yes. But
are we to take the suggestion made here (to drop tkinter) seriously,
based on the fact that there's a (rare - at least it appears that the
many IDLE users haven't hit it yet) race condition that causes a crash
in Python 2.7? (It appears that the problem doesn't happen in the
python.org 3.x builds, if I understand the description of the issue).


I got the same impression, and indeed got no crashes in 25 tries.


I and others actually suggested it seriously in the past.  Now,
admittedly, at least IDLE seems better maintained than it used to
be -- not sure about Tkinter itself.


Serhiy continues to work on tkinter even though he now works on much 
else.  We just made an overlooked 2to3 fix last week.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Terry Reedy]

On 5/2/2018 4:51 PM, Ivan Pozdeev via Python-Dev wrote:

As https://bugs.python.org/issue33257


As I report there, the 'crasher' does not crash on my Win 10 with either 
installed 3.7 or built 3.8.



https://bugs.python.org/issue33316 showed,


nothing about tkinter

Tkinter is broken, for both 


One can crash CPython with legal Python code.  I don't think that the 
language community should deprecate and drop CPython ;-).


Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at 
least, and no-one gives a damn.
The experience of perhaps hundred of thousands of people successfully 
writing or running tkinter-based programs says otherwise.


This seems to be a testament that very few people are actually 
interested in or are using it.


It is a testament that most people write sane code, for which tkinter 
(and Python) work well.


> If that is so

But it is not.  Tkinter is actively maintained.

> there's no use keeping it in the standard library

Ridiculous.  All you have done with this post is distract attention from 
real problems.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASE] Python 3.7.0b4, final 3.7 beta, now available for testing

[Ned Deily]

Python 3.7.0b4 is the final beta preview of Python 3.7, the next feature
release of Python. Beta releases are intended to give you the
opportunity to test new features and bug fixes and to prepare your
projects to support the new feature release. We strongly encourage you
to test your projects with 3.7 during the beta phase and report issues
found to bugs.python.org as soon as possible. While the release is
feature complete entering the beta phase, it is possible that features
may be modified or, in rare cases, deleted up until the start of the
release candidate phase. Please keep in mind that this is a preview
release and its use is not recommended for production environments.
Attention macOS users: there is now a new installer variant for macOS
10.9+ that includes a built-in version of Tcl/Tk 8.6. This variant is
expected to become the default version when 3.7.0 releases. Check it out!

The next preview release will be the release candidate and is planned
for 2018-05-21 followed by the official release of 3.7.0, planned for
2018-06-15. You can find Python 3.7.0b4 and more information here:

https://www.python.org/downloads/release/python-370b4/

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-33257: seeking advice & approval on the course of action

[Terry Reedy]

On 5/2/2018 4:38 PM, Ivan Pozdeev via Python-Dev wrote:

The bottom line is: Tkinter is currently broken


This is way over-stated.  Many modules have bugs, somethings in features 
more central to their main purpose.



-- as in, it's not thread-safe (in both Py2 and Py3)


Meaning that tkinter calls in threads sometimes work, and sometimes not. 
Most people do not think of trying this, and are therefore not affected. 
 Others who want do either play it safe and desist or experiment to 
find out what does dependable work on their system.



despite being designed [as such].


Martin Loewis said this on a tracker issue several years ago, when he 
invited submission of patches he could review.  Too bad he is not active 
now that someone (you) finally submitted one.


The intention for Tkinter to be thread safe may have predated tcl/tk 
having a decent Mac version.


> and advertizing itself as such.

Where?  According to Firefox, the current 3.6 tkinter chapter does not 
contain the string 'thread'.


All the fix options require some redesign of either `_tkinter', or some 
of the core as well.


This should be discussed on the tracker.  Posting here was premature.

So, I'd like to get some kind of core team's feedback and/or approval 
before pursuing any of them.


Serhiy Storchaka is the tkinter maintainer.  He is aware of your patch,
https://github.com/python/cpython/pull/6444
having added himself as a reviewer.  Your comments since then on
https://bugs.python.org/issue33257
suggest that this is a first-draft patch that you yourself consider 
obsolete. In particular, your message today seems to, in effect, cancel 
the patch pending discussion of which fix option to pursue.  He might be 
waiting for you to push updates.


In any case, Serhiy is an extremely productive core developer, either 
submitting or merging nearly a patch a day.  I am sure he saw fixing 
other issues, including some other and older tkinter issues, before the 
releases this week, as higher priority.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[MRAB]

On 2018-05-02 22:56, Josh Stephens wrote:

Hello list,

   If I may voice my opinion I would like to say that I just built an
application using Tkinter using python3. I used it because it was
included in python by default and I didn't have to using something
like PyQT or any other framework that was heavy. While I agree that
the docs can sometimes be confusing, I am not sure that it warrants
tossing it out. I am not even sure that my opinion gives much weight
but I figured I would just toss in a quick here is my vote and my
story about using Tkinter with SqlAlchemy and Py2App to build a native
Mac OS X app as of last month.

I have a few applications that use tkiner. Whilst it has its 
limitations, it's too useful to throw out.


It's a battery, not a power station, but batteries have their uses.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Skip Montanaro]

I still use it a bit, in simple contexts to be sure, but I do find it
useful. Others think so as well. I think TkAgg is probably the most
commonly used backend in Matplotlib. I wrote a single Matplotlib-using
program which plots columns from CSV files. I use it almost daily with no
problems. Again, I use the TkAgg backend by default.

So, does it have problems? Almost certainly. It seems you've encountered
some. If you want to see something change though, just screaming at the
developers is almost certainly the least productive thing you could do.

Here are some things you *could* do:

* submit some bug reports
* review patches related to the problems, assuming there are some
* write some patches for the documentation adding warnings about sketchy
bits

Skip
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Ivan Pozdeev via Python-Dev]

On 03.05.2018 1:01, Antoine Pitrou wrote:

On Wed, 2 May 2018 22:54:04 +0100
Paul Moore  wrote:

On 2 May 2018 at 22:37, Antoine Pitrou  wrote:

To elaborate a bit: the OP, while angry, produced both a detailed
analysis *and* a PR.  It's normal to be angry when an advertised
feature doesn't work and it makes you lose hours of work (or, even,
forces you to a wholesale redesign). Producing a detailed analysis and a
PR is more than most people will ever do.

His *other* email seems reasonable, and warrants a response, yes. But
are we to take the suggestion made here (to drop tkinter) seriously,
based on the fact that there's a (rare - at least it appears that the
many IDLE users haven't hit it yet) race condition that causes a crash
in Python 2.7? (It appears that the problem doesn't happen in the
python.org 3.x builds, if I understand the description of the issue).
In 3.x, Tkinter+threads is broken too, albeit in a different way -- see 
https://bugs.python.org/issue33412 (this should've been the 2nd link in 
the initial message, sorry for the mix-up).
The 2.x bug also shows in 3.x if it's linked with a nonthreaded version 
of Tcl (dunno how rare that is, but the code still supports this setup).

I and others actually suggested it seriously in the past.  Now,
admittedly, at least IDLE seems better maintained than it used to
be -- not sure about Tkinter itself.


Nor do I think the tone of his message here is acceptable - regardless
of how annoyed he is, posting insults ("no-one gives a damn") about
volunteer contributors in a public mailing list isn't reasonable or
constructive. Call that "playing speech police" if you want, but I
think that being offended or annoyed and saying so is perfectly
reasonable.

Will all due respect, it's sometimes unpredictable what kind of wording
Anglo-Saxons will take as an insult, as there's lot of obsequiosity
there that doesn't exist in other cultures. To me, "not give a damn"
reads like a familiar version of "not care about something", but
apparently it can be offensive.

Confirm, never meant this as an insult.

I had to use emotional language to drive the point home that it's not 
some nitpick, it really causes people serious trouble (I lost a source 
of income, for the record).
Without the emotional impact, my message could easily be ignored as some 
noise not worth attention. This time, it's just too damn important to 
allow this possibility.


The module being abandoned and unused is truly the only explanation I 
could think of when seeing that glaring bugs have stayed unfixed for 15 
years (an infinity in IT), in an actively developed and highly used 
software.
This may be flattering for my ego, but if the module really is in any 
production use to speak of, then in all these years, with all this 
humongous user base, someone, somewhere in the world, at some point, 
should have looked into this. I don't even program in C professionally, 
yet was able to diagnose it and make a PR!


---

I'll make a PR with the doc warning as Guido suggested unless there are 
any better ideas.


Meanwhile, I'd really appreciate any response to my other message -- it 
is about actually fixing the issue, and I do need feedback to be able to 
proceed.
No need to delve all the way in and give an official authorization or 
something. I'm only looking for an opinion poll on which redesign option 
(if any) looks like the most reasonable way to proceed and/or in line 
with the big picture (the last one -- to provide a unifying vision -- is 
_the_ job of a BDFL IIRC).



Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru


--
Regards,
Ivan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Barry Warsaw]

On May 2, 2018, at 15:24, Gregory Szorc  wrote:
> 
> FWIW, Google has a patched glibc that implements dlopen_with_offset().
> It allows you to do things like memory map the current binary and then
> dlopen() a shared library embedded in an ELF section.
> 
> I've seen the code in the branch at
> https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/google/grte/v4-2.19/master.
> It likely exists elsewhere. An attempt to upstream it occurred at
> https://sourceware.org/bugzilla/show_bug.cgi?id=11767. It is probably
> well worth someone's time to pick up the torch and get this landed in
> glibc so everyone can be a massive step closer to self-contained, single
> binary applications. Of course, it will take years before you can rely
> on a glibc version with this API being deployed universally. But the
> sooner this lands...

Oh, I’m well aware of the history of this patch. :)  I’d love to see it 
available on the platforms I use, and agree it’s well worth someone’s time to 
continue to shepherd this through the processes to make that happen.  Even if 
it did take years to roll out, Python could use it with the proper compile-time 
checks.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Greg Ewing]

Guido van Rossum wrote:
So what do *you* think. Do you agree with the OP that Tkinter (and hence 
IDLE) should be scrapped?


I don't have an opinion on that, but the issue of
whether tkinter should be in the stdlib has been
debated at least once before, and I took the OP as
saying "maybe we should talk about that again".

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Gregory Szorc]

On 5/2/18 2:24 PM, Barry Warsaw wrote:
> On May 2, 2018, at 09:42, Gregory Szorc  wrote:
> 
>> As for things Python could do to make things better, one idea is for 
>> "package bundles." Instead of using .py, .pyc, .so, etc files as separate 
>> files on the filesystem, allow Python packages to be distributed as 
>> standalone "archive" files.
> 
> Of course, .so files have to be extracted to the file system, because we have 
> to live with dlopen()’s API.  In our first release of shiv, we had a loader 
> that did exactly that for just .so files.  We ended up just doing .pyz file 
> unpacking unconditionally, ignoring zip-safe, mostly because too many 
> packages still use __file__, which doesn’t work in a zipapp.

FWIW, Google has a patched glibc that implements dlopen_with_offset().
It allows you to do things like memory map the current binary and then
dlopen() a shared library embedded in an ELF section.

I've seen the code in the branch at
https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/google/grte/v4-2.19/master.
It likely exists elsewhere. An attempt to upstream it occurred at
https://sourceware.org/bugzilla/show_bug.cgi?id=11767. It is probably
well worth someone's time to pick up the torch and get this landed in
glibc so everyone can be a massive step closer to self-contained, single
binary applications. Of course, it will take years before you can rely
on a glibc version with this API being deployed universally. But the
sooner this lands...

> 
> I’ll plug shiv and importlib.resources (and the standalone 
> importlib_resources) again here. :)
> 
>> If you go this route, please don't require the use of zlib for file 
>> compression, as zlib is painfully slow compared to alternatives like lz4 and 
>> zstandard.
> 
> shiv works in a similar manner to pex, although it’s a completely new 
> implementation that doesn’t suffer from huge sys.paths or the use of 
> pkg_resources.  shiv + importlib.resources saves us 25-50% of warm cache 
> startup time.  That makes things better but still not ideal.  Ultimately 
> though that means we don’t suffer from the slowness of zlib since we don’t 
> count cold cache times (i.e. before the initial pyz unpacking operation).
> 
> Cheers,
> -Barry
> 
> 
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Ronald Oussoren]

> On 2 May 2018, at 22:51, Ivan Pozdeev via Python-Dev  
> wrote:
> 
> As https://bugs.python.org/issue33257 and https://bugs.python.org/issue33316 
> showed, Tkinter is broken, for both Py2 and Py3, with both threaded and 
> non-threaded Tcl, since 2002 at least, and no-one gives a damn.

The second issue number doesn’t refer to a Tkinter issue, the former is about a 
month old and has reactions from a core developer. That’s not “nobody cares”. 

> 
> This seems to be a testament that very few people are actually interested in 
> or are using it.

Not necessarily, it primarily reflects that CPython is volunteer-driven 
project.  This appears to be related to the interaction of Tkinter and threads, 
and requires hacking on C code.  That seriously shrinks the pool of people that 
feel qualified to work on this.

> 
> If that's so, there's no use keeping it in the standard library -- if 
> anything, because there's not enough incentive and/or resources to support 
> it. And to avoid screwing people (=me) up when they have the foolishness to 
> think they can rely on it in their projects -- nowhere in the docs it is said 
> that the module is only partly functional.

Tkinter is used fairly often as an easily available GUI library and is not much 
as you imply. 

I don’t know how save calling GUI code from multiple threads is in general 
(separate from this Tkinter issue), but do know that this is definitely not 
save across platforms: at least on macOS calling GUI methods in Apple’s 
libraries from secondary threads is unsafe unless those methods are explicitly 
documented as thread-safe.

Ronald

> 
> -- 
> 
> Regards,
> Ivan
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Antoine Pitrou]

On Wed, 2 May 2018 22:54:04 +0100
Paul Moore  wrote:
> On 2 May 2018 at 22:37, Antoine Pitrou  wrote:
> > To elaborate a bit: the OP, while angry, produced both a detailed
> > analysis *and* a PR.  It's normal to be angry when an advertised
> > feature doesn't work and it makes you lose hours of work (or, even,
> > forces you to a wholesale redesign). Producing a detailed analysis and a
> > PR is more than most people will ever do.  
> 
> His *other* email seems reasonable, and warrants a response, yes. But
> are we to take the suggestion made here (to drop tkinter) seriously,
> based on the fact that there's a (rare - at least it appears that the
> many IDLE users haven't hit it yet) race condition that causes a crash
> in Python 2.7? (It appears that the problem doesn't happen in the
> python.org 3.x builds, if I understand the description of the issue).

I and others actually suggested it seriously in the past.  Now,
admittedly, at least IDLE seems better maintained than it used to
be -- not sure about Tkinter itself.

> Nor do I think the tone of his message here is acceptable - regardless
> of how annoyed he is, posting insults ("no-one gives a damn") about
> volunteer contributors in a public mailing list isn't reasonable or
> constructive. Call that "playing speech police" if you want, but I
> think that being offended or annoyed and saying so is perfectly
> reasonable.

Will all due respect, it's sometimes unpredictable what kind of wording
Anglo-Saxons will take as an insult, as there's lot of obsequiosity
there that doesn't exist in other cultures. To me, "not give a damn"
reads like a familiar version of "not care about something", but
apparently it can be offensive.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Josh Stephens]

Hello list,

  If I may voice my opinion I would like to say that I just built an
application using Tkinter using python3. I used it because it was
included in python by default and I didn't have to using something
like PyQT or any other framework that was heavy. While I agree that
the docs can sometimes be confusing, I am not sure that it warrants
tossing it out. I am not even sure that my opinion gives much weight
but I figured I would just toss in a quick here is my vote and my
story about using Tkinter with SqlAlchemy and Py2App to build a native
Mac OS X app as of last month.

Best Regards,
Josh Stephens

On May 2, 2018 at 4:46:29 PM, Antoine Pitrou (anto...@python.org) wrote:
>
> I have no opinion about scrapping IDLE and Tkinter, but if we don't, I
> think his concerns deserve addressing instead of being dismissed by
> wielding the CoC magic wand.
>
> Regards
>
> Antoine.
>
>
> Le 02/05/2018 à 23:41, Guido van Rossum a écrit :
> > So what do *you* think. Do you agree with the OP that Tkinter (and hence
> > IDLE) should be scrapped?
> >
> > On Wed, May 2, 2018 at 2:37 PM, Antoine Pitrou > > > wrote:
> >
> > On Wed, 2 May 2018 23:28:22 +0200
> > Antoine Pitrou > wrote:
> > > On Wed, 02 May 2018 21:24:07 +
> > > Brian Curtin > wrote:
> > > > On Wed, May 2, 2018 at 16:55 Ivan Pozdeev via Python-Dev <
> > > > python-dev@python.org > wrote:
> > > >
> > > > > As https://bugs.python.org/issue33257
> > and
> > > > > https://bugs.python.org/issue33316
> > showed, Tkinter is broken, for both
> > > > > Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at
> > > > > least, and no-one gives a damn.
> > > > >
> > > > > This seems to be a testament that very few people are actually
> > > > > interested in or are using it.
> > > > >
> > > > > If that's so, there's no use keeping it in the standard library -- if
> > > > > anything, because there's not enough incentive and/or resources to
> > > > > support it. And to avoid screwing people (=me) up when they have the
> > > > > foolishness to think they can rely on it in their projects -- nowhere 
> > > > > in
> > > > > the docs it is said that the module is only partly functional.
> > > >
> > > >
> > > > For the future, this is not how you communicate with the development
> > > > mailing list of any open source software project. I would suggest 
> > > > reading
> > > > https://www.python.org/psf/codeofconduct/
> > for some pointers on how
> > people
> > > > typically behave around here in particular.
> > >
> > > Perhaps it would be more constructive to address the OP's point than to
> > > play speech police.
> >
> > To elaborate a bit: the OP, while angry, produced both a detailed
> > analysis *and* a PR. It's normal to be angry when an advertised
> > feature doesn't work and it makes you lose hours of work (or, even,
> > forces you to a wholesale redesign). Producing a detailed analysis and a
> > PR is more than most people will ever do.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> >
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido )
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/bsdtux%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Paul Moore]

On 2 May 2018 at 22:37, Antoine Pitrou  wrote:
> To elaborate a bit: the OP, while angry, produced both a detailed
> analysis *and* a PR.  It's normal to be angry when an advertised
> feature doesn't work and it makes you lose hours of work (or, even,
> forces you to a wholesale redesign). Producing a detailed analysis and a
> PR is more than most people will ever do.

His *other* email seems reasonable, and warrants a response, yes. But
are we to take the suggestion made here (to drop tkinter) seriously,
based on the fact that there's a (rare - at least it appears that the
many IDLE users haven't hit it yet) race condition that causes a crash
in Python 2.7? (It appears that the problem doesn't happen in the
python.org 3.x builds, if I understand the description of the issue).

I don't have an opinion on the proposed fixes to tkinter, but I
definitely don't think that dropping it is a reasonable option.

Nor do I think the tone of his message here is acceptable - regardless
of how annoyed he is, posting insults ("no-one gives a damn") about
volunteer contributors in a public mailing list isn't reasonable or
constructive. Call that "playing speech police" if you want, but I
think that being offended or annoyed and saying so is perfectly
reasonable.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Brian Curtin]

On Wed, May 2, 2018 at 5:37 PM Antoine Pitrou  wrote:

> On Wed, 2 May 2018 23:28:22 +0200
> Antoine Pitrou  wrote:
> > On Wed, 02 May 2018 21:24:07 +
> > Brian Curtin  wrote:
> > > On Wed, May 2, 2018 at 16:55 Ivan Pozdeev via Python-Dev <
> > > python-dev@python.org> wrote:
> > >
> > > > As https://bugs.python.org/issue33257 and
> > > > https://bugs.python.org/issue33316 showed, Tkinter is broken, for
> both
> > > > Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at
> > > > least, and no-one gives a damn.
> > > >
> > > > This seems to be a testament that very few people are actually
> > > > interested in or are using it.
> > > >
> > > > If that's so, there's no use keeping it in the standard library -- if
> > > > anything, because there's not enough incentive and/or resources to
> > > > support it. And to avoid screwing people (=me) up when they have the
> > > > foolishness to think they can rely on it in their projects --
> nowhere in
> > > > the docs it is said that the module is only partly functional.
> > >
> > >
> > > For the future, this is not how you communicate with the development
> > > mailing list of any open source software project. I would suggest
> reading
> > > https://www.python.org/psf/codeofconduct/ for some pointers on how
> people
> > > typically behave around here in particular.
> >
> > Perhaps it would be more constructive to address the OP's point than to
> > play speech police.
>
> To elaborate a bit: the OP, while angry, produced both a detailed
> analysis *and* a PR.  It's normal to be angry when an advertised
> feature doesn't work and it makes you lose hours of work (or, even,
> forces you to a wholesale redesign). Producing a detailed analysis and a
> PR is more than most people will ever do.
>

It may be normal to be angry when something doesn't work the way it should,
but analyzing and creating a PR aren't the gateway to normalizing this
behavior. Sending thousands of people this type of email isn't how it works.

To address their point: no, next topic.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Antoine Pitrou]

I have no opinion about scrapping IDLE and Tkinter, but if we don't, I
think his concerns deserve addressing instead of being dismissed by
wielding the CoC magic wand.

Regards

Antoine.


Le 02/05/2018 à 23:41, Guido van Rossum a écrit :
> So what do *you* think. Do you agree with the OP that Tkinter (and hence
> IDLE) should be scrapped?
> 
> On Wed, May 2, 2018 at 2:37 PM, Antoine Pitrou  > wrote:
> 
> On Wed, 2 May 2018 23:28:22 +0200
> Antoine Pitrou > wrote:
> > On Wed, 02 May 2018 21:24:07 +
> > Brian Curtin > wrote:
> > > On Wed, May 2, 2018 at 16:55 Ivan Pozdeev via Python-Dev <
> > > python-dev@python.org > wrote:
> > >   
> > > > As https://bugs.python.org/issue33257
>  and
> > > > https://bugs.python.org/issue33316
>  showed, Tkinter is broken, for both
> > > > Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at
> > > > least, and no-one gives a damn.
> > > >
> > > > This seems to be a testament that very few people are actually
> > > > interested in or are using it.
> > > >
> > > > If that's so, there's no use keeping it in the standard library -- 
> if
> > > > anything, because there's not enough incentive and/or resources to
> > > > support it. And to avoid screwing people (=me) up when they have the
> > > > foolishness to think they can rely on it in their projects -- 
> nowhere in
> > > > the docs it is said that the module is only partly functional.    
> > > 
> > > 
> > > For the future, this is not how you communicate with the development
> > > mailing list of any open source software project. I would suggest 
> reading
> > > https://www.python.org/psf/codeofconduct/
>  for some pointers on how
> people
> > > typically behave around here in particular.  
> > 
> > Perhaps it would be more constructive to address the OP's point than to
> > play speech police.
> 
> To elaborate a bit: the OP, while angry, produced both a detailed
> analysis *and* a PR.  It's normal to be angry when an advertised
> feature doesn't work and it makes you lose hours of work (or, even,
> forces you to a wholesale redesign). Producing a detailed analysis and a
> PR is more than most people will ever do.
> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> https://mail.python.org/mailman/listinfo/python-dev
> 
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org 
> 
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido )
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Guido van Rossum]

So what do *you* think. Do you agree with the OP that Tkinter (and hence
IDLE) should be scrapped?

On Wed, May 2, 2018 at 2:37 PM, Antoine Pitrou  wrote:

> On Wed, 2 May 2018 23:28:22 +0200
> Antoine Pitrou  wrote:
> > On Wed, 02 May 2018 21:24:07 +
> > Brian Curtin  wrote:
> > > On Wed, May 2, 2018 at 16:55 Ivan Pozdeev via Python-Dev <
> > > python-dev@python.org> wrote:
> > >
> > > > As https://bugs.python.org/issue33257 and
> > > > https://bugs.python.org/issue33316 showed, Tkinter is broken, for
> both
> > > > Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at
> > > > least, and no-one gives a damn.
> > > >
> > > > This seems to be a testament that very few people are actually
> > > > interested in or are using it.
> > > >
> > > > If that's so, there's no use keeping it in the standard library -- if
> > > > anything, because there's not enough incentive and/or resources to
> > > > support it. And to avoid screwing people (=me) up when they have the
> > > > foolishness to think they can rely on it in their projects --
> nowhere in
> > > > the docs it is said that the module is only partly functional.
> > >
> > >
> > > For the future, this is not how you communicate with the development
> > > mailing list of any open source software project. I would suggest
> reading
> > > https://www.python.org/psf/codeofconduct/ for some pointers on how
> people
> > > typically behave around here in particular.
> >
> > Perhaps it would be more constructive to address the OP's point than to
> > play speech police.
>
> To elaborate a bit: the OP, while angry, produced both a detailed
> analysis *and* a PR.  It's normal to be angry when an advertised
> feature doesn't work and it makes you lose hours of work (or, even,
> forces you to a wholesale redesign). Producing a detailed analysis and a
> PR is more than most people will ever do.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Antoine Pitrou]

On Wed, 2 May 2018 23:28:22 +0200
Antoine Pitrou  wrote:
> On Wed, 02 May 2018 21:24:07 +
> Brian Curtin  wrote:
> > On Wed, May 2, 2018 at 16:55 Ivan Pozdeev via Python-Dev <
> > python-dev@python.org> wrote:
> >   
> > > As https://bugs.python.org/issue33257 and
> > > https://bugs.python.org/issue33316 showed, Tkinter is broken, for both
> > > Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at
> > > least, and no-one gives a damn.
> > >
> > > This seems to be a testament that very few people are actually
> > > interested in or are using it.
> > >
> > > If that's so, there's no use keeping it in the standard library -- if
> > > anything, because there's not enough incentive and/or resources to
> > > support it. And to avoid screwing people (=me) up when they have the
> > > foolishness to think they can rely on it in their projects -- nowhere in
> > > the docs it is said that the module is only partly functional.
> > 
> > 
> > For the future, this is not how you communicate with the development
> > mailing list of any open source software project. I would suggest reading
> > https://www.python.org/psf/codeofconduct/ for some pointers on how people
> > typically behave around here in particular.  
> 
> Perhaps it would be more constructive to address the OP's point than to
> play speech police.

To elaborate a bit: the OP, while angry, produced both a detailed
analysis *and* a PR.  It's normal to be angry when an advertised
feature doesn't work and it makes you lose hours of work (or, even,
forces you to a wholesale redesign). Producing a detailed analysis and a
PR is more than most people will ever do.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Antoine Pitrou]

On Wed, 02 May 2018 21:24:07 +
Brian Curtin  wrote:
> On Wed, May 2, 2018 at 16:55 Ivan Pozdeev via Python-Dev <
> python-dev@python.org> wrote:
> 
> > As https://bugs.python.org/issue33257 and
> > https://bugs.python.org/issue33316 showed, Tkinter is broken, for both
> > Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at
> > least, and no-one gives a damn.
> >
> > This seems to be a testament that very few people are actually
> > interested in or are using it.
> >
> > If that's so, there's no use keeping it in the standard library -- if
> > anything, because there's not enough incentive and/or resources to
> > support it. And to avoid screwing people (=me) up when they have the
> > foolishness to think they can rely on it in their projects -- nowhere in
> > the docs it is said that the module is only partly functional.  
> 
> 
> For the future, this is not how you communicate with the development
> mailing list of any open source software project. I would suggest reading
> https://www.python.org/psf/codeofconduct/ for some pointers on how people
> typically behave around here in particular.

Perhaps it would be more constructive to address the OP's point than to
play speech police.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop/deprecate Tkinter?

[Brian Curtin]

On Wed, May 2, 2018 at 16:55 Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

> As https://bugs.python.org/issue33257 and
> https://bugs.python.org/issue33316 showed, Tkinter is broken, for both
> Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at
> least, and no-one gives a damn.
>
> This seems to be a testament that very few people are actually
> interested in or are using it.
>
> If that's so, there's no use keeping it in the standard library -- if
> anything, because there's not enough incentive and/or resources to
> support it. And to avoid screwing people (=me) up when they have the
> foolishness to think they can rely on it in their projects -- nowhere in
> the docs it is said that the module is only partly functional.


For the future, this is not how you communicate with the development
mailing list of any open source software project. I would suggest reading
https://www.python.org/psf/codeofconduct/ for some pointers on how people
typically behave around here in particular.

>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Barry Warsaw]

On May 2, 2018, at 09:42, Gregory Szorc  wrote:

> As for things Python could do to make things better, one idea is for "package 
> bundles." Instead of using .py, .pyc, .so, etc files as separate files on the 
> filesystem, allow Python packages to be distributed as standalone "archive" 
> files.

Of course, .so files have to be extracted to the file system, because we have 
to live with dlopen()’s API.  In our first release of shiv, we had a loader 
that did exactly that for just .so files.  We ended up just doing .pyz file 
unpacking unconditionally, ignoring zip-safe, mostly because too many packages 
still use __file__, which doesn’t work in a zipapp.

I’ll plug shiv and importlib.resources (and the standalone importlib_resources) 
again here. :)

> If you go this route, please don't require the use of zlib for file 
> compression, as zlib is painfully slow compared to alternatives like lz4 and 
> zstandard.

shiv works in a similar manner to pex, although it’s a completely new 
implementation that doesn’t suffer from huge sys.paths or the use of 
pkg_resources.  shiv + importlib.resources saves us 25-50% of warm cache 
startup time.  That makes things better but still not ideal.  Ultimately though 
that means we don’t suffer from the slowness of zlib since we don’t count cold 
cache times (i.e. before the initial pyz unpacking operation).

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575: Unifying function/method classes

[Robert Bradshaw]

This would be really useful for Cython, as well as a nice cleanup in
general (e.g. replacing 4 special cases with one check).

It seems the main concern is the user-visible change in types. If this is
determined to be too backwards incompatible (I would be surprised if many
projects are impacted, but also surprised if none are--more data is
warranted) I think the main points of this proposal could be addressed by
introducing the common superclass(es) while keeping the "leaf" types of
builtin_function_or_method, etc. exactly the same similar to the two-phase
proposal (though of course it'd be a nice to split this up, as well as
unify normal-method and c-defined-method if that's palatable).

- Robert




On Mon, Apr 30, 2018 at 8:55 AM, Jeroen Demeyer  wrote:

> On 2018-04-30 15:38, Mark Shannon wrote:
>
>> While a unified *interface* makes sense, a unified class hierarchy and
>> implementation, IMO, do not.
>>
>
> The main reason for the common base class is performance: in the bytecode
> interpreter, when we call an object, CPython currently has a special case
> for calling Python functions, a special case for calling methods, a special
> case for calling method descriptors, a special case for calling built-in
> functions.
>
> By introducing a common base class, we reduce the number of special cases.
> Second, we allow using this fast path for custom classes. With PEP 575, it
> is possible to create new classes with the same __call__ performance as the
> current built-in function class.
>
> Bound-methods may be callables, but they are not functions, they are a
>> pair of a function and a "self" object.
>>
>
> From the Python language point of view, that may be true but that's not
> how you want to implement methods. When I write a method in C, I want that
> it can be called either as unbound method or as bound method: the C code
> shouldn't see the difference between the calls X.foo(obj) or obj.foo(). And
> you want both calls to be equally fast, so you don't want that the bound
> method just wraps the unbound method. For this reason, it makes sense to
> unify functions and methods.
>
> IMO, there are so many versions of "function" and "bound-method", that a
>> unified class hierarchy and the resulting restriction to the
>> implementation will make implementing a unified interface harder, not
>> easier.
>>
>
> PEP 575 does not add any restrictions: I never claimed that all callables
> should inherit from base_function. Regardless, why would the common base
> class add restrictions? You can still add attributes and customize whatever
> you want in subclasses.
>
>
>
> Jeroen.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/robertwb%
> 40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Barry Warsaw]

Thanks for bringing this topic up again.  At $day_job, this is a highly visible 
and important topic, since the majority of our command line tools are written 
in Python (of varying versions from 2.7 to 3.6).  Some of those tools can take 
upwards of 5 seconds or more just to respond to —help, which causes lots of 
pain for developers, who complain (rightly so) up the management chain. ;)

We’ve done a fair bit of work to bring those numbers down without super radical 
workarounds.  Often there are problems not strictly related to the Python 
interpreter that contribute to this.  Python gets blamed, but it’s not always 
the interpreter’s fault.  Common issues include:

* Modules that have import-time side effects, such as network access or 
expensive creation of data structures.  Python 3.7’s `-X importtime` switch is 
a really wonderful way to identify the worst offenders.  Once 3.7 is released, 
I do plan to spend some time using this to collect data internally so we can 
attack our own libraries, and perhaps put automated performance testing into 
our build stack, to identify start up time regressions.

* pkg_resources.  When you have tons of entries on sys.path, pkg_resources does 
a lot of work at import time, and because of common patterns which tend to use 
pkg_resources namespace package support in __init__.py files, this just kills 
start up times.  Of course, pkg_resources has other uses too, so even in a 
purely Python 3 world (where your namespace packages can omit the __init__.py), 
you’ll often get clobbered as soon as you want to use the Basic Resource Access 
API.  This is also pretty common, and it’s the main reason why Brett and I 
created importlib.resources for 3.7 (with a standalone API-compatible library 
for older Pythons).  That’s one less reason to use pkg_resources, but it 
doesn’t address the __init__.py use.  Brett and I have been talking about 
addressing that for 3.8.

* pex - which we use as our single file zipapp tool.  Especially the 
interaction between pex and pkg_resources introduces pretty significant 
overhead.  My colleague Loren Carvalho created a tool called shiv which 
requires at least Python 3.6, avoids the use of pkg_resources, and implements 
other tricks to be much more performant than pex.   Shiv is now open source and 
you can find it on RTD and GitHub.

The switch to shiv and importlib.resources can shave 25-50% off of warm cache 
start up times for zipapp style executables.

Another thing we’ve done, although I’m much less sanguine about them as a 
general approach, is to move imports into functions, but we’re trying to only 
use that trick on the most critical cases.

Some import time effects can’t be changed.  Decorators come to mind, and click 
is a popular library for CLIs that provides some great features, but decorators 
do prevent a lazy loading approach.

> On May 1, 2018, at 20:26, Gregory Szorc  wrote:

>> You might think "what's a few milliseconds matter".  But if you run
>> hundreds of commands in a shell script it adds up.  git's speed is one
>> of the few bright spots in its UX, and hg's comparative slowness here is
>> a palpable disadvantage.

Oh, for command line tools, milliseconds absolutely matter.

> As a concrete example, I recently landed a Mercurial patch [2] that
> stubs out zope.interface to prevent the import of 9 modules on every
> `hg` invocation.

I have a similar dastardly plan to provide a pkg_resources stub :).

> Mercurial provides a `chg` program that essentially spins up a daemon
> `hg` process running a "command server" so the `chg` program [written in
> C - no startup overhead] can dispatch commands to an already-running
> Python/`hg` process and avoid paying the startup overhead cost. When you
> run Mercurial's test suite using `chg`, it completes *minutes* faster.
> `chg` exists mainly as a workaround for slow startup overhead.

A couple of our developers demoed a similar approach for one of our CLIs that 
almost everyone uses.  It’s a big application with lots of dependencies, so 
particularly vulnerable to pex and pkg_resources overhead.  While it was just a 
prototype, it was darn impressive to see subsequent invocations produce output 
almost immediately.  It’s unfortunate that we have to utilize all these tricks 
to get even moderately performant Python CLIs.

A few of us spent some time at last year’s core Python dev talking about other 
things we could do to improve Python’s start up time, not just with the 
interpreter itself, but within the larger context of the Python ecosystem.  
Many ideas seem promising until you dive into the details, so it’s definitely 
hard to imagine maintaining all of Python’s dynamic semantics and still making 
it an order of magnitude faster to start up.  But that’s not an excuse to give 
up, and I’m hoping we can continue to attack the problem, both in the micro and 
the macro, for 3.8 and beyond, because the alternative is that Python becomes 
less 

Re: [Python-Dev] Drop/deprecate Tkinter?

[Guido van Rossum]

Wow. I guess your code was broken and now you seem really upset. Go punch a
bag or something, and then propose something a little more constructive,
like adding a warning to the docs. I can assure you that there are many
people using apps written using Tkinter (e.g. IDLE) and there's a mailing
list as well (tkinter-disc...@python.org).

On Wed, May 2, 2018 at 1:51 PM, Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

> As https://bugs.python.org/issue33257 and https://bugs.python.org/issue3
> 3316 showed, Tkinter is broken, for both Py2 and Py3, with both threaded
> and non-threaded Tcl, since 2002 at least, and no-one gives a damn.
>
> This seems to be a testament that very few people are actually interested
> in or are using it.
>
> If that's so, there's no use keeping it in the standard library -- if
> anything, because there's not enough incentive and/or resources to support
> it. And to avoid screwing people (=me) up when they have the foolishness to
> think they can rely on it in their projects -- nowhere in the docs it is
> said that the module is only partly functional.
>
> --
>
> Regards,
> Ivan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Drop/deprecate Tkinter?

[Ivan Pozdeev via Python-Dev]

As https://bugs.python.org/issue33257 and 
https://bugs.python.org/issue33316 showed, Tkinter is broken, for both 
Py2 and Py3, with both threaded and non-threaded Tcl, since 2002 at 
least, and no-one gives a damn.


This seems to be a testament that very few people are actually 
interested in or are using it.


If that's so, there's no use keeping it in the standard library -- if 
anything, because there's not enough incentive and/or resources to 
support it. And to avoid screwing people (=me) up when they have the 
foolishness to think they can rely on it in their projects -- nowhere in 
the docs it is said that the module is only partly functional.


--

Regards,
Ivan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bpo-33257: seeking advice & approval on the course of action

[Ivan Pozdeev via Python-Dev]

The bottom line is: Tkinter is currently broken -- as in, it's not 
thread-safe (in both Py2 and Py3) despite being designed and advertizing 
itself as such.
All the fix options require some redesign of either `_tkinter', or some 
of the core as well.


So, I'd like to get some kind of core team's feedback and/or approval 
before pursuing any of them.


The options are outlined in https://bugs.python.org/issue33257#msg316087 .

If anyone of you is in Moscow, we can meet up and discuss this in a more 
time-efficient manner.


--
Regards,
Ivan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 2.7.15

[Larry Hastings]

On 05/02/2018 11:14 AM, Nathaniel Smith wrote:
I would guess that the folks who end up supporting python 2 past 2020 
(either as distributors or as library authors) will have an easier 
time of it if python 2's ssl module gets resynced with python 3 before 
the eol. But I suppose it's up to them to do the work...


You mean feature-wise, or do you mean "use more modern SSL libraries"?  
IIUC Windows doesn't use the system-provided SSL libraries, *and* 2.7 is 
built with a nasty old compiler.  It's entirely possible that on Windows 
2.7 *can't* use current SSL libraries because they won't build under 
said nasty old compiler.


(summoning Steve Dower etc here)


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Neil Schemenauer]

Antoine:
> The overhead of importing is not in trying too many names, but in
> loading the module and executing its bytecode.

That was my conclusion as well when I did some profiling last fall
at the Python core sprint.  My lazy execution experiments are an
attempt to solve this:

https://github.com/python/cpython/pull/6194

I expect that Mercurial is already doing a lot of tricks to make
execution more lazy.  They have a lazy module import hook but they
probably do other things to not execute more bytecode at startup
then is needed.  My lazy execution idea is that this could happen
more automatically.  I.e. don't pay for something you don't use.
Right now, with eager module imports, you usually pay a price for
every bit of bytecode that your program potentially uses.

Another idea, suggested to me by Carl Shapiro, is to store
unmarshalled Python data in the heap section of the executable (or
in DLLs).  Then, the OS page fault handling would take care of only
loading the data into RAM that is actually being used.  The linker
would take care of fixing up pointer references.  There are a lot of
details to work out with this idea but I have heard that Jeethu Rao
(Carl's colleague at Instagram) has a prototype implementation that
shows promise.

Regards,

  Neil
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 2.7.15

[Nathaniel Smith]

I would guess that the folks who end up supporting python 2 past 2020
(either as distributors or as library authors) will have an easier time of
it if python 2's ssl module gets resynced with python 3 before the eol. But
I suppose it's up to them to do the work... and probably other changes like
tls 1.3 support are more important than MemoryBIO? The only way I'd expect
the MemoryBIO backport to really matter is if there ends up being a period
when pip drops support for everything except the last python 2 micro
release, and wants to switch to PEP 543 mode everywhere. (Pip is special
because everyone else who needs fancy SSL features can 'pip install
pyopenssl', but that doesn't work for pip itself.)

But so far pip has never done anything like this, and I don't think keeping
support for pre-PEP 543 Pythons will be difficult either, so this doesn't
seem super compelling.

On Wed, May 2, 2018, 07:07 Benjamin Peterson  wrote:

> The lack of movement for a year makes me wonder if PEP 546 should be moved
> to Withdrawn status.
>
> On Wed, May 2, 2018, at 02:35, Victor Stinner wrote:
> > Sadly, Python 2.7.15 still miss the implementation of the "PEP 546 --
> > Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7":
> > https://www.python.org/dev/peps/pep-0546/
> >
> > Last time I checked, the tests failed on Travis CI and I failed to
> > reproduce the issue:
> > https://bugs.python.org/issue22559
> >
> > I expected Cory Benfield to jump into this issue since his "PEP 543 --
> > A Unified TLS API for Python" was my motivation for the PEP 546, but
> > it seems like he is busy and the TLS PEP doesn't move anymore :-(
> > https://www.python.org/dev/peps/pep-0543/
> >
> > Victor
> >
> > 2018-05-01 6:09 GMT+02:00 Benjamin Peterson :
> > > Greetings,
> > > I'm pleased to announce the immediate availability of Python 2.7.15,
> the latest bug fix release in the senescent Python 2.7 series.
> > >
> > > Source and binary downloads may be found on python.org:
> > >
> > >  https://www.python.org/downloads/release/python-2715/
> > >
> > > Bugs should be reported to https://bugs.python.org/
> > >
> > > The source tarball contains a complete changelog in the Misc/NEWS
> file. The only change since the release candidate is a fix for undefined C
> behavior that newer compilers (including GCC 8) have started to exploit.
> > >
> > > Users of the macOS binaries should note that all python.org macOS
> installers now ship with a builtin copy of OpenSSL. Additionally, there is
> a new additional installer variant for macOS 10.9+ that includes a built-in
> version of Tcl/Tk 8.6. See the installer README for more information.
> > >
> > > Happy May,
> > > Benjamin
> > > 2.7 release manager
> > > ___
> > > Python-Dev mailing list
> > > Python-Dev@python.org
> > > https://mail.python.org/mailman/listinfo/python-dev
> > > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Nathaniel Smith]

On Wed, May 2, 2018, 09:51 Gregory Szorc  wrote:

> Correct me if I'm wrong, but aren't there downsides with regards to C
> extension compatibility to not having a shared libpython? Or does all the
> packaging tooling "just work" without a libpython? (It's possible I have my
> wires crossed up with something else regarding a statically linked Python.)
>

IIRC, the rule on Linux is that if you build an extension on a statically
built python, then it can be imported on a shared python, but not
vice-versa. Manylinux wheels are therefore always built on a static python
so that they'll work everywhere. (We should probably clean this up upstream
at some point, but there's not a lot of appetite for touching this stuff –
very obscure, very easy to break things without realizing it, not much
upside.)

On Windows I don't think there is such a thing as a static build, because
extensions have to link to the python dll to work at all. And on MacOS I'm
not sure, though from knowing how their linker works my guess is that all
extensions act like static extensions do on Linux.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Process to remove a Python feature

[Brett Cannon]

On Wed, 2 May 2018 at 02:12 Victor Stinner  wrote:

> Hi,
>
> As a follow-up to the "[Python-Dev] (Looking for) A Retrospective on
> the Move to Python 3" thread, I will like to clarify how a feature
> should be removed from Python.
>
> We have multiple tools:
>
> * Emit a PendingDeprecationWarning warning at runtime for *slow*
> deprecation (remove a feature in at least 3 cycles)
> * Emit a DeprecationWarning warning at runtime for fast deprecation
> (remove a feature in 2 cycles)
> * Document that a feature is deprecated in the module documentation
> * "What's New in Python X.Y?" documents: especially Deprecated,
> Removed and Porting to Python X.Y sections.
> * Communicate on python-dev, Twitter, Python Insider blog, etc.
> * Collaborate with major Python projects to help them to migrate the
> alternative
>
> IMHO a feature should not be removed if there is no alternative, or if
> the alternative is too new.
>
> The usual process is:
>
> * Document the deprecation and emit DeprecationWarning in version N
> * Document the removal and remove the feature in version N+1
>
> Slow deprecation:
>
> * Document the deprecation and emit PendingDeprecationWarning in version N
> * Emit a DeprecationWarning in version N+1
> * Document the removal and remove the feature in version N+2
>
> The hidden ghost is the old rule:
>
>"keep old APIs around to make porting from Python 2 easier"
>
> Is this rule still applicable?
>

Python 2 is still supported, so I assume so.


>
> Does it mean that the Python 3 release following Python 2 end-of-life
> (2020) will be our next feared "Python 4"? Are we going to remove all
> deprecated features at once, to maximize incompatibilities and make
> users unhappy?
>

I don't see why removing features that already raise a DeprecationWarning
would require bumping the major version number. Personally, I assumed
either Python 3.9 or 3.10 would be the version where we were okay clearing
out the stuff that had been raising DeprecationWarning for years.


>
> Should we always document in which version a feature will be removed?
>

We should at least open an issue to track when the removal is scheduled.
But if we know ahead of time then I so no reason not to document it.


> Some features are deprecated since many versions, and the deprecated
> features are still there. In most cases, it's because of the Python 2
> rule.
>
> Sometimes, we forget to remove features which has been scheduled for
> removal in a specific version.
>

Right, which is why we should open an issue immediately. I had a "remove
pyvenv" issue open and assigned to myself for years while I waited for
Python 3.8 development to open. Staring at that issue for so long made sure
I didn't forget. ;)


>
> Maybe we should create a tool to list features scheduled for removal,
> and open a discussion to check each removal?
>

I don't know if a tool is necessary. We could have a meta issue or text
file somewhere to track what's to be removed in a certain version.


>
> Ten years ago, I wanted to remove most modules and many functions from
> the standard library. Now my experience showed me that *each* removal
> is very painful, hurt more projects than expected, and takes longer
> than 3 years to be fully effective (no longer used in most common 3rd
> party modules).
>
> The usual issue is to write a single code base working on all Python
> versions. For example, the new alternative is only available on recent
> Python versions, which requires to have at least two code paths to
> support all versions. Sometimes, there are 3 code paths...
>
> For a recent example, see "remove platform.linux_distribution()":
> https://bugs.python.org/issue28167
>
> Removing a feature from the C API is more complex, since there is no
> portable way to emit a deprecation warning at compilation. There is
> Py_DEPRECATED() which seems to only be available on GCC (3.1 and
> newer).
>
> Maybe we should stop to remove features, except if there is really a
> good reason to do that?
>

I thought that already was the policy. ;) I think the real question is what
is people's definition of a "good reason". For instance, I'm all for
removing unsupported code so we don't have to maintain it, even if it's
just from code modernization and such (i.e. pruning down the stdlib). But I
also know others disagree with me and are fine just having modules sit
there as long as the issue rate is low enough to not notice. It's just one
of those things we don't have an official policy on (yet?).

-Brett


>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 

Re: [Python-Dev] Python startup time

[Gregory Szorc]

 On Tue, May 1, 2018 at 11:55 PM, Ray Donnelly 
wrote:

> Is your Python interpreter statically linked? The Python 3 ones from the
anaconda distribution (use Miniconda!) are for Linux and macOS and that
roughly halved our startup times.

My Python interpreters use a shared library. I'll definitely investigate
the performance of a statically-linked interpreter.

Correct me if I'm wrong, but aren't there downsides with regards to C
extension compatibility to not having a shared libpython? Or does all the
packaging tooling "just work" without a libpython? (It's possible I have my
wires crossed up with something else regarding a statically linked Python.)

On Wed, May 2, 2018 at 2:26 AM, Victor Stinner  wrote:

> What do you propose to make Python startup faster?
>

That's a very good question. I'm not sure I'm able to answer it because I
haven't dug too much into CPython's internals much farther than what is
required to implement C extensions. But I can share insight from what the
Mercurial project has collectively learned.


>
> As I wrote in my previous emails, many Python core developers care of
> the startup time and we are working on making it faster.
>
> INADA Naoki added -X importtime to identify slow imports and
> understand where Python spent its startup time.
>

-X importtime is a great start! For a follow-up enhancement, it would be
useful to see what aspects of import are slow. Is it finding modules
(involves filesystem I/O)? Is it unmarshaling pyc files? Is it executing
the module code? If executing code, what part is slow? Inline
statements/expressions? Compiling types? Printing the microseconds it takes
to import a module is useful. But it only gives me a general direction: I
want to know what parts of the import made it slow so I know if I should be
focusing on code running during module import, slimming down the size of a
module, eliminating the module import from fast paths, pursuing alternative
module importers, etc.


>
> Recent example: Barry Warsaw identified that pkg_resources is slow and
> added importlib.resources to Python 3.7:
> https://docs.python.org/dev/library/importlib.html#module-
> importlib.resources
>
> Brett Cannon is also working on a standard solution for lazy imports
> since many years:
> https://pypi.org/project/modutil/
> https://snarky.ca/lazy-importing-in-python-3-7/
>

Mercurial has used lazy module imports for years. On 2.7.14, it reduces `hg
version` from ~160ms to ~55ms (~34% of original). On Python 3, we're using
`importlib.util.LazyLoader` and it reduces `hg version` on 3.7 from ~245ms
to ~120ms (~49% of original). I'm not sure why Python 3's built-in module
importer doesn't yield the speedup that our custom Python 2 importer does.
One explanation is our custom importer is more advanced than importlib.
Another is that Python 3's import mechanism is slower (possibly due to
being written in Python instead of C). We haven't yet spent much time
optimizing Mercurial for Python 3: our immediate goal is to get it working
first. Given the startup performance problem on Python 3, it is only a
matter of time before we dig into this further.

It's worth noting that lazy module importing can be undone via common
patterns. Most commonly, `from foo import X`. It's *really* difficult to
implement a proper object proxy. Mercurial's lazy importer gives up in this
case and imports the module and exports the symbol. (But if the imported
module is a package, we detect that and make the module exports proxies to
a lazy module.)

Another common undermining of the lazy importer is code that runs during
import time module exec that accesses an attribute. e.g.

```
import foo

class myobject(foo.Foo):
pass
```

Mercurial goes out of its way to avoid these patterns so modules can be
delay imported as much as possible. As long as import times are
problematic, it would be helpful if the standard library adopted similar
patterns. Although I recognize there are backwards compatibility concerns
that tie your hands a bit.


> Nick Coghlan is working on the C API to configure Python startup: PEP
> 432. When it will be ready, maybe Mercurial could use a custom Python
> optimized for its use case.
>

That looks great!

The direction Mercurial is going in is that `hg` will likely become a Rust
binary (instead of a #!python script) that will use an embedded Python
interpreter. So we will have low-level control over the interpreter via the
C API. I'd also like to see us distribute a copy of Python in our official
builds. This will allow us to take various shortcuts, such as not having to
probe various sys.path entries since certain packages can only exist in one
place. I'd love to get to the state Google is at where they have
self-contained binaries with ELF sections containing Python modules. But
that requires a bit of very low-level hacking. We'll likely have a Rust
binary (that possibly static links libpython) and a separate JAR/zip-like
file 

Re: [Python-Dev] [RELEASE] Python 2.7.15

[Benjamin Peterson]

The lack of movement for a year makes me wonder if PEP 546 should be moved to 
Withdrawn status.

On Wed, May 2, 2018, at 02:35, Victor Stinner wrote:
> Sadly, Python 2.7.15 still miss the implementation of the "PEP 546 --
> Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7":
> https://www.python.org/dev/peps/pep-0546/
> 
> Last time I checked, the tests failed on Travis CI and I failed to
> reproduce the issue:
> https://bugs.python.org/issue22559
> 
> I expected Cory Benfield to jump into this issue since his "PEP 543 --
> A Unified TLS API for Python" was my motivation for the PEP 546, but
> it seems like he is busy and the TLS PEP doesn't move anymore :-(
> https://www.python.org/dev/peps/pep-0543/
> 
> Victor
> 
> 2018-05-01 6:09 GMT+02:00 Benjamin Peterson :
> > Greetings,
> > I'm pleased to announce the immediate availability of Python 2.7.15, the 
> > latest bug fix release in the senescent Python 2.7 series.
> >
> > Source and binary downloads may be found on python.org:
> >
> >  https://www.python.org/downloads/release/python-2715/
> >
> > Bugs should be reported to https://bugs.python.org/
> >
> > The source tarball contains a complete changelog in the Misc/NEWS file. The 
> > only change since the release candidate is a fix for undefined C behavior 
> > that newer compilers (including GCC 8) have started to exploit.
> >
> > Users of the macOS binaries should note that all python.org macOS 
> > installers now ship with a builtin copy of OpenSSL. Additionally, there is 
> > a new additional installer variant for macOS 10.9+ that includes a built-in 
> > version of Tcl/Tk 8.6. See the installer README for more information.
> >
> > Happy May,
> > Benjamin
> > 2.7 release manager
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: 
> > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Antoine Pitrou]

On Wed, 2 May 2018 11:26:35 +0200
Victor Stinner  wrote:
> 
> Brett Cannon is also working on a standard solution for lazy imports
> since many years:
> https://pypi.org/project/modutil/
> https://snarky.ca/lazy-importing-in-python-3-7/

AFAIK, Mercurial already has its own lazy importer.

> Nick Coghlan is working on the C API to configure Python startup: PEP
> 432. When it will be ready, maybe Mercurial could use a custom Python
> optimized for its use case.
> 
> IMHO Python import system is inefficient. We try too many alternative names.

The overhead of importing is not in trying too many names, but in
loading the module and executing its bytecode.

> Why do we still check for the .pyc file outside __pycache__ directories?

Because we support sourceless distributions.

> Why do we have to check for 3 different names for .so files?

See https://bugs.python.org/issue32387

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Process to remove a Python feature

[Victor Stinner]

2018-05-02 11:17 GMT+02:00 Antoine Pitrou :
> It's at least possible with gcc, clang and MSVC:
> https://stackoverflow.com/questions/295120/c-mark-as-deprecated/295229#295229
>
> You can even add a deprecation message.

Aha, I opened an issue: https://bugs.python.org/issue33407 "Implement
Py_DEPRECATED() macro for Visual Studio"

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 2.7.15

[Victor Stinner]

Sadly, Python 2.7.15 still miss the implementation of the "PEP 546 --
Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7":
https://www.python.org/dev/peps/pep-0546/

Last time I checked, the tests failed on Travis CI and I failed to
reproduce the issue:
https://bugs.python.org/issue22559

I expected Cory Benfield to jump into this issue since his "PEP 543 --
A Unified TLS API for Python" was my motivation for the PEP 546, but
it seems like he is busy and the TLS PEP doesn't move anymore :-(
https://www.python.org/dev/peps/pep-0543/

Victor

2018-05-01 6:09 GMT+02:00 Benjamin Peterson :
> Greetings,
> I'm pleased to announce the immediate availability of Python 2.7.15, the 
> latest bug fix release in the senescent Python 2.7 series.
>
> Source and binary downloads may be found on python.org:
>
>  https://www.python.org/downloads/release/python-2715/
>
> Bugs should be reported to https://bugs.python.org/
>
> The source tarball contains a complete changelog in the Misc/NEWS file. The 
> only change since the release candidate is a fix for undefined C behavior 
> that newer compilers (including GCC 8) have started to exploit.
>
> Users of the macOS binaries should note that all python.org macOS installers 
> now ship with a builtin copy of OpenSSL. Additionally, there is a new 
> additional installer variant for macOS 10.9+ that includes a built-in version 
> of Tcl/Tk 8.6. See the installer README for more information.
>
> Happy May,
> Benjamin
> 2.7 release manager
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Victor Stinner]

What do you propose to make Python startup faster?

As I wrote in my previous emails, many Python core developers care of
the startup time and we are working on making it faster.

INADA Naoki added -X importtime to identify slow imports and
understand where Python spent its startup time.

Recent example: Barry Warsaw identified that pkg_resources is slow and
added importlib.resources to Python 3.7:
https://docs.python.org/dev/library/importlib.html#module-importlib.resources

Brett Cannon is also working on a standard solution for lazy imports
since many years:
https://pypi.org/project/modutil/
https://snarky.ca/lazy-importing-in-python-3-7/

Nick Coghlan is working on the C API to configure Python startup: PEP
432. When it will be ready, maybe Mercurial could use a custom Python
optimized for its use case.

IMHO Python import system is inefficient. We try too many alternative names.

Example with Python 3.8

$ ./python -vv:
>>> import dontexist
# trying 
/home/vstinner/prog/python/master/dontexist.cpython-38dm-x86_64-linux-gnu.so
# trying /home/vstinner/prog/python/master/dontexist.abi3.so
# trying /home/vstinner/prog/python/master/dontexist.so
# trying /home/vstinner/prog/python/master/dontexist.py
# trying /home/vstinner/prog/python/master/dontexist.pyc
# trying 
/home/vstinner/prog/python/master/Lib/dontexist.cpython-38dm-x86_64-linux-gnu.so
# trying /home/vstinner/prog/python/master/Lib/dontexist.abi3.so
# trying /home/vstinner/prog/python/master/Lib/dontexist.so
# trying /home/vstinner/prog/python/master/Lib/dontexist.py
# trying /home/vstinner/prog/python/master/Lib/dontexist.pyc
# trying 
/home/vstinner/prog/python/master/build/lib.linux-x86_64-3.8-pydebug/dontexist.cpython-38dm-x86_64-linux-gnu.so
# trying 
/home/vstinner/prog/python/master/build/lib.linux-x86_64-3.8-pydebug/dontexist.abi3.so
# trying 
/home/vstinner/prog/python/master/build/lib.linux-x86_64-3.8-pydebug/dontexist.so
# trying 
/home/vstinner/prog/python/master/build/lib.linux-x86_64-3.8-pydebug/dontexist.py
# trying 
/home/vstinner/prog/python/master/build/lib.linux-x86_64-3.8-pydebug/dontexist.pyc
# trying 
/home/vstinner/.local/lib/python3.8/site-packages/dontexist.cpython-38dm-x86_64-linux-gnu.so
# trying /home/vstinner/.local/lib/python3.8/site-packages/dontexist.abi3.so
# trying /home/vstinner/.local/lib/python3.8/site-packages/dontexist.so
# trying /home/vstinner/.local/lib/python3.8/site-packages/dontexist.py
# trying /home/vstinner/.local/lib/python3.8/site-packages/dontexist.pyc
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 983, in _find_and_load
  File "", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dontexist'

Why do we still check for the .pyc file outside __pycache__ directories?

Why do we have to check for 3 different names for .so files?

Does Mercurial need all directories of sys.path?

What's the status of the "system python" project? :-)

I also would prefer Python without the site module. Can we rewrite
this module in C maybe? Until recently, the site module was needed on
Python to create the "mbcs" encoding alias. Hopefully, the feature has
been removed into Lib/encodings/__init__.py (new private _alias_mbcs()
function).

Python 3.7b3+:

$ python3.7 -X importtime -c pass
import time: self [us] | cumulative | imported package
import time:95 | 95 | zipimport
import time:   589 |589 | _frozen_importlib_external
import time:67 | 67 | _codecs
import time:   498 |565 |   codecs
import time:   425 |425 |   encodings.aliases
import time:   641 |   1629 | encodings
import time:   228 |228 | encodings.utf_8
import time:   143 |143 | _signal
import time:   335 |335 | encodings.latin_1
import time:58 | 58 | _abc
import time:   265 |322 |   abc
import time:   298 |619 | io
import time:69 | 69 |   _stat
import time:   196 |265 | stat
import time:   169 |169 |   genericpath
import time:   336 |505 | posixpath
import time:  1190 |   1190 | _collections_abc
import time:   600 |   2557 |   os
import time:   223 |223 |   _sitebuiltins
import time:   214 |214 |   sitecustomize
import time:74 | 74 |   usercustomize
import time:   477 |   3544 | site

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Process to remove a Python feature

[Antoine Pitrou]

On Wed, 2 May 2018 11:11:16 +0200
Victor Stinner  wrote:
> 
> Removing a feature from the C API is more complex, since there is no
> portable way to emit a deprecation warning at compilation. There is
> Py_DEPRECATED() which seems to only be available on GCC (3.1 and
> newer).

It's at least possible with gcc, clang and MSVC:
https://stackoverflow.com/questions/295120/c-mark-as-deprecated/295229#295229

You can even add a deprecation message.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Process to remove a Python feature

[Victor Stinner]

Hi,

As a follow-up to the "[Python-Dev] (Looking for) A Retrospective on
the Move to Python 3" thread, I will like to clarify how a feature
should be removed from Python.

We have multiple tools:

* Emit a PendingDeprecationWarning warning at runtime for *slow*
deprecation (remove a feature in at least 3 cycles)
* Emit a DeprecationWarning warning at runtime for fast deprecation
(remove a feature in 2 cycles)
* Document that a feature is deprecated in the module documentation
* "What's New in Python X.Y?" documents: especially Deprecated,
Removed and Porting to Python X.Y sections.
* Communicate on python-dev, Twitter, Python Insider blog, etc.
* Collaborate with major Python projects to help them to migrate the alternative

IMHO a feature should not be removed if there is no alternative, or if
the alternative is too new.

The usual process is:

* Document the deprecation and emit DeprecationWarning in version N
* Document the removal and remove the feature in version N+1

Slow deprecation:

* Document the deprecation and emit PendingDeprecationWarning in version N
* Emit a DeprecationWarning in version N+1
* Document the removal and remove the feature in version N+2

The hidden ghost is the old rule:

   "keep old APIs around to make porting from Python 2 easier"

Is this rule still applicable?

Does it mean that the Python 3 release following Python 2 end-of-life
(2020) will be our next feared "Python 4"? Are we going to remove all
deprecated features at once, to maximize incompatibilities and make
users unhappy?

Should we always document in which version a feature will be removed?
Some features are deprecated since many versions, and the deprecated
features are still there. In most cases, it's because of the Python 2
rule.

Sometimes, we forget to remove features which has been scheduled for
removal in a specific version.

Maybe we should create a tool to list features scheduled for removal,
and open a discussion to check each removal?

Ten years ago, I wanted to remove most modules and many functions from
the standard library. Now my experience showed me that *each* removal
is very painful, hurt more projects than expected, and takes longer
than 3 years to be fully effective (no longer used in most common 3rd
party modules).

The usual issue is to write a single code base working on all Python
versions. For example, the new alternative is only available on recent
Python versions, which requires to have at least two code paths to
support all versions. Sometimes, there are 3 code paths...

For a recent example, see "remove platform.linux_distribution()":
https://bugs.python.org/issue28167

Removing a feature from the C API is more complex, since there is no
portable way to emit a deprecation warning at compilation. There is
Py_DEPRECATED() which seems to only be available on GCC (3.1 and
newer).

Maybe we should stop to remove features, except if there is really a
good reason to do that?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Ray Donnelly]

On Wed, May 2, 2018, 4:53 AM Gregory Szorc  wrote:

> On 7/19/2017 12:15 PM, Larry Hastings wrote:
> >
> >
> > On 07/19/2017 05:59 AM, Victor Stinner wrote:
> >> Mercurial startup time is already 45.8x slower than Git whereas tested
> >> Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial
> >> developers, with a startup time 2x - 3x slower...
> >
> > When Matt Mackall spoke at the Python Language Summit some years back, I
> > recall that he specifically complained about Python startup time.  He
> > said Python 3 "didn't solve any problems for [them]"--they'd already
> > solved their Unicode hygiene problems--and that Python's slow startup
> > time was already a big problem for them.  Python 3 being /even slower/
> > to start was absolutely one of the reasons why they didn't want to
> upgrade.
> >
> > You might think "what's a few milliseconds matter".  But if you run
> > hundreds of commands in a shell script it adds up.  git's speed is one
> > of the few bright spots in its UX, and hg's comparative slowness here is
> > a palpable disadvantage.
> >
> >
> >> So please continue efforts for make Python startup even faster to beat
> >> all other programming languages, and finally convince Mercurial to
> >> upgrade ;-)
> >
> > I believe Mercurial is, finally, slowly porting to Python 3.
> >
> > https://www.mercurial-scm.org/wiki/Python3
> >
> > Nevertheless, I can't really be annoyed or upset at them moving slowly
> > to adopt Python 3, as Matt's objections were entirely legitimate.
>
> I just now found found this thread when searching the archive for
> threads about startup time. And I was searching for threads about
> startup time because Mercurial's startup time has been getting slower
> over the past few months and this is causing substantial pain.
>
> As I posted back in 2014 [1], CPython's startup overhead was >10% of the
> total CPU time in Mercurial's test suite. And when you factor in the
> time to import modules that get Mercurial to a point where it can run
> commands, it was more like 30%!
>
> Mercurial's full test suite currently runs `hg` ~25,000 times. Using
> Victor's startup time numbers of 6.4ms for 2.7 and 14.5ms for
> 3.7/master, Python startup overhead contributes ~160s on 2.7 and ~360s
> on 3.7/master. Even if you divide this by the number of available CPU
> cores, we're talking dozens of seconds of wall time just waiting for
> CPython to get to a place where Mercurial's first bytecode can execute.
>
> And the problem is worse when you factor in the time it takes to import
> Mercurial's own modules.
>
> As a concrete example, I recently landed a Mercurial patch [2] that
> stubs out zope.interface to prevent the import of 9 modules on every
> `hg` invocation. This "only" saved ~6.94ms for a typical `hg`
> invocation. But this decreased the CPU time required to run the test
> suite on my i7-6700K from ~4450s to ~3980s (~89.5% of original) - a
> reduction of almost 8 minutes of CPU time (and over 1 minute of wall time)!
>
> By the time CPython gets Mercurial to a point where we can run useful
> code, we've already blown most of or past the time budget where humans
> perceive an action/command as instantaneous. If you ignore startup
> overhead, Mercurial's performance compares quite well to Git's for many
> operations. But the reality is that CPython startup overhead makes it
> look like Mercurial is non-instantaneous before Mercurial even has the
> opportunity to execute meaningful code!
>
> Mercurial provides a `chg` program that essentially spins up a daemon
> `hg` process running a "command server" so the `chg` program [written in
> C - no startup overhead] can dispatch commands to an already-running
> Python/`hg` process and avoid paying the startup overhead cost. When you
> run Mercurial's test suite using `chg`, it completes *minutes* faster.
> `chg` exists mainly as a workaround for slow startup overhead.
>
> Changing gears, my day job is maintaining Firefox's build system. We use
> Python heavily in the build system. And again, Python startup overhead
> is problematic. I don't have numbers offhand, but we invoke likely a few
> hundred Python processes as part of building Firefox. It should be
> several thousand. But, we've had to "hack" parts of the build system to
> "batch" certain build actions in single process invocations in order to
> avoid Python startup overhead. This undermines the ability of some build
> tools to formulate a reasonable understanding of the DAG and it causes a
> bit of pain for build system developers and makes it difficult to
> achieve "no-op" and fast incremental builds because we're always
> invoking certain Python processes because we've had to move DAG
> awareness out of the build backend and into Python. At some point, we'll
> likely replace Python code with Rust so the build system is more "pure"
> and easier to maintain and reason about.
>
> I've seen posts in this thread and elsewhere in the CPython