[Python-Dev] Re: PEP 640: Unused variable syntax.
Hello, On Tue, 20 Oct 2020 00:00:49 +0200 Thomas Wouters wrote: > One of the problems I have with the Pattern Matching proposal (PEP 622 > originally, now PEPs 634, 635, 636) is the special-casing of '_' to > not actually assign to the name, which is a subtle but meaningful > divergence from the rest of Python. In discussions with the authors I > proposed using '?' instead *and* extending that to existing unpacking One problem with this PEP, which I didn't see mentioned in the other replies, is that it tries to grab "?" character, which is already sought-for by another pending PEP: "PEP 505 -- None-aware operators", https://www.python.org/dev/peps/pep-0505/ . Use of "?" in PEP640 can be disambiguated enough from PEP505's "??", "?.", "?[" from a compiler token perspective perspective, but what about confusion/clarity to humans: ?, ?, c = a d = b?.c ?, c = b ?? (None, 2) e = b?[i] (And PEP 505 would need to be addressed sooner or later, now that it's part of other mainstream languages. In Python's conceptual debt tower, non-aware operators definitely precede pattern matching). [] -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RWG25CCT22OYXVNGJ4EKXQE74GBLZUWY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On 10/20/2020 2:49 PM, Dan Stromberg wrote: I suspect what it needs most is HPY work, which could benefit a lot of Python language implementations in the long term: https://github.com/hpyproject/hpy $2e6 spent on HPY could be pretty amazing. I don't think the two projects are mutually exclusive. I would like to see the PSF raise and spend a few million a year on improvements. -- Terry Jan Reedy ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QXYIROIWMSZ4ZHF23C5TCHO2DFLQVZAZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Tue, Oct 20, 2020 at 5:59 AM Mark Shannon wrote: > Hi everyone, > > CPython is slow. We all know that, yet little is done to fix it. > > I'd like to change that. > I have a plan to speed up CPython by a factor of five over the next few > years. But it needs funding. > > I am aware that there have been several promised speed ups in the past > that have failed. You might wonder why this is different. > > Here are three reasons: > 1. I already have working code for the first stage. > 2. I'm not promising a silver bullet. I recognize that this is a > substantial amount of work and needs funding. > 3. I have extensive experience in VM implementation, not to mention a > PhD in the subject. > > My ideas for possible funding, as well as the actual plan of > development, can be found here: > > https://github.com/markshannon/faster-cpython > > I'd love to hear your thoughts on this. > > Cheers, > Mark. > +1 Overall I think you are making quite a reasonable proposal. It sounds like effectively bringing your hotpy2 concepts into the CPython interpreter with an intent to help maintain them over the long term. People worried that you are doing this out of self interest may not know who you are. Sure, you want to be paid to do work that you appear to love and have been mulling over for a decade+. There is nothing wrong with that. Payment is proposed as on delivery per phase. I like the sound of that, nice! Challenges I expect we'll face, that seem tractable to me, are mostly around what potential roadblocks we, us existing python-committers and our ultimate decider steering council might introduce intentionally or not that prevents landing such work. Payment on delivery helps that a lot, if we opt out of some work, it is both our losses. One potential outcome is that you'd burn out and go away if we didn't accept something meaning payment wasn't going to happen. That already happens amongst all core devs today so I don't have a problem with this even though it isn't what we'd rightfully want to happen. Middle grounds are quite reasonable renegotiations. The deciders on this would be the PSF (because money) and the PSF would presumably involve the Steering Council in decisions of terms and judgements. Some background materials for those who don't already know Mark's past work along these lines that is where this proposal comes from: https://sites.google.com/site/makingcpythonfast/ (hotpy) and the associated presentation in 2012: https://www.youtube.com/watch?v=c6PYnZUMF7o The amount of money seems entirely reasonable to me. Were it to be taken on, part of the PSF's job is to drum up the money. This would be a contract with outcomes that could effectively be sold to funders in order to do so. There are many companies who use CPython a lot that we could solicit funding from, many of whom have employees among our core devs already. Will they bite? It doesn't even have to be from a single source or be all proposed phases up front, that is what the PSF exists to decide and coordinate on. We've been discussing on and off in the past many years how to pay people for focused work on CPython and the ecosystem and balance that with being an open source community project. We've got some people employed along these lines already, this would become more of that and in many ways just makes sense to me. Summarizing some responses to points others have brought up: Performance estimates: * Don't fret about claimed speedups of each phase. We're all going to doubt different things or expect others to be better. The proof is ultimately in the future pudding. JIT topics: * JITs rarely stand alone. The phase 1+2 improved interpreter will always exist. It is normal to start with an interpreter for fast startup and initial tracing before performing JIT compilations, and as a fallback mechanism when the JIT isn't appropriate or available. (my background: Transmeta. We had an Interpreter and at least two levels of Translators behind our x86 compatible CPUs, all were necessary) * Sometimes you simply want to turn tracing and jit stuff off to save memory. That knob always winds up existing. If nothing else it is normal to run our test suite with such a knob in multiple positions for proper coverage. * It is safe to assume any later phase actual JIT would target at least one important platform (ex: amd64 or aarch64) and if successful should easily elicit contributions supporting others either as work or as funding to create it. "*Why this, why not fund XyZ?*" whataboutism: * This conversation is separate from other projects. The way attracting funding for a project works can involve spelling out what it is for. It isn't my decision, but I'd be amazed if anything beyond maybe phase 1 came solely out of a PSF general no obligation fund. CPython is the most used Python VM in the world. A small amount of funding is not going to get maintainers and users to switch to PyPy. There is unlikely to be a major this or
[Python-Dev] Re: Speeding up CPython
On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote: > What I don't see is where the money's coming from. It's fine to ask, > but will anyone come up with that sort of funding? I don't think Mark is asking for you or I to fund the exercise. He's asking for the PSF to fund it. I think that Mark's asking price is perhaps a bit optimistic. With the Covid-19 pandemic, lock downs and shutting down of international travel, I expect that PSF conference income will be way down. And income in the previous financial year was not exactly at Facebook levels: Revenue: $3.1 million Expences: $2.6 million Nett income: $52 https://www.python.org/psf/annual-report/2019/ Mark is asking for half of that, justified that this is the going rate for a full-time developer at one of the top tier tech firms. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IZGFSDJMVO7UWWPD2WEHUPIVN6SATEJB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
Since HPy was mentioned, hello from the HPy team! If anyone is thinking about Python performance or new Python VMs, we'd love them to take a look at HPy and come and talk to us. HPy is meant to provide a new C API layer that any Python VM could implement in order to efficiently support Python extensions written on top of HPy. Amazingly, such extensions would also be backwards compatible (i.e. they would also run on CPython as it exists today). If you'd like to talk to us (and we would like to talk to you) you can find us at: * Mailing list: hpy-...@python.org * IRC: hpy-...@python.org If you would like to contribute as a developer (or any other way), we're here to help (and at the moment I am prioritising landing other people's contributions pretty much full-time). If someone has money to donate, they can donate to the project at https://opencollective.com/hpy/contribute. We have quite a few contributors who could contribute more time than they do in exchange for money. HPy is currently in quite an early stage of development (we can already run some simple Python C extensions without any performance penalties on CPython). The upside is that right now we can consider almost any suggestion for improving HPy because none of the design is set in stone. In a few months it might be helpful to have people try porting some (many) C extensions to HPy to see how that goes. Yours happily, Simon Cross (random person working on HPy) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ARSII7KMZZ5W6UMKR7LIQDIWFSWWYQLZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
I'd love to hear more about what workloads you're targeting and how you came up with the anticipated numbers for the improvements. For comparison, our new jit provides a single-digit-percentage speedup on our django and flask benchmarks. On Tue, Oct 20, 2020 at 9:03 AM Mark Shannon wrote: > Hi everyone, > > CPython is slow. We all know that, yet little is done to fix it. > > I'd like to change that. > I have a plan to speed up CPython by a factor of five over the next few > years. But it needs funding. > > I am aware that there have been several promised speed ups in the past > that have failed. You might wonder why this is different. > > Here are three reasons: > 1. I already have working code for the first stage. > 2. I'm not promising a silver bullet. I recognize that this is a > substantial amount of work and needs funding. > 3. I have extensive experience in VM implementation, not to mention a > PhD in the subject. > > My ideas for possible funding, as well as the actual plan of > development, can be found here: > > https://github.com/markshannon/faster-cpython > > I'd love to hear your thoughts on this. > > Cheers, > Mark. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/RDXLCH22T2EZDRCBM6ZYYIUTBWQVVVWH/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QPQSR5UNTGLEL3GFQXQW4LA2OQX76YOG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Tue, Oct 20, 2020 at 9:33 AM Steven D'Aprano wrote: > > What would happen if $2M were spent on improving PyPy3 instead? > > Then both of the PyPy3 users will be very happy *wink* > Wow, I didn't know I was 50% of Pypy3 users :) Anyway, Pypy3 is already pretty great. I'm sure it can be improved, but I suspect what it needs most is HPY work, which could benefit a lot of Python language implementations in the long term: https://github.com/hpyproject/hpy $2e6 spent on HPY could be pretty amazing. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OJCDOMSSK2DICZARJZMZ2SOQAKIDPZ4Q/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 11: Drop support for AIX releases without dlopen
Brett Cannon wrote: > On Mon, Oct 19, 2020 at 4:43 PM Kevin Adler kad...@linux.vnet.ibm.com > wrote: > > Should this list be updated to mention that AIX 5.3 and below are no > > longer supported? > > Only if we are going to rip out all AIX-related code. Ok. Looking through the history, I think I found an equivalent change to model after: when support for FreeBSD 9 and older was removed. https://github.com/python/cpython/commit/13ff24582c99dfb439b1af7295b401415e7eb05b ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LQLPA3MCD4PKZZ7HHNH7EW4ZYKUD3IWA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 11: Drop support for AIX releases without dlopen
On Mon, Oct 19, 2020 at 4:43 PM Kevin Adler wrote: > Brett Cannon wrote: > > > Updated how? AIX is not mentioned in that PEP anywhere, so I'm not > quite > > sure what update you're suggesting. > > I'm referring to > https://www.python.org/dev/peps/pep-0011/#unsupporting-platforms. > > "If a certain platform that currently has special code in CPython is > deemed to be without > enough Python users or lacks proper support from the Python development > team and/or > a buildbot, a note must be posted in this PEP that this platform is no > longer actively supported." > > https://www.python.org/dev/peps/pep-0011/#no-longer-supported-platforms > > Should this list be updated to mention that AIX 5.3 and below are no > longer supported? > Only if we are going to rip out all AIX-related code. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4UXB3REOBM2X637DLAEE2666UNMPWWVQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On 20/10/2020 5:48 pm, Chris Angelico wrote: On Wed, Oct 21, 2020 at 3:31 AM Mark Shannon wrote: Hi Chris, On 20/10/2020 4:37 pm, Chris Angelico wrote: On Wed, Oct 21, 2020 at 12:03 AM Mark Shannon wrote: Hi everyone, CPython is slow. We all know that, yet little is done to fix it. I'd like to change that. I have a plan to speed up CPython by a factor of five over the next few years. But it needs funding. The overall aim is to speed up CPython by a factor of (approximately) five. We aim to do this in four distinct stages, each stage increasing the speed of CPython by (approximately) 50%. This is a very bold estimate. Particularly, you're proposing a number of small tweaks in stage 2 and expecting that (combined) they can give a 50% improvement in overall performance? 20 tweaks each providing a 2% is a 49% speedup. Stage 1 will open up optimizations that are currently worthwhile. Yes, I understand mathematics. Do you have evidence that shows that each of the twenty tweaks can give a two percent speedup though? My point was that small changes can easily add up to a large change. And yes, I have a long list of possible small optimizations. Do you have any details to back this up? You're not just asking for a proposal to be accepted, you're actually asking for (quite a bit of) money, and then hoping to find a contractor to do the actual work. I am offering to do the work. Sure, that takes away some of the uncertainty, but you're still asking for a considerable amount of money sight-unseen. I'm not asking for money up front. I'm asking for some promise of payment, once the work is done. If I fail, only I suffer a loss. BIG BIG concern: You're basically assuming that all this definition of performance is measured for repeated executions of code. That's how PyPy already works, and it most often suffers quite badly in startup performance to make this happen. Will your proposed changes mean that CPython has to pay the same startup costs that PyPy does? Could you clarify what you think I'm assuming? When you say start up, do you mean this? $ time python3 -S -c "" real0m0.010s $ time pypy -S -c "" real0m0.017s No, there would be no slower startup. In fact the tier 0 interpreter should start a fraction faster than 3.9. That's a microbenchmark, but yes, that's the kind of thing I'm talking about. For short scripts, will "python3.13 script.py" be slower than "python3.9 script.py"? Tiered execution means that 3.10+ should be no slower than 3.9 for any program, and faster for all but really short ones. Tier 0 would be a bit slower than 3.9, but will start faster. Tier 1 should kick in before 3.9 would catch up. Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HTFM5SREGCL3E232WSRXPTBP4S3DGVKO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Tue, 20 Oct 2020 at 14:01, Mark Shannon wrote: > > Hi everyone, > > CPython is slow. We all know that, yet little is done to fix it. > > I'd like to change that. > I have a plan to speed up CPython by a factor of five over the next few > years. But it needs funding. > > I am aware that there have been several promised speed ups in the past > that have failed. You might wonder why this is different. > > Here are three reasons: > 1. I already have working code for the first stage. > 2. I'm not promising a silver bullet. I recognize that this is a > substantial amount of work and needs funding. > 3. I have extensive experience in VM implementation, not to mention a > PhD in the subject. > > My ideas for possible funding, as well as the actual plan of > development, can be found here: > > https://github.com/markshannon/faster-cpython > > I'd love to hear your thoughts on this. A very blunt (apologies if it's too blunt) restatement of this proposal (at least stage 1) seems to me to be "Give me $250k, and donate $250k to the PSF for ongoing support, and I'll let you have code that I've already written that gives CPython a 50% speedup. If my code doesn't deliver a 50% speedup, I'll give the money back. No money, no code. We can also discuss 3 more incremental steps of the same sort of magnitude, but I don't have code already written for them". Honestly, if someone's able to get together $500k, that sounds like a fairly good deal (no risk). If you're asking for a commitment to the full $2M, even if stages 2-4 are contingent on delivery, that's a bit of a harder ask (because the cashflow implication of committing that sort of money becomes relevant). But maybe someone can do it. I'm fine with this, I guess. I don't have $500k to offer myself, so all my agreement is worth is that I don't have a problem with this much work being funded/provided via one big donation. I assume that part of "delivery" would involve code review, etc. - we wouldn't be bypassing normal development workflow. So there's still work to be done in putting the code through review, responding to review comments, etc, and ensuring that the code is delivered in a form that the core devs are happy to maintain (PSF donation for support notwithstanding). Actually, a chunk of that support allocation to the PSF might be needed to pay for reviewers, if (as I suspect) this is a large and complex PR. What I don't see is where the money's coming from. It's fine to ask, but will anyone come up with that sort of funding? Paul ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DNM27AX7Z3VULDY7XNJPVZYFX6Z5FTRZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Wed, Oct 21, 2020 at 3:38 AM Steven D'Aprano wrote: > > On Wed, Oct 21, 2020 at 02:37:02AM +1100, Chris Angelico wrote: > > > Do you have any details to back this up? You're not just asking for a > > proposal to be accepted, you're actually asking for (quite a bit of) > > money, and then hoping to find a contractor to do the actual work. > > Payment is on delivery. At each stage, if the contractor fails to > deliver the promised gains, they get nothing. > > (I believe that Mark is being polite by referring to a generic > contractor. I think he is referring to himself.) It's a little unclear from the proposal, as there was something about whether a suitable contractor could be found, but sure. TBH I'd be happier with this proposal as a direct offer/request for money than as "hey let's go look for potential coders", but it sounds like that's the plan anyway? > > That means you're expecting that anyone would be able to achieve this, > > given sufficient development time. > > With sufficient time, maybe the horse will learn to sing. > > https://everything2.com/title/And+maybe+the+horse+will+learn+how+to+sing > > But I don't think Mark believes *anyone* will be able to improve > performance. If it were that easy that anyone could do it, Python would > already be blazingly fast. Yeah. And the "anyone" part is the concern I had - that the proposal was asking for funding and then for a search for a contractor. But if it's "pay me and I'll write this", then it's a bit more concrete. > > BIG BIG concern: You're basically assuming that all this definition of > > performance is measured for repeated executions of code. > > That's not what the proposal says. > > "Performance should improve for all code, not just loop-heavy and > numeric code." > > In fact Mark goes further: he says that he's willing to allow some > performance degradation on loop heavy code, if the overall performance > increases. "Overall performance" is a myth, and there's no way that CPython will magically be able to execute any code with the exact same performance improvement. So my question is: what happens to startup performance, what happens to short scripts, what happens to the interpreter's load times, etc, etc, etc? It could be that all code becomes faster, but only after it's been run a few times. That would be great for, say, a web app - it handles a request, goes back and waits for another, and then handles the next one a bit faster - but not for a command-line script. (And yes, I'm aware that it'd theoretically be possible to dump the compiler state to disk, but that has its own risks.) > > What would happen if $2M were spent on improving PyPy3 instead? > > Then both of the PyPy3 users will be very happy *wink* > > (I know, that's a terrible, horrible, inaccurate slight on the PyPy > community, which is very probably thriving, and I would delete it if I > hadn't already hit Send.) Yes, you're being horribly insulting to the third user of PyPy3, who is probably right now warming up his interpreter so he can send you an angry response :) I guess my biggest concern with this proposal is that it's heavy on mathematically pure boasts and light on actual performance metrics, and I'm talking here about the part where (so we're told) the code is all done and it just takes a swipe of a credit card to unlock it. And without the ability to run it myself, I can't be sure that it'd actually give *any* performance improvement on my code or use-cases. So there's a lot that has to be taken on faith, and I guess I'm just a bit dubious of how well it'd fulfil that. ChrisA ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4N4SSXNDAP2GZQSL7F2JBC2RJVZ3WEL6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Wed, Oct 21, 2020 at 3:31 AM Mark Shannon wrote: > > Hi Chris, > > On 20/10/2020 4:37 pm, Chris Angelico wrote: > > On Wed, Oct 21, 2020 at 12:03 AM Mark Shannon wrote: > >> > >> Hi everyone, > >> > >> CPython is slow. We all know that, yet little is done to fix it. > >> > >> I'd like to change that. > >> I have a plan to speed up CPython by a factor of five over the next few > >> years. But it needs funding. > >> > > > >> The overall aim is to speed up CPython by a factor of (approximately) > >> five. We aim to do this in four distinct stages, each stage increasing the > >> speed of CPython by (approximately) 50%. > >> > > > > This is a very bold estimate. Particularly, you're proposing a number > > of small tweaks in stage 2 and expecting that (combined) they can give > > a 50% improvement in overall performance? > > 20 tweaks each providing a 2% is a 49% speedup. > Stage 1 will open up optimizations that are currently worthwhile. Yes, I understand mathematics. Do you have evidence that shows that each of the twenty tweaks can give a two percent speedup though? > > Do you have any details to back this up? You're not just asking for a > > proposal to be accepted, you're actually asking for (quite a bit of) > > money, and then hoping to find a contractor to do the actual work. > > I am offering to do the work. Sure, that takes away some of the uncertainty, but you're still asking for a considerable amount of money sight-unseen. > > BIG BIG concern: You're basically assuming that all this definition of > > performance is measured for repeated executions of code. That's how > > PyPy already works, and it most often suffers quite badly in startup > > performance to make this happen. Will your proposed changes mean that > > CPython has to pay the same startup costs that PyPy does? > > Could you clarify what you think I'm assuming? > > When you say start up, do you mean this? > > $ time python3 -S -c "" > > real0m0.010s > > $ time pypy -S -c "" > > real0m0.017s > > No, there would be no slower startup. In fact the tier 0 interpreter > should start a fraction faster than 3.9. That's a microbenchmark, but yes, that's the kind of thing I'm talking about. For short scripts, will "python3.13 script.py" be slower than "python3.9 script.py"? > > What would happen if $2M were spent on improving PyPy3 instead? > > The PSF loses $1M to spend on CPython maintenance, to start with. > > What would happen to PyPy? I have no idea. > > Partial success of speeding up CPython is very valuable. > Partial success in getting PyPy to support C extensions well and perform > well when it currently does, is much less valuable. > > CPython that is "only" 2 or 3 times faster is a major improvement, but a > PyPy that supports 80% of the C extensions that it currently does not is > still not a replacement for CPython. > True, but it does sound like you're pushing for the sorts of changes that PyPy already has. Not sure whether your proposed changes would be better done to PyPy or to CPython. ChrisA ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/I4XKZNAZBLPWCRS5EQ4UY6DMV3LDMJPN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Wed, Oct 21, 2020 at 02:38:25AM +1100, Chris Angelico wrote: > On Wed, Oct 21, 2020 at 12:55 AM Steven D'Aprano wrote: > > A minor point, and I realise that the costs are all in very round > > figures, but they don't quite match up: $2 million split over five > > stages is $400K per stage, not $500K. > > The proposal is for four stages. D'oh! I mean, I knew that, I was just checking to see if others were paying attention. Well done, you pass! -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UZFNJK636NJQOSURFW5ANZJS5F76LITU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Wed, Oct 21, 2020 at 02:37:02AM +1100, Chris Angelico wrote: > Do you have any details to back this up? You're not just asking for a > proposal to be accepted, you're actually asking for (quite a bit of) > money, and then hoping to find a contractor to do the actual work. Payment is on delivery. At each stage, if the contractor fails to deliver the promised gains, they get nothing. (I believe that Mark is being polite by referring to a generic contractor. I think he is referring to himself.) > That means you're expecting that anyone would be able to achieve this, > given sufficient development time. With sufficient time, maybe the horse will learn to sing. https://everything2.com/title/And+maybe+the+horse+will+learn+how+to+sing But I don't think Mark believes *anyone* will be able to improve performance. If it were that easy that anyone could do it, Python would already be blazingly fast. > BIG BIG concern: You're basically assuming that all this definition of > performance is measured for repeated executions of code. That's not what the proposal says. "Performance should improve for all code, not just loop-heavy and numeric code." In fact Mark goes further: he says that he's willing to allow some performance degradation on loop heavy code, if the overall performance increases. But why are we nitpicking Stage Two? The beauty of Mark's proposal is: 1. there is no committment to any stage until the previous stage is complete; 2. there is no committment to pay for any stage unless the contractor actually delivers the goods. If you don't think that Mark, or the contractor, will be able to deliver a 50% speed up in Stage 2, what have you got to lose? If he fails to deliver, you pay nothing and don't commit to Stage 3. If he does deliver, you get the desired result. (The details allow for some slack: if the speed up is only 49%, the contractor still gets paid. Presumably it will be on some sort of sliding scale.) > That's how > PyPy already works, and it most often suffers quite badly in startup > performance to make this happen. Will your proposed changes mean that > CPython has to pay the same startup costs that PyPy does? Good question. > What would happen if $2M were spent on improving PyPy3 instead? Then both of the PyPy3 users will be very happy *wink* (I know, that's a terrible, horrible, inaccurate slight on the PyPy community, which is very probably thriving, and I would delete it if I hadn't already hit Send.) I think this isn't a matter of just throwing money at a project. Mark knows the CPython architecture. I don't know if he knows the PyPy architecture, and unless a PyPy expert comes up with a counter proposal, we don't know that spending $2M on PyPy will see any sort of comparable gains. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WVVDCZ34LY5UD6LEZBH33GFP7BSMFTRL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
Hi Chris, On 20/10/2020 4:37 pm, Chris Angelico wrote: On Wed, Oct 21, 2020 at 12:03 AM Mark Shannon wrote: Hi everyone, CPython is slow. We all know that, yet little is done to fix it. I'd like to change that. I have a plan to speed up CPython by a factor of five over the next few years. But it needs funding. The overall aim is to speed up CPython by a factor of (approximately) five. We aim to do this in four distinct stages, each stage increasing the speed of CPython by (approximately) 50%. This is a very bold estimate. Particularly, you're proposing a number of small tweaks in stage 2 and expecting that (combined) they can give a 50% improvement in overall performance? 20 tweaks each providing a 2% is a 49% speedup. Stage 1 will open up optimizations that are currently worthwhile. Do you have any details to back this up? You're not just asking for a proposal to be accepted, you're actually asking for (quite a bit of) money, and then hoping to find a contractor to do the actual work. I am offering to do the work. That means you're expecting that anyone would be able to achieve this, given sufficient development time. No, I can (with paid help) achieve this. What matters is that someone can, not that anyone can. BIG BIG concern: You're basically assuming that all this definition of performance is measured for repeated executions of code. That's how PyPy already works, and it most often suffers quite badly in startup performance to make this happen. Will your proposed changes mean that CPython has to pay the same startup costs that PyPy does? Could you clarify what you think I'm assuming? When you say start up, do you mean this? $ time python3 -S -c "" real0m0.010s $ time pypy -S -c "" real0m0.017s No, there would be no slower startup. In fact the tier 0 interpreter should start a fraction faster than 3.9. What would happen if $2M were spent on improving PyPy3 instead? The PSF loses $1M to spend on CPython maintenance, to start with. What would happen to PyPy? I have no idea. Partial success of speeding up CPython is very valuable. Partial success in getting PyPy to support C extensions well and perform well when it currently does, is much less valuable. CPython that is "only" 2 or 3 times faster is a major improvement, but a PyPy that supports 80% of the C extensions that it currently does not is still not a replacement for CPython. Cheers, Mark. ChrisA ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3NBP3KLTMXNDJ2ME4QPSATW2ZIMKVICG/ Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/575BK2RBWDGXL4DNRJO5AM3GLXRCH45Q/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Wed, Oct 21, 2020 at 12:55 AM Steven D'Aprano wrote: > A minor point, and I realise that the costs are all in very round > figures, but they don't quite match up: $2 million split over five > stages is $400K per stage, not $500K. The proposal is for four stages. ChrisA ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DFFCUKWRBR557SWJEQPUFV6WEKWLFKKR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Wed, Oct 21, 2020 at 12:03 AM Mark Shannon wrote: > > Hi everyone, > > CPython is slow. We all know that, yet little is done to fix it. > > I'd like to change that. > I have a plan to speed up CPython by a factor of five over the next few > years. But it needs funding. > > The overall aim is to speed up CPython by a factor of (approximately) five. > We aim to do this in four distinct stages, each stage increasing the speed of > CPython by (approximately) 50%. > This is a very bold estimate. Particularly, you're proposing a number of small tweaks in stage 2 and expecting that (combined) they can give a 50% improvement in overall performance? Do you have any details to back this up? You're not just asking for a proposal to be accepted, you're actually asking for (quite a bit of) money, and then hoping to find a contractor to do the actual work. That means you're expecting that anyone would be able to achieve this, given sufficient development time. BIG BIG concern: You're basically assuming that all this definition of performance is measured for repeated executions of code. That's how PyPy already works, and it most often suffers quite badly in startup performance to make this happen. Will your proposed changes mean that CPython has to pay the same startup costs that PyPy does? What would happen if $2M were spent on improving PyPy3 instead? ChrisA ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3NBP3KLTMXNDJ2ME4QPSATW2ZIMKVICG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On 20/10/2020 2:47 pm, Steven D'Aprano wrote: A very interesting proposal. A couple of thoughts... Can we have an executive summary of how your proposed approach differs from those of PyPy, Unladen Swallow, and various other attempts? https://github.com/markshannon/faster-cpython/blob/master/tiers.md should cover it. You suggest that payment should be on delivery, or meeting the target, rather than up-front. That's good for the PSF, but it also means that the contractor not only takes all the risk of failure, but also needs an independent source of income, or at least substantial savings (enough for, what, eighteen months development per stage?). Doesn't that limit the available pool of potential contractors? We only need one. I don't think financial constraints are the main problem. I think domain knowledge is probably more of a constraint. I think there's always tension between community driven development and paid work. If the PSF pays person A to develop something, might not people B, C, D and E feel slighted that they didn't get paid? The PSF already pays people to work on PyPI On the other hand, I guess we already deal with that. There are devs who are paid by their employers to work on Python for N hours a months, for some value of N, or to develop something and then open source it. And then there are devs who aren't. You have suggested that the cost of each stage be split 50:50 between development and maintenance. But development is a one-off cost; maintenance is an forever cost, and unpredictable, and presumably some of that maintenance will be done by people other than the contractor. Any new feature will require ongoing maintenance. I'm just suggesting that we budget for it. Who is going to pay for the maintenance of PEP 634? A minor point, and I realise that the costs are all in very round figures, but they don't quite match up: $2 million split over five stages is $400K per stage, not $500K. I meant four stages. Did I write "five" somewhere? 1. I already have working code for the first stage. I don't mean to be negative, or hostile, but this sounds like you are saying "I have a patch for Python that will make it 1.5 times faster, but you will never see it unless you pay me!" I believe that's how business works ;) I have this thing, e.g an iPhone, if you want it you must pay me. I think that speeding CPython 50% is worth a few hundred iPhones. I realise that is a very uncharitable way of looking at it, sorry about that, it's nothing personal. But $500K is a lot of money. Remember the contractor only gets roughly half of that. The rest stays with the PSF to fund maintenance of CPython. $250k only pays for one engineer for one year at one of the big tech firms. Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NDLJBM35DCRSOGBEYRDUAELVOHZKFSWU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Tue, Oct 20, 2020 at 01:41:02PM +0200, Thomas Wouters wrote: > > a, *?, b = expression > > print(?) # wait this doesn't work; > > > > I'm not sure how this is different than, say, > > a, _, _ = range(3) > print(_) For starters, that will actually print something, not fail with a SyntaxError or runtime exception. You are correct that if I use the same `_` variable in multiple places: a, _, _, b, _ = something and then need to see the "ignore" values for debugging, I can't. I will only see the value in the final `_` unless I rename them. But with your proposal, I can't even see a *single* `?` (non-)target. So that's even more inconvenient than `_` because it affects more cases. - `_` is inconvenient for debugging if you have two or more in one assignment statement; - `?` is inconvenient for debugging if you have two or more in one assignment statement; - PLUS if you have only a single one in the statement. So strictly more inconvenient. Each time I use a `?` target, I expect that eventually I will need to debug that piece of code. When I need to debug it, I will need to change the `?` into a real variable. I don't plan to change it back: why churn the code unnecessarily? So there's this ratchett effect, where over time every `?` I use will eventually be converted into a real variable. So why not just use real variables in the first place? Unless there is a seriously large performance gain from `?` I wouldn't bother using it in the first place. I don't hate the idea, but I don't think it's very useful either. > That is to say, some things are impossible; if you want to print the value, > don't assign to '?'. Indeed, but the problem is that when I name the variable *today*, I might not know that *tomorrow* I will need to see its value. > > In my opinion, having a convention to treat certain variables as > > "unused" is great (I'm partial to `__` myself, to avoid clobbering the > > special variable `_` in the REPL). But having that be a pseudo-variable > > which is *actually* unused and unuseable strikes me as being an > > attractive nuisance. > > > > And yet that's exactly what is being proposed in pattern matching. > https://www.python.org/dev/peps/pep-0634/#id3 Quote: "A wildcard pattern always succeeds. It binds no name." That says nothing about variables. It defines a *pattern* which always matches anything. Patterns are a different kind of thing to variables, just as keywords are not variables. A better argument for your position woud be this quote from a different PEP, 635: "The wildcard pattern is a special case of a 'capture' pattern: it accepts any value, but does not bind it to a variable." https://www.python.org/dev/peps/pep-0635/#id2 But I think that description makes a conceptual error. I think it is a mistake to think of the wildcard pattern as a non-binding capture pattern, since that's a contradiction: if it never captures any value, how can it be a capture pattern? I think that it is important to recognise that the wildcard pattern is *distinct* from capture patterns, literal patterns etc. Like literal patterns, it captures nothing. Unlike literal and other patterns, it matches everything. There's a certainly similarity to capture patterns No, that's not "exactly what is being proposed". The *pattern* "_" is not an assignment target. Capture patterns are similar to assigment targets, but patterns in general are not: case [] does not attempt to use [] as an assignment target. Likewise for value patterns such as `case HttpStatus.OK`. Value patterns like HttpStatus.OK are perfectly legal assignment targets, but that's not what they mean inside a case statement. Symbols having multiple meanings are not so unusual: - inside a float, a . is part of the float; - outside of a float, a . is an attribute delimiter; - the chars 'None' have special meaning on their own; - but otherwise inside an identifier, or a string, they're just letters; - round brackets (parentheses) can mean grouping or calling; - square brackets create lists, or subscripting; etc. Why are we so hung up over the fact that inside a case statement, the underscore means something different from outside of a match statement? It seems like a very minor issue to worry about. If we can accept that `None` and other keywords are syntactically legal identifiers, but not actually usable as identifiers as they are reserved for other purposes, then we ought to be able to accept that `_` is syntactically a legal assignment target, but not usable as a capture pattern as it is reserved for another purpose (wildcard pattern). -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
[Python-Dev] Re: Speeding up CPython
On 10/20/20 2:53 PM, Mark Shannon wrote: > I'd love to hear your thoughts on this. a VM needs a separate backend for each architecture (maybe even OS) - which architectures do you include into your proposal? what's your estimate for a new port? - do you plan for a fall-back to a slow non-VM mode, e.g. the existing one? Do you plan to support that in parallel such that Python still can be used on architectures with a working compiler (and a libffi port). E.g. OpenJDK has the zero port, which is slow (interpreted), but runs on practically all architectures. Thanks, Matthias ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H4O5TFWGL5SDG4NSESNGNKIZMK6W7L7V/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
Hi Antoine, On 20/10/2020 2:32 pm, Antoine Pitrou wrote: On Tue, 20 Oct 2020 13:53:34 +0100 Mark Shannon wrote: Hi everyone, CPython is slow. We all know that, yet little is done to fix it. I'd like to change that. I have a plan to speed up CPython by a factor of five over the next few years. But it needs funding. I am aware that there have been several promised speed ups in the past that have failed. You might wonder why this is different. Here are three reasons: 1. I already have working code for the first stage. 2. I'm not promising a silver bullet. I recognize that this is a substantial amount of work and needs funding. 3. I have extensive experience in VM implementation, not to mention a PhD in the subject. My ideas for possible funding, as well as the actual plan of development, can be found here: https://github.com/markshannon/faster-cpython Do you plan to do all this in C, or would you switch to C++ (or something else)? All C, no C++. I promise :) Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DZUH4PSWAD7MFVVXS3RBYFHVTCFLC4ZA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Pickle for C extension?
Ah, okay. This is a problem for me, since I can't pass the 5th argument to __reduce__, since it's an immutable type. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JYVCHTK2NIZI4BQDV33U6QYMM6FYY6XQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
Where is your working code for the first stage? October 20, 2020 8:53 AM, "Mark Shannon" wrote: > Hi everyone, > > CPython is slow. We all know that, yet little is done to fix it. > > I'd like to change that. > I have a plan to speed up CPython by a factor of five over the next few > years. But it needs > funding. > > I am aware that there have been several promised speed ups in the past that > have failed. You might > wonder why this is different. > > Here are three reasons: > 1. I already have working code for the first stage. > 2. I'm not promising a silver bullet. I recognize that this is a substantial > amount of work and > needs funding. > 3. I have extensive experience in VM implementation, not to mention a PhD in > the subject. > > My ideas for possible funding, as well as the actual plan of development, can > be found here: > > https://github.com/markshannon/faster-cpython > > I'd love to hear your thoughts on this. > > Cheers, > Mark. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/RDXLCH22T2EZDRCBM6ZYYIUTBWQVVVWH > Code of Conduct: http://python.org/psf/codeofconduct ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DXTPGPEKDJTVLEWWMD2JT7HAWD7P6K5E/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: os.scandir bug in Windows?
On 10/19/20, Greg Ewing wrote: > On 20/10/20 4:52 am, Gregory P. Smith wrote: >> Those of us with a traditional posix filesystem background may raise >> eyeballs at this duplication, seeing a directory as a place that merely >> maps names to inodes > > This is probably a holdover from MS-DOS, where there was no separate > inode-like structure -- it was all in the directory entry. DOS implemented a find-first/find-next API (int 21h 4E/4F) that provided a file's name, attributes, size, and last write time/date. I think it's clear that the design was influenced by the readily-available contents of a FAT dirent. The Win32 API extended this to FindFirstFile/FindNextFile, with added support for the long filename, create and access times, and, in NT 5+, the reparse tag for a reparse point. NTFS had to support this metadata in the directory index, else FindFirstFile/FindNextFile would be too expensive if the filesystem had to fetch the metadata from the MFT for every matching file in a listing. It tries to keep the duplicated metadata in sync -- such as when a file is open, closed, manually extended in size, when the cache is flushed, or when metadata is explicitly set (e.g. SetFileInformationByHandle: FileBasicInfo). But for performance it doesn't update the duplicated data every time a file is read from or written to. And, in particular, if it's just the access time that changed, it updates the duplicated access time with a one-hour granularity. (There's also a registry value, as I mentioned previously, that disables updating access times completely -- in both the MFT record and the directory index.) That said, if a file has multiple hardlinks the current NTFS implementation for updating duplicated data is totally unreliable. It only updates the accessed link. All other links go stale. We don't have any reasonable way to special case this situation because the directory entry doesn't include the number of links a file has. It has to be opened and queried directly, but then one might as well do a full stat() for every file. I recommend relying on only the high-level is_dir(), is_file(), and is_symlink() methods of os.scandir() items, to quickly process a directory. inode() is reliable -- as much as is possible in Windows -- because the implementation gets the full stat info, but check to ensure it's not 0. It's based on the file ID, which Windows filesystems aren't required to support (or reliably support; it's not stable in FAT). NTFS and ReFS support reliable 64-bit file IDs, and opening by file ID. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JKK47AWKUOWPPBEAIRGIFRMW6FCPZILG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
A very interesting proposal. A couple of thoughts... Can we have an executive summary of how your proposed approach differs from those of PyPy, Unladen Swallow, and various other attempts? You suggest that payment should be on delivery, or meeting the target, rather than up-front. That's good for the PSF, but it also means that the contractor not only takes all the risk of failure, but also needs an independent source of income, or at least substantial savings (enough for, what, eighteen months development per stage?). Doesn't that limit the available pool of potential contractors? I think there's always tension between community driven development and paid work. If the PSF pays person A to develop something, might not people B, C, D and E feel slighted that they didn't get paid? On the other hand, I guess we already deal with that. There are devs who are paid by their employers to work on Python for N hours a months, for some value of N, or to develop something and then open source it. And then there are devs who aren't. You have suggested that the cost of each stage be split 50:50 between development and maintenance. But development is a one-off cost; maintenance is an forever cost, and unpredictable, and presumably some of that maintenance will be done by people other than the contractor. A minor point, and I realise that the costs are all in very round figures, but they don't quite match up: $2 million split over five stages is $400K per stage, not $500K. > 1. I already have working code for the first stage. I don't mean to be negative, or hostile, but this sounds like you are saying "I have a patch for Python that will make it 1.5 times faster, but you will never see it unless you pay me!" I realise that is a very uncharitable way of looking at it, sorry about that, it's nothing personal. But $500K is a lot of money. If the PSF says "No thanks", what happens to your code? - delete it; - donate it to Python for free; - fork Python and try to make a commercial, non-FOSS version that you can sell to recoup your development time; - something else? If this was a closed-source proprietary project, there would be no question in my mind. You took a bet that you could sell the code, and you lost. You swallow your loss and move on, that's how the proprietary world works. But this is FOSS and community driven, and I don't think that fits well with that model. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OFQKVMU4QPKVD2NIOTZOK5H4HA2RQLHK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Tue, 20 Oct 2020 at 14:26, Thomas Wouters wrote: > I'm not sure how to put it differently than I have in the PEP or the email: I > proposed they use ? instead of _ and also apply that to regular unpacking > (because it is very easy to see pattern matching as an extension of unpacking > assignment), and (besides other disagreements) they were uncomfortable > including non-pattern-matching proposals in their PEP. This PEP covers the > non-pattern-matching uses of '?'. OK, fair. In which case, I'll probably wait for the revised pattern matching PEP, but I expect to be -1 on using ? there as well. I was fine with _ and the special rule TBH. Paul ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NXHCGZ3P7GPRPEVTVBZVL2JEOFZWRHTX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Speeding up CPython
On Tue, 20 Oct 2020 13:53:34 +0100 Mark Shannon wrote: > Hi everyone, > > CPython is slow. We all know that, yet little is done to fix it. > > I'd like to change that. > I have a plan to speed up CPython by a factor of five over the next few > years. But it needs funding. > > I am aware that there have been several promised speed ups in the past > that have failed. You might wonder why this is different. > > Here are three reasons: > 1. I already have working code for the first stage. > 2. I'm not promising a silver bullet. I recognize that this is a > substantial amount of work and needs funding. > 3. I have extensive experience in VM implementation, not to mention a > PhD in the subject. > > My ideas for possible funding, as well as the actual plan of > development, can be found here: > > https://github.com/markshannon/faster-cpython Do you plan to do all this in C, or would you switch to C++ (or something else)? Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N65UEJTSX35KJOSSUFZNKKJPT224CNRS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Tue, Oct 20, 2020 at 2:44 PM Paul Moore wrote: > On Tue, 20 Oct 2020 at 13:25, Thomas Wouters wrote: > > > > On Tue, Oct 20, 2020 at 2:22 PM Paul Moore wrote: > >> > >> On Tue, 20 Oct 2020 at 13:13, Thomas Wouters wrote: > >> > The reason for this PEP is that pattern matching will make '_' (but > not any other names) have the behaviour suggested in this PEP, but *only* > in pattern matching. > >> > >> That's something that should be addressed or debated in the pattern > >> matching PEP. I'm -1 on this PEP being *solely* to patch over a wart > >> in the pattern matching PEP, and the other justifications for the PEP > >> as a standalone proposal don't seem to be convincing people (they > >> don't convince me either, FWIW). > > > > > > I did say, in the original email: > > > > This proposal doesn't necessarily require pattern matching to be > accepted -- the new syntax stands well enough on its own -- but I'm > recommending this *not* be accepted if pattern matching using the same > syntax is not also accepted. The benefit without pattern matching is real > but small, and in my opinion it's not worth the added complexity. > > Understood. But unless I'm missing something, the pattern matching > PEP(s) is/are in limbo at the moment, there's a lot going on in github > but nothing has been posted here. So I'm not clear what there is to > discuss here at the moment, if the proposal is only relevant if > pattern matching includes it, but no published pattern matching PEP > has suggested it... > They are not in limbo. They are actively being worked on. (At the sprints Brandt mentioned they expect to post updated PEPs later this week.) The Steering Council had a conversation with the PEP authors a while back, discussing various objections and alternatives, including using something else instead of '_'. At that time they were already talking about splitting the PEP up into three parts (which they've since done, but not posted about yet). I'm not sure how to put it differently than I have in the PEP or the email: I proposed they use ? instead of _ and also apply that to regular unpacking (because it is very easy to see pattern matching as an extension of unpacking assignment), and (besides other disagreements) they were uncomfortable including non-pattern-matching proposals in their PEP. This PEP covers the non-pattern-matching uses of '?'. (Sorry if the above sounds a little disgruntled, it feels like there's > a lot going in "in private" with the pattern matching PEP and I sort > of feel like a bit more transparency would be good. Maybe I'm > mistaken...) > It's not so much 'private' as 'in separate groups', and they're really still processing all the feedback they've received about PEP 622. There's a #pattern-matching channel on the discord server used for the core dev sprints right now (that all sprinters have access to), and the work on PEPs 634, 635 and 636 is happening on the peps repo. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NDSTSSO64JN5WBUNO3DSN2QZ6XYVMIEU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Speeding up CPython
Hi everyone, CPython is slow. We all know that, yet little is done to fix it. I'd like to change that. I have a plan to speed up CPython by a factor of five over the next few years. But it needs funding. I am aware that there have been several promised speed ups in the past that have failed. You might wonder why this is different. Here are three reasons: 1. I already have working code for the first stage. 2. I'm not promising a silver bullet. I recognize that this is a substantial amount of work and needs funding. 3. I have extensive experience in VM implementation, not to mention a PhD in the subject. My ideas for possible funding, as well as the actual plan of development, can be found here: https://github.com/markshannon/faster-cpython I'd love to hear your thoughts on this. Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RDXLCH22T2EZDRCBM6ZYYIUTBWQVVVWH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Tue, 20 Oct 2020 at 13:25, Thomas Wouters wrote: > > On Tue, Oct 20, 2020 at 2:22 PM Paul Moore wrote: >> >> On Tue, 20 Oct 2020 at 13:13, Thomas Wouters wrote: >> > The reason for this PEP is that pattern matching will make '_' (but not >> > any other names) have the behaviour suggested in this PEP, but *only* in >> > pattern matching. >> >> That's something that should be addressed or debated in the pattern >> matching PEP. I'm -1 on this PEP being *solely* to patch over a wart >> in the pattern matching PEP, and the other justifications for the PEP >> as a standalone proposal don't seem to be convincing people (they >> don't convince me either, FWIW). > > > I did say, in the original email: > > This proposal doesn't necessarily require pattern matching to be accepted -- > the new syntax stands well enough on its own -- but I'm recommending this > *not* be accepted if pattern matching using the same syntax is not also > accepted. The benefit without pattern matching is real but small, and in my > opinion it's not worth the added complexity. Understood. But unless I'm missing something, the pattern matching PEP(s) is/are in limbo at the moment, there's a lot going on in github but nothing has been posted here. So I'm not clear what there is to discuss here at the moment, if the proposal is only relevant if pattern matching includes it, but no published pattern matching PEP has suggested it... (Sorry if the above sounds a little disgruntled, it feels like there's a lot going in "in private" with the pattern matching PEP and I sort of feel like a bit more transparency would be good. Maybe I'm mistaken...) Paul ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZZLSVASCO2NMXDCMAEEU4MF2A5727PTN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On 20Oct2020 1309, Thomas Wouters wrote: The reason for this PEP is that pattern matching will make '_' (but not any other names) have the behaviour suggested in this PEP, but *only* in pattern matching. Then why is this PEP proposing a different syntax? At the very least, wait for pattern matching to get in before proposing an expansion, so then you won't be caught out suggesting the wrong thing :) Cheers, Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RXIACSYMIP22ILGYYFVPKHIUH62N7233/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Tue, Oct 20, 2020 at 2:22 PM Paul Moore wrote: > On Tue, 20 Oct 2020 at 13:13, Thomas Wouters wrote: > > The reason for this PEP is that pattern matching will make '_' (but not > any other names) have the behaviour suggested in this PEP, but *only* in > pattern matching. > > That's something that should be addressed or debated in the pattern > matching PEP. I'm -1 on this PEP being *solely* to patch over a wart > in the pattern matching PEP, and the other justifications for the PEP > as a standalone proposal don't seem to be convincing people (they > don't convince me either, FWIW). > I did say, in the original email: This proposal doesn't necessarily require pattern matching to be accepted -- the new syntax stands well enough on its own -- but I'm recommending this *not* be accepted if pattern matching using the same syntax is not also accepted. The benefit without pattern matching is real but small, and in my opinion it's not worth the added complexity. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZPDPG3SX5WLRR3ZATSO54A26GASGUEQ6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Tue, 20 Oct 2020 at 13:13, Thomas Wouters wrote: > The reason for this PEP is that pattern matching will make '_' (but not any > other names) have the behaviour suggested in this PEP, but *only* in pattern > matching. That's something that should be addressed or debated in the pattern matching PEP. I'm -1 on this PEP being *solely* to patch over a wart in the pattern matching PEP, and the other justifications for the PEP as a standalone proposal don't seem to be convincing people (they don't convince me either, FWIW). Paul ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S44G7E725R44WG44VA7YNE7TDSHVQUVG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Tue, Oct 20, 2020 at 2:02 PM Chris Jerdonek wrote: > On Mon, Oct 19, 2020 at 3:11 PM Thomas Wouters wrote: > >> PEP: 640 >> Title: Unused variable syntax >> Author: Thomas Wouters >> > ... > >> In Python it is somewhat common to need to do an assignment without >> actually >> needing the result. Conventionally, people use either ``"_"`` or a name >> such >> as ``"unused"`` (or with ``"unused"`` as a prefix) for this. It's most >> common in *unpacking assignments*:: >> > > Many times I'm not using an assignment target, I still like to give a > descriptive name. The reason is that it lets me see what value I'm not > using. It helps to document and confirm my understanding of the value being > unpacked. It also lets you toggle easily between using and not using a > value if you're working on the code. > > To illustrate, I might do this-- > > scheme, _netloc, _path, params, query, fragment = urlparse(url) > > instead of this-- > > scheme, _, _, params, query, fragment = urlparse(url) > > So I'd prefer if the scheme would allow including a name (either by > prefixing or some other method), or at least not preclude such an extension > in the future. > It does not preclude it -- ?somename is not valid syntax, so it could be added later -- but please note that the pattern matching proposal also does not allow this. Using names instead of ? is still an option -- both in regular unpacking and in pattern matching -- it just does something subtly different. The reason for this PEP is that pattern matching will make '_' (but not any other names) have the behaviour suggested in this PEP, but *only* in pattern matching. > > --Chris > > > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4AVMQ3ZW5IEMSR2WWWMDAAK6Y36CH4JD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Mon, Oct 19, 2020 at 3:11 PM Thomas Wouters wrote: > PEP: 640 > Title: Unused variable syntax > Author: Thomas Wouters > ... > In Python it is somewhat common to need to do an assignment without > actually > needing the result. Conventionally, people use either ``"_"`` or a name > such > as ``"unused"`` (or with ``"unused"`` as a prefix) for this. It's most > common in *unpacking assignments*:: > Many times I'm not using an assignment target, I still like to give a descriptive name. The reason is that it lets me see what value I'm not using. It helps to document and confirm my understanding of the value being unpacked. It also lets you toggle easily between using and not using a value if you're working on the code. To illustrate, I might do this-- scheme, _netloc, _path, params, query, fragment = urlparse(url) instead of this-- scheme, _, _, params, query, fragment = urlparse(url) So I'd prefer if the scheme would allow including a name (either by prefixing or some other method), or at least not preclude such an extension in the future. --Chris ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IMU3N7YIFVBNAXWSWQ2BMXJUPNEZWCI7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On 20Oct2020 1021, Steven D'Aprano wrote: In my opinion, having a convention to treat certain variables as "unused" is great (I'm partial to `__` myself, to avoid clobbering the special variable `_` in the REPL). But having that be a pseudo-variable which is *actually* unused and unuseable strikes me as being an attractive nuisance. I agree entirely. The convention is fine, and the workaround for when you don't want to overwrite a legitimate `_` variable is also fine. Making `_` enough of an official convention (but how?) that linting tools stop warning about it (e.g. type checkers might warn about multiple conflicting assignments) seems like an overall happier path, that neither makes existing code forwards-incompatible nor future code backwards-incompatible. I don't think we have to codify every emergent coding pattern in syntax. Cheers, Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6GV3KPPPRNF5ISZK4YSAIUUTCQRMX77H/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: os.scandir bug in Windows?
On 20Oct2020 0520, Rob Cliffe wrote: On 19/10/2020 12:42, Steve Dower wrote: On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote: TLDR: In os.scandir directory entries, atime is always a copy of mtime rather than the actual access time. Correction - os.stat() updates the access time to _now_, while os.scandir() returns the last access time without updating it. Eryk replied with a deeper explanation of the cause, but fundamentally this is what you are seeing. Feel free to file a bug, but we'll likely only add a vague note to the docs about how Windows works here rather than changing anything. If anything, we should probably fix os.stat() to avoid updating the access time so that both functions behave the same, but that might be too complicated. Cheers, Steve Sorry - what you say does not match the behaviour I observe, which is that Yes, I posted a correction already (immediately after sending the first email). What you are seeing is what Windows decided was the best approach. If you want to avoid that, os.stat() will get the latest available information. But I don't want to penalise people who don't need it by slowing down their scandir calls unnecessarily. A documentation patch to make this difference between os.stat() and DirEntry even clearer would be fine. Cheers, Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NAR7LTW2XMBKAPKLVBQQFVK6EA4ZWQZP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
On Tue, Oct 20, 2020 at 11:26 AM Steven D'Aprano wrote: > I can't say that I like the look of pseudo-assignment to question mark: > > for ? in range(20): > ... > > but I could probably learn to live with it. But one of your > rationalisations: > > > > and makes it more obvious that > > the actual intent is for the value to be unused -- since it is entirely > > impossible to use it. > > is actually an anti-feature, in my opinion. > > I think that people might like the idea of not actually binding a value > in situations like this: > > a, *?, b = expression > > until you end up with something unexpected in a and b and need to debug > what it going on, either in a debugger or with print: > > a, *?, b = expression > print(?) # wait this doesn't work; > I'm not sure how this is different than, say, a, _, _ = range(3) print(_) or a = range(3)[0] print() That is to say, some things are impossible; if you want to print the value, don't assign to '?'. > > In my opinion, having a convention to treat certain variables as > "unused" is great (I'm partial to `__` myself, to avoid clobbering the > special variable `_` in the REPL). But having that be a pseudo-variable > which is *actually* unused and unuseable strikes me as being an > attractive nuisance. > And yet that's exactly what is being proposed in pattern matching. https://www.python.org/dev/peps/pep-0634/#id3 > > -- > Steve > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/OPUUJWBCK37CQXOYIFYSUIXFQSWTTCCA/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/I7S4C7CWW2LGELO7KA2PI4SNUT6ITAZE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Pickle for C extension?
19.10.20 20:39, Marco Sulla пише: > TL;DR Is it possible to use C code to implement the (un)pickling of an > type written in a C extension, as it was written in _pickle.c? > > Long explaining: I'm trying to create a C extension for frozendict. For > simplicity, first I wrote it in CPython, then I started to move it in a > C extension. It seems to work, but I have to move the code I wrote in > _pickle.c and pickle.py in the C extension. Is it possible, or I have to > create a slower `__reduce_ex__` method that simply converts it to dict? > > This is, for example, the C code for pickling frozendict in _pickle.c: > https://github.com/Marco-Sulla/cpython/blob/41a640a947c36007e56bbc28f362c261110d2001/Modules/_pickle.c#L3370 For adding new opcode to the pickle format you need a new pickle version and a PEP. Implement __reduce_ex__, it is the way to support pickling of new types (as it was for sets, bytearrays, and many extension types). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WLBLDASK25BWANWS347U4HIOBA763IEW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 640: Unused variable syntax.
I can't say that I like the look of pseudo-assignment to question mark: for ? in range(20): ... but I could probably learn to live with it. But one of your rationalisations: > and makes it more obvious that > the actual intent is for the value to be unused -- since it is entirely > impossible to use it. is actually an anti-feature, in my opinion. I think that people might like the idea of not actually binding a value in situations like this: a, *?, b = expression until you end up with something unexpected in a and b and need to debug what it going on, either in a debugger or with print: a, *?, b = expression print(?) # wait this doesn't work In my opinion, having a convention to treat certain variables as "unused" is great (I'm partial to `__` myself, to avoid clobbering the special variable `_` in the REPL). But having that be a pseudo-variable which is *actually* unused and unuseable strikes me as being an attractive nuisance. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OPUUJWBCK37CQXOYIFYSUIXFQSWTTCCA/ Code of Conduct: http://python.org/psf/codeofconduct/
