Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On 23 May 2013 22:02, Ronan Lamy wrote: > 2013/5/23 Łukasz Langa >> Last one wins. Just like with assigning names in a scope, defining methods >> in a class or overriding them in a subclass. > > This is a serious annoyance, considering that there are several places where > a large library can reasonably define the implementations (i.e. with the > class, with the function, or in some utility module). Note that in contrast > with the case of functions in a module or methods in a class, linting tools > cannot be expected to detect a duplication between functions with different > names defined in different modules. But isn't it much much worse than names in scope, as with assigning names in a scope it is only your scope that is affected : from os.path import join def join(wibble): 'overloads join in this module only' any other module is unaffected, os.path.join still calls os.path.join however with this all scopes globally are affected by the last one wins rule. -default.py--- from pkgutil import simplegeneric @simplegeneric def fun(x): print 'default impl' ---a.py from default import fun @fun.register(int) def impl_a(x): print 'impl_a' def f(): fun(0) # expect this to call impl_a ---b.py-- from default import fun @fun.register(int) def impl_b(x): print 'impl_b' def f(): fun(0) # expect this to call impl_b >>> import a, b >>> a.f() impl_b >>> b.f() impl_b >>> import b, a >>> a.f() impl_a >>> b.f() impl_a >>> exit() That is rather worrying. It is more analagous in the above example to sys.modules['os.path'].join = myjoin I don't have a solution mind though. Sam ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP397 no command line options to python?
Hello all, I was surprised to see that the excellent pylauncher doesn't do the magic shebang processing if you give it any python command line options. e.g. Given #!/usr/bin/env python2.6 import sys print(sys.executable) C:\>py test.py C:\Python26\python.exe C:\>py -utt test.py C:\Python27\python.exe It is spelled out that it shouldn't do so in the pep : "Only the first command-line argument will be checked for a shebang line and only if that argument does not start with a '-'." But I can't really see why that should be the case. What is the rational behind this? It is very surprising to the user that adding a simple option like -tt should change the way the launcher behaves. The PEP also states that the launcher should show the python help if '-h' is specified : "If the only command-line argument is "-h" or "--help", the launcher will print a small banner and command-line usage, then pass the argument to the default Python. This will cause help for the launcher being printed followed by help for Python itself. The output from the launcher will clearly indicate the extended help information is coming from the launcher and not Python." To me that would suggest to end users that they can use any of the command line options with the launcher, and they should behave as if you had called python directly. I am directing this to python-dev because this pylauncher is merely implementing the PEP as it currently stands, so the PEP would ideally need to be modified before the launcher. I would change the that first paragraph of the PEP to read something like : " The first command-line argument not beginning with a '-' will be checked for a shebang line, but only if : * That command-line argument is not preceded by a '-c' or '-m', and providing * There is no explicit version specifier, e.g. -2.7 as documumented later in this PEP. " Incidentally whilst implementing this I also noticed a bug in the pylauncher whereby the py launcher would incorrectly treat "py t3" as a request for python version as if '-3' had been specified. I have a small patch that fixes this and implements the above for pylauncher with extra tests if there is interest. Sam PS I have been lurking for a while, hello everyone. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP397 no command line options to python?
On 17 October 2011 13:23, Mark Hammond wrote: > On 17/10/2011 9:10 PM, Sam Partington wrote: >> >> "Only the first command-line argument will be checked for a shebang line >> and only if that argument does not start with a '-'." >> >> But I can't really see why that should be the case. What is the >> rational behind this? > > It really is a combination of 2 things: > > * The key use-case for the launcher is to be executed implicitly - ie, the > user types just "foo.py". In that scenario there is no opportunity for the > user to specify any args between the name of the executable and of the > script. IOW, the expectation is that people will not type "py foo.py", but > instead just type "foo.py". That sounds like an explanation of why it hasn't been implemented before, not an explanation of why it should continue that way. In any case, I think that expectation is not complete. In my case it was my editor that inserted the '-u' on my behalf. Or why might I not set the default action for .py files to be "py -tt %1", or "py -3 %1". Why deny any of the arguments to a pylauncher user? > * A desire to avoid command-line parsing in the launcher or to make some > options "more equal" then others. Eg, you mention later that -c and -m > should be special, but what about -w or -Q? What about new options in > future versions? Yes it is a bit annoying to have to treat those specially, but other than -c/-m it does not need to understand pythons args, just check that the arg is not an explicit version specifier. -q/-Q etc have no impact on how to treat the file. In fact there's no real need to treat -c differently as it's extremely unlikely that there is a file that might match. But for -m you can come up with a situation where if you it gets it wrong. e.g. 'module' and 'module.py' in the cwd. I would suggest that it is also unlikely that there will be any future options would need any special consideration. >> It is very surprising to the user that adding a >> simple option like -tt should change the way the launcher behaves. >> The PEP also states that the launcher should show the python help if >> '-h' is specified : >> >> "If the only command-line argument is "-h" or "--help", the launcher >> will >> print a small banner and command-line usage, then pass the argument to >> the default Python. This will cause help for the launcher being >> printed >> followed by help for Python itself. The output from the launcher will >> clearly indicate the extended help information is coming from the >> launcher and not Python." >> >> To me that would suggest to end users that they can use any of the >> command line options with the launcher, and they should behave as if >> you had called python directly. > > I think the language is fairly clear - only the help options are special and > no other options will work. The PEP is clear yes, but the the help output for the launcher displays all of the python switches, so I expected them to be available in the a py.exe call. >> Incidentally whilst implementing this I also noticed a bug in the >> pylauncher whereby the py launcher would incorrectly treat "py t3" as >> a request for python version as if '-3' had been specified. I have a >> small patch that fixes this and implements the above for pylauncher >> with extra tests if there is interest. > > That certainly sounds like a bug and a patch sent to > https://bitbucket.org/vinay.sajip/pylauncher will be appreciated! The patch does both the bug fix and the arg skipping at present, but I'll happily separate them if needs be. Sam ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP397 no command line options to python?
On 17 October 2011 17:24, PJ Eby wrote: > What about -S (no site.py) and -E (no environment)? These are needed for > secure setuid scripts on *nix; I don't know how often they'd be used in > practice on Windows. (Basically, they let you isolate a script's effective > sys.path; there may be some use case overlap with virtual envs. At the moment py -S test.py would mean that test.py would not be scanned for a shebang, and would be executed with the latest python. The change that I am suggesting would mean that it would be scanned for a shebang to select the python, and then that python would be called with -S. Do you think it would be necessary to have -S disable reading of the .ini files (in the users application data dir and in \windows)? Sam PS Sorry, I initially replied off-list by accident. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP397 no command line options to python?
On 17 October 2011 15:20, Vinay Sajip wrote: > Sam Partington gmail.com> writes: > >> That sounds like an explanation of why it hasn't been implemented >> before, not an explanation of why it should continue that way. > > From a desire to keep the launcher as simple as possible, and to minimise the > need to synchronise the launcher with command line parameter changes in future > versions of Python. As simple as possible yes... but no simpler. I think having pylauncher behave so differently in the two cases of : py -u test.py py test.py Is very unexpected. And to do so silently, without warning will cause real headaches_ for users, *especially* since py -h lists -u as one of the options, it does not say 'here are the python options but you must call PythonXX/python.exe to use them'. [headaches : it did for me as I ended up with a broken build of my app due to different parts of my app built for different pythons.] To fix this the launcher doesn't need to track all python command line options, only those that take two args. I don't really see that it will be such a maintenance burden to have the launcher track any new ones. Python has only two such options after 20 years of development. As for complexity it's less than 10 lines of C. >> In any case, I think that expectation is not complete. In my case it >> was my editor that inserted the '-u' on my behalf. >> >> Or why might I not set the default action for .py files to be "py -tt >> %1", or "py -3 %1". >> >> Why deny any of the arguments to a pylauncher user? > > Don't forget that customised commands allow Python to be invoked from shebang > lines with fair flexibility. That's a cool feature which I'd not really read up on, but that requires a global configuration file change, it's not doable on a per usage basis. > Don't worry about separating them for now, assuming that it's fairly easy to > figure out which bit is which :-) It wasn't hard to do and I see you've applied the patch already, thanks for the fast turn around! Sam PS Sorry, I replied off-list again! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP397 no command line options to python?
On 18 October 2011 08:10, Vinay Sajip wrote: >> Is very unexpected. And to do so silently, without warning will cause > > It's only unexpected if you don't read the PEP. From there: > > "The launcher may offer some conveniences for Python developers working > interactively - for example, starting the launcher with no command-line > arguments will launch the default Python with no command-line arguments. > Further, command-line arguments will be supported to allow a specific > Python version to be launched interactively - however, these conveniences > must not detract from the primary purpose of launching scripts and must > be easy to avoid if desired." I read the PEP, but didn't spot that subtelty.I wonder how many other people will read the PEP, or just think "Oh, I can just replace python with py" and have it figure out the python to call. >> real headaches_ for users, *especially* since py -h lists -u as one of >> the options, it does not say 'here are the python options but you must >> call PythonXX/python.exe to use them'. > > Well, it's easy enough to make that clearer in the help output of py.exe :-) Indeed, I would say that if nothing else then that should be done >> [headaches : it did for me as I ended up with a broken build of my app >> due to different parts of my app built for different pythons.] > > Why does the need arise to invoke py.exe in a build system? Why not just > reference the Python executable you want directly? That's rather OT here, but briefly as I can. We have transitioned our devel branch to py 2.7. Our stable branches are to remain py 2.6. The build system (also written in python) starts lots of sub build commands, (various SCons, make, bash and python). I added shebangs to all files as appropriate for devel/stable branch, and initially I changed the python build targets from "python -utt build.py" to "./build.py" and I lost the -utt functionality which I could live with. But on some of the windows machines the default action of python files was to open an editor with the build.py. So I changed it to "py -utt build.py". Everything seemed fine initially until tests started to fail which ensued some head scratching. I actually didn't figure out what was going on until I noticed that SCiTe was also calling the wrong python because it also had "py -utt" to run python scripts. Incidentally, one of my colleagues also discovered the same issue in his eclipse pydev setup. I also notice that Editra also does "python -u" by default, and I can imagine lots of users swapping "python" with "py". I am well aware that is is by no means a perfect system and I am working at making it more bulletproof, but as usual there are time constraints etc. We will also go through the whole thing again when wxPython supports python 3. Hopefully I will have solved all these issues by then :-) >> To fix this the launcher doesn't need to track all python command line >> options, only those that take two args. I don't really see that it >> will be such a maintenance burden to have the launcher track any new >> ones. Python has only two such options after 20 years of development. >> >> As for complexity it's less than 10 lines of C. > > Plus tests, presumably ... let's see 'em :-) > >> That's a cool feature which I'd not really read up on, but that >> requires a global configuration file change, it's not doable on a per >> usage basis. > > Per usage = interactively, which is more of a "by-the-by" feature for the > launcher, the main purpose being to bring shebang-line functionality to > Windows. Fair enough. I can see that I am asking more of pylauncher than the unix shebang parser does. But it seems to so nearly do it correctly that I was surprised that it didn't do what I had assumed it did do. I find this usage of it so useful in fact that irrespective of whether the PEP takes on my suggestions I will be using the patched one, and I will be writing a unix pylauncher to do the same. Would it not be an idea to have new installations of python actually install pylauncher as 'python' which then forwards onto the correct 'pythonX.X'. It would possibly help resolve the whole question of "Does python invoke python2 or python3" issue. The patch should be attached. It is of course 20% C and 80% python tests :-) Sam add_arg_skipping.patch Description: Binary data ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP397 no command line options to python?
On 19 October 2011 00:18, Mark Hammond wrote: > On 18/10/2011 8:59 PM, Sam Partington wrote: ... >> I added shebangs to >> all files as appropriate for devel/stable branch, and initially I >> changed the python build targets from "python -utt build.py" to >> "./build.py" and I lost the -utt functionality which I could live >> with. > > Can't you just put the -utt options in the shebang line of build.py? Yes I can but I didn't. There are many ways to skin this cat, but I chose what seemed the most straightforward way. It went wrong and I didn't expect it to. Adding -tt to the shebang line makes sense, the use of -tt depends on the content of the script (although it would require me to add -tt to thousands of shebang lines). But the addition of '-u' depends on the environment in which I want to run it. i.e. when run from a console it's unnecessary and a performance penalty, whilst when run as part of a sub-process (either from an editor or as part of a larger collection of scripts) then it's nice to avoid having the output arrive in lumps. >> But on some of the windows machines the default action of python >> files was to open an editor with the build.py. So I changed it to "py >> -utt build.py". Everything seemed fine initially until tests started >> to fail which ensued some head scratching. I actually didn't figure >> out what was going on until I noticed that SCiTe was also calling the >> wrong python because it also had "py -utt" to run python scripts. >> Incidentally, one of my colleagues also discovered the same issue in >> his eclipse pydev setup. I also notice that Editra also does "python >> -u" by default, and I can imagine lots of users swapping "python" with >> "py". > > Why would users choose to do that? Using "python" presumably already works > for them, so what benefit do they get? If the main advantage is they can > now use shebang lines, then the specific options the script wants can be > expressed in that line. But "python" does NOT work. If you have multiple pythons installed and scripts that are designed to run with various versions then 'python' will use the wrong one, I had thought that this was precisely what pylauncher was meant to solve, i.e. the ability to select the correct python version based on the contents of that file. Nearly all python editors have the ability to run the current script with python. None of the editors that I know of have the ability (out of the box) to parse the shebang line and dispatch to the correct python. There is however already a tool that can do that, called pylauncher. Swap "python" with "py" and it would work in all editors. The only problem is that it does not support unbuffered output which nearly all editors will need in order to display timely output. If you don't want to support all args, how about just supporting the -u argument, which is the one I really need? My issue is currently to select the right python from 2.4, 2.5, 2.6, 2.7. But the need for a tool like this is even stronger when encouraging users to move to py 3 - on all platforms. In the absence of .py3 file extensions the shebang seems the way to go. "Why would users choose to do that?" : Because they are in a project using mixed python versions and they want an easy way to use the correct python from within their editor. > I wonder if we just need to make it clear that py.exe is not designed to > simply be a replacement for python.exe - a simple replacement adds no value. > It is designed to bring shebang processing to Python on Windows and the > shebang line is where these args should live. If you want finer control > over things, just keep using python.exe. There I have to disagree. Yes, a simple replacement would not add value. But a replacement that detects the correct python but otherwise works like python adds lots of value and what is more it has value on all platforms. Shouldn't that argument have applied also for the version specifier args, because to my mind that is a less elegant feature with more baggage. (oddity of "py -2 -3" for example). "If you want finer control just keep using c:\PythonXX\python.exe". Incidentally it would be really really nice if the windows installer installed "pythonX[.Y].exe" then we could just put all the python dirs in PATH and it would just work. no need for the -X.Y flags at all. > Also, seeing it is much easier to add a feature later than to remove it, we > should err on the side of not adding the feature until it is clear that many > people want it and ensure we aren't creating other inconsistencies or issues > when we do. If it turns out to be true th
Re: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()?
On 30 March 2012 21:52, Guido van Rossum wrote: > Oh dear. I really want to say that 15 ms is good enough. Some possible > exceptions I can think of: > > - Profiling. But this really wants to measure CPU time anyways, and it > already uses a variety of hacks and heuristics to pick the best timer, > so I don't really care. That depends on what you're profiling. If you're profiling CPU bound algorithms then yes CPU time is better. But if your trying to profile/measure hardware device/comms performance for example then CPU time is of no interest. There, on windows the 15ms resolution of time.time makes it useless. For some reason I always forget this and sit looking at trace outs for 5 minutes wondering why everything takes either 0, 15, or 30ms. For nearly all my use cases I'm not terribly interested in monotonicity, or stability in suspend/resume states so I won't give my opinions on thiose (though I can see they are good things and can well imagine needing them one day), I just want an easy way of getting at least micro second resolution cross platform. I don't mind particularly what you call it but FWIW 'highres' seems a bit odd to me. It seems that highres is likely to seem lowres one day, and then you need to add higherres() and then evenhigherthanthatres(). I would go with microtime(), or nanotime() it doesn't make any promises about anything other than the resolution. Sam ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Highlighting reference-stealing APIs [was Re: cpython: Fix email post-commit review comments.]
On 19 April 2012 02:20, Nick Coghlan wrote:
> On Thu, Apr 19, 2012 at 12:21 AM, Antoine Pitrou wrote:
>> (and here we see why reference-stealing APIs are a nuisance: because
>> you never know in advance whether a function will steal a reference or
>> not, and you have to read the docs for each and every C API call you
>> make)
>
> Yeah, it would have been nice if there was an explicit hint in the API
> names when reference stealing was involved, but I guess it's far too
> late now :(
It's too late to change the fn names sure, but you could change the
argument names in question for reference stealing apis with some kind
of markup.
That would make it fairly easy to write a script that did the checking for you :
int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *stolen_o)
Or better yet would be to mark the types :
int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyStolenObject* o)
PyBorrowedObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
PyStolenObject and PyBorrowedObject would just be typedefs to PyObject
normally. But a consenting user could define PyENABLE_CHECKED_REFS
before including Python.h which would given
#if defined(PyENABLE_CHECKED_STOLEN_REFS)
struct PyStolenObject;
struct PyBorrowedObject;
#define PyYesIKnowItsStolen(o) ((PyStolenObject*)o)
#define PyYesIKnowItsBorrowed(o) ((PyObject*)o)
#else
typedef PyStolenObject PyObject;
typedef PyBorrowedObject PyObject;
#endif
Forcing the user to use
PyTuple_SetItem(p, pos, PyYesIKnowItsStolen(o))
PyObject * ref = PyYesIKnowItsBorrowed(PyTuple_GetItem(p, pos));
Or else it would fail to compile. The user could even add her own :
PyStolenObject* IncRefBecauseIKnowItsStolen(PyObject* o) {
PyIncRef(o); return (PyStolenObject*)o; }
PyObject* IncRefBecauseIKnowItsBorrowed(PyBorrowedObject* o) {
PyIncRef(o); return (PyObject*)o; }
This would not require a special gcc build and would be available to
anyone who wanted it. I use a similar, C++ based trick in my python
extension code to avoid the whole issue of ref leaking, but I have to
be careful at the point of calling the python api, having it automatic
would be great.
On a similar note, I have just implemented a wrapper around Python.h
which runtime checks that the GIL is held around every call to the
Python API or else fails very noisily. This was done because it turns
out that wxPython had a ton of non-GIL calls to the API causing random
sporadic set faults in our app. We now use it on several of our
extensions. It doesn't require any changes to Python.h, you just
need to add an include path before the python include path. Would
there be any interest in this?
Sam
On 19 April 2012 02:25, Nick Coghlan wrote:
> On Thu, Apr 19, 2012 at 11:04 AM, Gregory P. Smith wrote:
>> +1 Adding these annotations and setting up a buildbot that builds using
>> cpychecker would be a great.
>
> Even without the extra annotations, running cpychecker on at least one
> of the buildbots might be helpful.
>
> I'm in the process of setting up a buildbot for RHEL 6, once I get it
> up and running normally, I'll look into what it would take to install
> and enable cpychecker for the builds. (Or, alternatively, I may make
> it a separate cron job, similar to the daily refcount leak detection
> run).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan | [email protected] | Brisbane, Australia
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/sam.partington%40gmail.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.2): #14957: clarify splitlines docs.
> On Jun 2, 2012 6:21 AM, "r.david.murray" wrote: >> + For example, ``'ab c\n\nde fg\rkl\r\n'.splitlines()`` returns >> + ``['ab c', '', 'de fg', 'kl']``, while the same call with >> ``splinelines(True)`` >> + returns ``['ab c\n', '\n, 'de fg\r', 'kl\r\n']`` Wouldn't that be better written as a doctest and so avoid any other typos? Sam ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment"
On 7 November 2012 13:57, Ulrich Eckhardt wrote: > Am 31.10.2012 23:15, schrieb Steven D'Aprano: > I'll take a shot, since I was also bitten by this when trying to learn > Python. The important point is that some code earlier or later in that > function does an assignment, so this location should be referenced in the > error. +1 > How about: > > "UnboundLocalError: Local variable 'FONT_NAMES' (created on >line 11) referenced before assignment." > > What I don't really like is the term "created". Maybe "implicitly created on > line 11"? Or "implied by line 11"? Or how about "Local variable FONT_NAMES > (implied by line 11) doesn't refer to an object", to avoid the multiple > interpretations of the term "assignment"? Why not make use of the well defined words already there in the error message, but at the line number "UnboundLocalError: Local variable 'FONT_NAMES' referenced before it has been assigned to on line 11." Sam ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
