Re: [Python-Dev] PEP 405 (pyvenv) and system Python upgrades
On Sat, May 5, 2012 at 6:49 AM, Carl Meyer wrote: > 1) Document it and provide a tool for easily upgrading a venv in this > situation. This may be adequate. In practice the situation is quite rare: > 2.6.8/2.7.3 is the only actual example in the history of virtualenv that I'm > aware of. The disadvantage is that if the problem does occur, the error will > probably be quite confusing and seemingly unrelated to pyvenv. I think this is the way to go, for basically the same reasons that we did it this way this time: there's no good reason to pay an ongoing cost to further mitigate the risks associated with an already incredibly rare event. It would become part of the standard venv debugging toolkit: Q X.1: Does the problem only occur inside a particular virtual environment? Q X.2: If yes, did you recently upgrade the system Python to a new point release? Q X.3: If yes, did you run ? Q X.4: If no, do so and see if the problem goes away. Even if it still doesn't work, at least you've eliminated this particular error as a possible cause. Personally, I expect that "always update your virtual environment binaries after updating the system Python to a new point release" will itself become a recommended practice when using virtual environments. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 405 (pyvenv) and system Python upgrades
On 5/4/2012 9:39 PM, Terry Reedy wrote: The gist of my response is that the venv 'tail' should way the python 'dog' as little as possbile. Yes, that was exactly my thought too. But I'm not sure the technology permits, with Windows not having exec. On the other hand, one might speculate about how venv, instead of copying Python.exe, might instead install the launcher in the place where python.exe is currently copied. The launcher does the "next best thing to exec". Plus it would save a wee bit of space, being smaller than python.exe. On platforms that have symlinks, they could be used instead. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 405 (pyvenv) and system Python upgrades
On 5/4/2012 11:58 PM, Glenn Linderman wrote: On 5/4/2012 8:48 PM, Terry Reedy wrote: CPython is developed, tested, packaged, distributed, and installed as one unit. It is intended to be run as one package. If something caches a copy of python.exe, it seems to me that it should check and update as needed. Could venv check the file date of the current python.exe versus that of the one cached, much like is done with .pyc compiled code caches? I almost wrote this response (using different words, but the same idea) but concluded that: 1) Python wouldn't run far without its standard library, so a venv check would have to be very early, and likely coded in C, and therefore probably has to be part of Python.exe 2) If it was not part of Python.exe, it would have to work similarly to the launcher, and there would be yet one more process sitting around waiting for Python to exit (on Windows, where there is no exec). So I concluded that probably Python.exe needs to make the check, but if it is aware it existing in venv, it might be able to put out a better message than "just" the mismatch between exe and lib; or at least the message should mention the possibility of an old venv cache. The gist of my response is that the venv 'tail' should way the python 'dog' as little as possbile. I also wonder how often such incompatibility occurs. Optionally changing the de facto semantics of CPython's built-in dict in bug-fix releases was a rather unique event. I am sure we would all be happy to never have to do such again. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 405 (pyvenv) and system Python upgrades
On 5/4/2012 8:48 PM, Terry Reedy wrote: CPython is developed, tested, packaged, distributed, and installed as one unit. It is intended to be run as one package. If something caches a copy of python.exe, it seems to me that it should check and update as needed. Could venv check the file date of the current python.exe versus that of the one cached, much like is done with .pyc compiled code caches? I almost wrote this response (using different words, but the same idea) but concluded that: 1) Python wouldn't run far without its standard library, so a venv check would have to be very early, and likely coded in C, and therefore probably has to be part of Python.exe 2) If it was not part of Python.exe, it would have to work similarly to the launcher, and there would be yet one more process sitting around waiting for Python to exit (on Windows, where there is no exec). So I concluded that probably Python.exe needs to make the check, but if it is aware it existing in venv, it might be able to put out a better message than "just" the mismatch between exe and lib; or at least the message should mention the possibility of an old venv cache. Glenn ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 405 (pyvenv) and system Python upgrades
On 5/4/2012 4:49 PM, Carl Meyer wrote: Hi all, The recent virtualenv breakage in Python 2.6.8 and 2.7.3 reveals an issue that deserves to be explicitly addressed in PEP 405: what happens when the system Python underlying a venv gets an in-place bugfix upgrade. If the bugfix includes a simultaneous change to the interpreter and standard library such that the older interpreter will not work with the newer standard library, all venvs created from that Python installation will be broken until the new interpreter is copied into them. CPython is developed, tested, packaged, distributed, and installed as one unit. It is intended to be run as one package. If something caches a copy of python.exe, it seems to me that it should check and update as needed. Could venv check the file date of the current python.exe versus that of the one cached, much like is done with .pyc compiled code caches? Choices for how to address this: 1) Document it and provide a tool for easily upgrading a venv in this situation. Right. 4) Adopt a policy of interpreter/stdlib cross-compatibility within a given X.Y version of Python. I don't expect this to be a popular choice, What a droll sense of humor ;=). -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debian wheezy, amd64: make not finding files for bz2 and other packages
On Sat, May 5, 2012 at 4:07 AM, Edward C. Jones wrote: > /usr/include/bzlib.h > /usr/lib/x86_64-linux-gnu/libbz2.a > /usr/lib/x86_64-linux-gnu/libbz2.so > /lib/x86_64-linux-gnu/libbz2.so.1 > /lib/x86_64-linux-gnu/libbz2.so.1.0 > /lib/x86_64-linux-gnu/libbz2.so.1.0.4 I have an Ubuntu Maverick 64-bit system, not identical but hopefully similar to your Debian. I have /usr/include/bzlib.h, but the others are all one directory level higher - /usr/lib/libbz2.a, /lib/libbz2.so.1.0.4, etc. Does your /etc/ld.so.conf.d mention the appropriate directories? ChrisA ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 405 (pyvenv) and system Python upgrades
Hi all, The recent virtualenv breakage in Python 2.6.8 and 2.7.3 reveals an issue that deserves to be explicitly addressed in PEP 405: what happens when the system Python underlying a venv gets an in-place bugfix upgrade. If the bugfix includes a simultaneous change to the interpreter and standard library such that the older interpreter will not work with the newer standard library, all venvs created from that Python installation will be broken until the new interpreter is copied into them. Choices for how to address this: 1) Document it and provide a tool for easily upgrading a venv in this situation. This may be adequate. In practice the situation is quite rare: 2.6.8/2.7.3 is the only actual example in the history of virtualenv that I'm aware of. The disadvantage is that if the problem does occur, the error will probably be quite confusing and seemingly unrelated to pyvenv. 2) In addition to the above, introduce a versioning marker in the standard library (is there one already?) and have some code somewhere (insert hand-waving here) check sys.version_info against the stdlib version, and fail fast with an unambiguous error if there is a mismatch. This makes the failure more explicit, but at the significant cost of making it more common: at every mismatch, not just in the apparently-rare case of a breaking change. 3) Symlink the interpreter rather than copying. I include this here for the sake of completeness, but it's already been rejected due to significant problems on older Windows' and OS X. 4) Adopt a policy of interpreter/stdlib cross-compatibility within a given X.Y version of Python. I don't expect this to be a popular choice, given the additional testing requirements it imposes, but it would certainly be the nicest option from the PEP 405 standpoint (and may also be complementary to proposals for splitting out the stdlib). In the 2.6.8/2.7.3 case, this would have been technically trivial to do, but the choice was made not to do it in order to force virtualenv users to adopt the security-fixed Python interpreter. Thoughts? Carl ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debian wheezy, amd64: make not finding files for bz2 and other packages
On Fri, May 04, 2012 at 02:07:28PM -0400, "Edward C. Jones" wrote: > From the Debian website, I got the list of all the > files in these three packages: Don't know about amd64 arch, sorry. You can list content of a package from command line: dpkg [-L|--listfiles] libbz2-dev Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phdru.name Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Another buildslave - Ubuntu again
On Fri, 4 May 2012 13:21:17 +0800 Senthil Kumaran wrote: > On Thu, May 3, 2012 at 12:46 AM, Antoine Pitrou wrote: > > Daily code coverage builds would be nice, but that's probably beyond > > what the current infrastructure can offer. It would be nice if someone > > wants to investigate that. > > Code coverage buildbots would indeed be good. I could give a try on > this. What kind of infra changes would be required? I presume, it is > the server side that you are referring to. It doesn't *need* to be a buildbot. Just have a cron script somewhere to run coverage on the test suite every day and published the results somewhere in a readable format. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Debian wheezy, amd64: make not finding files for bz2 and other packages
I use up-to-date Debian testing (wheezy), amd64 architecture. I have made a "clone" of the developmental version of Python 3.3. "make -s -j3" prints: ... Python build finished, but the necessary bits to build these modules were not found: _bz2 _curses_curses_panel _dbm _gdbm _lzma _sqlite3 _ssl readline zlib To find the necessary bits, look in setup.py in detect_modules() for the module's name. Failed to build these modules: _crypt nis [101752 refs] I looked into bz2. My system already contained the Debian packages libbz2-dev, libbz2-1.0, and bzip2. From the Debian website, I got the list of all the files in these three packages: Filelist of package libbz2-dev in wheezy of architecture amd64 /usr/include/bzlib.h /usr/lib/x86_64-linux-gnu/libbz2.a /usr/lib/x86_64-linux-gnu/libbz2.so /usr/share/doc/libbz2-dev Filelist of package libbz2-1.0 in wheezy of architecture amd64 /lib/x86_64-linux-gnu/libbz2.so.1 /lib/x86_64-linux-gnu/libbz2.so.1.0 /lib/x86_64-linux-gnu/libbz2.so.1.0.4 /usr/share/doc/libbz2-1.0/changelog.Debian.gz /usr/share/doc/libbz2-1.0/changelog.gz /usr/share/doc/libbz2-1.0/copyright Filelist of package bzip2 in wheezy of architecture amd64 /bin/bunzip2 /bin/bzcat /bin/bzcmp /bin/bzdiff /bin/bzegrep /bin/bzexe /bin/bzfgrep /bin/bzgrep /bin/bzip2 /bin/bzip2recover /bin/bzless /bin/bzmore /usr/share/doc/bzip2/changelog.Debian.gz /usr/share/doc/bzip2/changelog.gz /usr/share/doc/bzip2/copyright /usr/share/man/man1/bunzip2.1.gz /usr/share/man/man1/bzcat.1.gz /usr/share/man/man1/bzcmp.1.gz /usr/share/man/man1/bzdiff.1.gz /usr/share/man/man1/bzegrep.1.gz /usr/share/man/man1/bzexe.1.gz /usr/share/man/man1/bzfgrep.1.gz /usr/share/man/man1/bzgrep.1.gz /usr/share/man/man1/bzip2.1.gz /usr/share/man/man1/bzip2recover.1.gz /usr/share/man/man1/bzless.1.gz /usr/share/man/man1/bzmore.1.gz What is the problem? Does wheezy amd64 put files in unusual places? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Summary of Python tracker Issues
On Fri, May 4, 2012 at 10:07 AM, Python tracker wrote: > > ACTIVITY SUMMARY (2012-04-27 - 2012-05-04) > Python tracker at http://bugs.python.org/ > > To view or respond to any of the issues listed below, click on the issue. > Do NOT respond to this message. > > Issues counts and deltas: > open 3399 ( -6) > closed 23101 (+45) > total 26500 (+39) Negative delta for open, FTW! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Does trunk still support any compilers that *don't* allow declaring variables after code?
I don't have the time to figure it out right now, but I'll look more into it later. I recently did an analysis here: http://mail.python.org/pipermail/python-dev/2012-January/115375.html The motivation for C++ compilation is gone meanwhile, as VS now supports C in WinRT apps quite well. However, the conclusions still stand: dealing with static type objects will be tricky. Of course, I would also like to eliminate static type objects as much as possible. This then leaves the issue with the casts, which might be considered clutter. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2012-04-27 - 2012-05-04) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3399 ( -6) closed 23101 (+45) total 26500 (+39) Open issues with patches: 1452 Issues opened (26) == #13183: pdb skips frames after hitting a breakpoint and running step http://bugs.python.org/issue13183 reopened by loewis #14428: Implementation of the PEP 418 http://bugs.python.org/issue14428 reopened by neologix #14684: zlib set dictionary support inflateSetDictionary http://bugs.python.org/issue14684 opened by Sam.Rushing #14689: make PYTHONWARNINGS variable work in libpython http://bugs.python.org/issue14689 opened by petere #14690: Use monotonic time for sched, trace and subprocess modules http://bugs.python.org/issue14690 opened by haypo #14692: json.joads parse_constant callback not working anymore http://bugs.python.org/issue14692 opened by Jakob.Simon-Gaarde #14693: hashlib fallback modules should be built even if openssl *is* http://bugs.python.org/issue14693 opened by dov #14695: Tools/parser/unparse.py is out of date. http://bugs.python.org/issue14695 opened by mark.dickinson #14697: parser module doesn't support set displays or set comprehensio http://bugs.python.org/issue14697 opened by mark.dickinson #14698: test_posix failures - getpwduid()/initgroups()/getgroups() http://bugs.python.org/issue14698 opened by neologix #14700: Integer overflow in classic string formatting http://bugs.python.org/issue14700 opened by storchaka #14701: parser module doesn't support 'raise ... from' http://bugs.python.org/issue14701 opened by mark.dickinson #14702: os.makedirs breaks under autofs directories http://bugs.python.org/issue14702 opened by amcnabb #14703: Update PEP metaprocesses to describe PEP czar role http://bugs.python.org/issue14703 opened by ncoghlan #14705: Add 'bool' format character to PyArg_ParseTuple* http://bugs.python.org/issue14705 opened by larry #14709: http.client fails sending read()able Object http://bugs.python.org/issue14709 opened by Tobias.Steinrücken #14710: pkgutil.get_loader is broken http://bugs.python.org/issue14710 opened by Pavel.Aslanov #14711: Remove os.stat_float_times http://bugs.python.org/issue14711 opened by aronacher #14712: Integrate PEP 405 http://bugs.python.org/issue14712 opened by vinay.sajip #14713: PEP 414 installation hook fails with an AssertionError http://bugs.python.org/issue14713 opened by vinay.sajip #14714: PEp 414 tokenizing hook does not preserve tabs http://bugs.python.org/issue14714 opened by vinay.sajip #14715: test.support.DirsOnSysPath should be replaced by importlib.tes http://bugs.python.org/issue14715 opened by eric.smith #14716: Use unicode_writer API for str.format() http://bugs.python.org/issue14716 opened by haypo #14720: sqlite3 microseconds http://bugs.python.org/issue14720 opened by frankmillman #14721: httplib doesn't specify content-length header for POST request http://bugs.python.org/issue14721 opened by Arve.Knudsen #14722: Overflow in parsing 'float' parameters in PyArg_ParseTuple* http://bugs.python.org/issue14722 opened by storchaka Most recent 15 issues with no replies (15) == #14714: PEp 414 tokenizing hook does not preserve tabs http://bugs.python.org/issue14714 #14713: PEP 414 installation hook fails with an AssertionError http://bugs.python.org/issue14713 #14712: Integrate PEP 405 http://bugs.python.org/issue14712 #14709: http.client fails sending read()able Object http://bugs.python.org/issue14709 #14703: Update PEP metaprocesses to describe PEP czar role http://bugs.python.org/issue14703 #14695: Tools/parser/unparse.py is out of date. http://bugs.python.org/issue14695 #14689: make PYTHONWARNINGS variable work in libpython http://bugs.python.org/issue14689 #14680: pydoc with -w option does not work for a lot of help topics http://bugs.python.org/issue14680 #14679: Changes to html.parser break third-party code http://bugs.python.org/issue14679 #14674: Add link to RFC 4627 from json documentation http://bugs.python.org/issue14674 #14652: Better error messages for wsgiref validator failures http://bugs.python.org/issue14652 #14649: doctest.DocTestSuite error misleading when module has no docst http://bugs.python.org/issue14649 #14645: Generator does not translate linesep characters in certain cir http://bugs.python.org/issue14645 #14616: subprocess docs should mention pipes.quote/shlex.quote http://bugs.python.org/issue14616 #14584: Add gzip support to xmlrpc.server http://bugs.python.org/issue14584 Most recent 15 issues waiting for review (15) = #14722: Overflow in parsing 'float' parameters in PyArg_ParseTuple* http://bugs.python.org/issue14722 #14716: Use unicode_writer API for str.format() http://bugs.python.
Re: [Python-Dev] Python program name
Antoine Pitrou pitrou.net> writes: > If that's the only failing test, we can simply skip it when run from a > venv. A non-existent argv[0] is arguably a borderline case which you > should only encounter when e.g. embedding Python. Actually there are four module failures: test_sys, test_packaging, test_distutils and test_subprocess. I haven't looked into all of them yet, but many of the failure messages were "unable to get the locale encoding". > getpath.c is sufficiently byzantine that we don't want to complexify it > too much, IMHO. Right, but the change is unlikely to add significantly to complexity. It would be one static function e.g. named get_executable_path and one call to it, conditional on !isfile(argv[0]), in calculate_path. That would be in two places - Modules/getpath.c and PC/getpathp.c. I'll skip that test_sys test for now, and see where the other failures lead me to. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Does trunk still support any compilers that *don't* allow declaring variables after code?
On Fri, May 4, 2012 at 10:08 AM, Dirkjan Ochtman wrote: > On Wed, May 2, 2012 at 10:43 AM, Larry Hastings wrote: >> Do we officially support any C compilers that *don't* permit "intermingled >> variable declarations and code"? Do we *unofficially* support any? And if >> we do, what do we gain? > > This might be of interest: > > http://herbsutter.com/2012/05/03/reader-qa-what-about-vc-and-c99/?nope > > Specifically, apparently MSVC 2010 supports variable declarations in > the middle of a block in C. > > Also, since full C99 support won't be coming to MSVC, perhaps Python > should move to compiling in C++ mode? After seeing that same article yesterday and having the VS2010 port open, I tried this and it appears it won't work without significant code changes at least as far as I saw. I enabled /TP on the pythoncore project and got over 1363 errors. I don't have the time to figure it out right now, but I'll look more into it later. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Does trunk still support any compilers that *don't* allow declaring variables after code?
On Wed, May 2, 2012 at 10:43 AM, Larry Hastings wrote: > Do we officially support any C compilers that *don't* permit "intermingled > variable declarations and code"? Do we *unofficially* support any? And if > we do, what do we gain? This might be of interest: http://herbsutter.com/2012/05/03/reader-qa-what-about-vc-and-c99/?nope Specifically, apparently MSVC 2010 supports variable declarations in the middle of a block in C. Also, since full C99 support won't be coming to MSVC, perhaps Python should move to compiling in C++ mode? Cheers, Dirkjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: avoid unitialized memory
2012/5/4 Eric V. Smith :
> On 5/4/2012 1:14 AM, benjamin.peterson wrote:
>> http://hg.python.org/cpython/rev/b0deafca6c02
>> changeset: 76743:b0deafca6c02
>> user: Benjamin Peterson
>> date: Fri May 04 01:14:03 2012 -0400
>> summary:
>> avoid unitialized memory
>>
>> files:
>> Modules/posixmodule.c | 2 +-
>> 1 files changed, 1 insertions(+), 1 deletions(-)
>>
>>
>> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
>> --- a/Modules/posixmodule.c
>> +++ b/Modules/posixmodule.c
>> @@ -3576,7 +3576,7 @@
>> split_py_long_to_s_and_ns(PyObject *py_long, time_t *s, long *ns)
>> {
>> int result = 0;
>> - PyObject *divmod;
>> + PyObject *divmod = NULL;
>> divmod = PyNumber_Divmod(py_long, billion);
>
> How is that uninitialized if it's being set on the next line?
It was a misreading on my part.
--
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #14127: Add ns= parameter to utime, futimes, and lutimes.
2012/5/4 Larry Hastings :
> + if (times && (times != Py_None)) {
>
> Conditions in parenthesis like this is not style.
>
>
> Can you point me to where this is described in PEP 7? I can't find it.
It's not explicitly stated, but there is the following nice example:
if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
type->tp_dictoffset == b_size &&
(size_t)t_size == b_size + sizeof(PyObject *))
return 0; /* "Forgive" adding a __dict__ only */
There's also the consistency with surrounding code imperative.
--
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 : Allow __enter__() methods to skip the statement body : real world case
On Fri, May 4, 2012 at 10:17 PM, Pierre Chanial wrote: > Hello, > > PEP 377 has been rejected for lack of use cases. Here is one. > > I'm writing an MPI-based application and in some cases, when there is less > work items than processes, I need to create a new communicator excluding the > processes that have nothing to do. This new communicator should finally be > freed by the processes that had work to do (and only by them). If there is > more work than processes, the usual communicator should be used. > > A neat way to do that would be to write: > > with filter_comm(comm, nworkitems) as newcomm: > > ... do work with communicator newcomm... > > the body being executed only by the processes that have work to do. > > It looks better than: > > if comm.size < nworkitems: > newcomm = get_new_communicator(comm, nworkitems) > > else: > newcomm = comm > > if comm.rank < nworkitems: > try: > ... do work with communicator newcomm... > finally: > if comm.size < nworkitems: > newcomm.Free() > > Especially since I have to use that quite often. However, your original code is not substantially better than: with filter_comm(comm, nworkitems) as newcomm: if newcomm is not None: ... do work with communicator newcomm... Where filtercomm is a context manager that: - returns None from __enter__ if this process has no work to do - cleans up in __exit__ if a new communicator was allocated in __enter__ It isn't that there are no use cases for skipping the statement body: it's that the extra machinery needed to allow a context manager to do so implicitly is quite intrusive, and the control flow is substantially clearer to the reader of the code if the context manager is instead paired with an appropriate nested if statement. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python program name
On Fri, 4 May 2012 08:44:25 + (UTC) Vinay Sajip wrote: > IIUC, the program name of the Python executable is set to whatever argv[0] is. > Is there a reason for this, rather than using one of the various OS-specific > APIs [1] for getting the name of the running executable? The reason I ask is > that in a virtual environment (venv), the exe's path is the only thing you > have > to go on, and if you don't have that, you can't find the pyvenv.cfg file and > hence the base Python from which the venv was created. > > Of course argv[0] is normally set to the executable's path, but there's at > least > one test (in test_sys) where Python is spawned (via subprocess) with argv[0] > set > to "nonexistent". If run from a venv created from a source build, with no > Python > 3.3 installed, this test fails because the spawned Python can't locate the > locale encoding, and bails. If that's the only failing test, we can simply skip it when run from a venv. A non-existent argv[0] is arguably a borderline case which you should only encounter when e.g. embedding Python. > I would prefer to use option 2 and change getpath.c / getpathp.c accordingly. > Does anyone here see problems with that approach? getpath.c is sufficiently byzantine that we don't want to complexify it too much, IMHO. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python program name
On Fri, 4 May 2012 13:29:14 +0100 Michael Foord wrote: > > On 4 May 2012, at 09:44, Vinay Sajip wrote: > > > IIUC, the program name of the Python executable is set to whatever argv[0] > > is. > > Is there a reason for this, rather than using one of the various OS-specific > > APIs [1] for getting the name of the running executable? The reason I ask is > > that in a virtual environment (venv), the exe's path is the only thing you > > have > > to go on, and if you don't have that, you can't find the pyvenv.cfg file and > > hence the base Python from which the venv was created. > > > argv[0] is the *script* name, not the executable name - surely? > > The executable path is normally available in sys.executable. I think Vinay is talking about C argv, not sys.argv. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: avoid unitialized memory
Eric V. Smith writes: > > -PyObject *divmod; > > +PyObject *divmod = NULL; > > divmod = PyNumber_Divmod(py_long, billion); > > How is that uninitialized if it's being set on the next line? Maybe they finally developed a Sufficiently Stupid Compiler? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python program name
On 4 May 2012, at 09:44, Vinay Sajip wrote:
> IIUC, the program name of the Python executable is set to whatever argv[0] is.
> Is there a reason for this, rather than using one of the various OS-specific
> APIs [1] for getting the name of the running executable? The reason I ask is
> that in a virtual environment (venv), the exe's path is the only thing you
> have
> to go on, and if you don't have that, you can't find the pyvenv.cfg file and
> hence the base Python from which the venv was created.
argv[0] is the *script* name, not the executable name - surely?
The executable path is normally available in sys.executable.
Michael
>
> Of course argv[0] is normally set to the executable's path, but there's at
> least
> one test (in test_sys) where Python is spawned (via subprocess) with argv[0]
> set
> to "nonexistent". If run from a venv created from a source build, with no
> Python
> 3.3 installed, this test fails because the spawned Python can't locate the
> locale encoding, and bails.
>
> It works when run from a source build ("./python ...") because the getpath.c
> code to find a prefix looks in the directory implied by argv[0] (in the case
> of
> "nonexistent" => "", i.e. the current directory) for "Modules/Setup", and also
> works from a venv if created from an installed Python 3.3 (since the value of
> sys.prefix is used as a fallback check, and that value will contain that
> Python). However, when run from a venv created from a source build, with no
> Python 3.3 installed, the error occurs.
>
> A workaround might be one of these:
>
> 1. Use an OS-specific API rather than argv[0] to get the executable's path for
> the processing done by getpath.c in all cases, or
>
> 2. If the file named by argv[0] doesn't exist, then use the OS-specific API to
> find the executable's path, and try with that, or
>
> 3. If using the current logic, no prefix is found, then use the OS-specific
> API
> to to find the executable's path, and try with that.
>
> I would prefer to use option 2 and change getpath.c / getpathp.c accordingly.
> Does anyone here see problems with that approach?
>
> Regards,
>
> Vinay Sajip
>
> [1] http://stackoverflow.com/a/933996
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
--
http://www.voidspace.org.uk/
May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing
http://www.sqlite.org/different.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 377 : Allow __enter__() methods to skip the statement body : real world case
Hello, PEP 377 has been rejected for lack of use cases. Here is one. I'm writing an MPI-based application and in some cases, when there is less work items than processes, I need to create a new communicator excluding the processes that have nothing to do. This new communicator should finally be freed by the processes that had work to do (and only by them). If there is more work than processes, the usual communicator should be used. A neat way to do that would be to write: with filter_comm(comm, nworkitems) as newcomm: ... do work with communicator newcomm... the body being executed only by the processes that have work to do. It looks better than: if comm.size < nworkitems: newcomm = get_new_communicator(comm, nworkitems) else: newcomm = comm if comm.rank < nworkitems: try: ... do work with communicator newcomm... finally: if comm.size < nworkitems: newcomm.Free() Especially since I have to use that quite often. Cheers, Pierre ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #14127: Fix two bugs with the Windows implementation.
On 05/04/2012 11:32 AM, larry.hastings wrote: > http://hg.python.org/cpython/rev/fc5d2f4291ac > changeset: 76747:fc5d2f4291ac > user:Larry Hastings > date:Fri May 04 02:31:57 2012 -0700 > summary: > Issue #14127: Fix two bugs with the Windows implementation. Would be nice to mention what these bugs were, otherwise the commit message is not very helpful when doing e.g. bisect or annotate. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #14127: Add ns= parameter to utime, futimes, and lutimes.
On Fri, 4 May 2012 01:07:04 -0400
Benjamin Peterson wrote:
> > + if (times && (times != Py_None)) {
>
> Conditions in parenthesis like this is not style.
If it's not style, then what is it? :-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #14127: Add ns= parameter to utime, futimes, and lutimes.
Larry Hastings wrote:
On 05/03/2012 10:07 PM, Benjamin Peterson wrote:
+if (times && ns) {
+PyErr_Format(PyExc_RuntimeError,
Why not a ValueError or TypeError?
Well it's certainly not a TypeError.
TypeError is not just for values of the wrong type,
it's also used for passing the wrong number of arguments
to a function and the like. So TypeError would be a
reasonable choice here, I think.
--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: what is a invalid tuple?
benjamin.peterson wrote:
> files:
> Modules/posixmodule.c | 4 ++--
> 1 files changed, 2 insertions(+), 2 deletions(-)
>
>
> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
> --- a/Modules/posixmodule.c
> +++ b/Modules/posixmodule.c
> @@ -3702,7 +3702,7 @@
> if (!PyTuple_CheckExact(times) || (PyTuple_Size(times) != 2)) {
> PyErr_Format(PyExc_TypeError,
> "%s: 'time' must be either"
> - " a valid tuple of two ints or None",
> + " a tuple of two ints or None",
Unrelated to this commit, but 'time' should be 'times'.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #14127: Add ns= parameter to utime, futimes, and lutimes.
Larry Hastings wrote:
> On 05/03/2012 10:07 PM, Benjamin Peterson wrote:
>
> +if (times && ns) {
> +PyErr_Format(PyExc_RuntimeError,
>
> Why not a ValueError or TypeError?
>
>
> Well it's certainly not a TypeError. The 3.2 documentation defines TypeError
> as:
>
> Raised when an operation or function is applied to an object of
> inappropriate type. The associated value is a string giving details about
> the type mismatch.
>
> If someone called os.utime with both times and ns, and the values of each
> would
> have been legal if they'd been passed in in isolation, what would be the type
> mismatch?
I had the same question a while ago, and IIRC Raymond said that the convention
is to raise a TypeError if a combination of arguments cannot be handled by a
function.
In OCaml this would be quite natural:
$ ocaml
Objective Caml version 3.12.0
# type kwargs = TIMES | NS;;
type kwargs = TIMES | NS
let utime args =
match args with
| (_, TIMES) -> "Got times"
| (_, NS) -> "Got NS";;
val utime : 'a * kwargs -> string =
# utime ("/etc/passwd", TIMES);;
- : string = "Got times"
# utime ("/etc/passwd", NS);;
- : string = "Got NS"
# utime ("/etc/passwd", TIMES, NS);;
Error: This expression has type string * kwargs * kwargs
but an expression was expected of type 'a * kwargs
In Python it makes sense if (for the purpose of raising an error) one assumes
that {"times":(0, 0)}, {"ns":(0, 0)} and {"times":(0, 0), "ns":(0, 0)} have
different types.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: avoid unitialized memory
benjamin.peterson wrote:
> summary:
> avoid unitialized memory
>
> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
> --- a/Modules/posixmodule.c
> +++ b/Modules/posixmodule.c
> @@ -3576,7 +3576,7 @@
> split_py_long_to_s_and_ns(PyObject *py_long, time_t *s, long *ns)
> {
> int result = 0;
> -PyObject *divmod;
> +PyObject *divmod = NULL;
> divmod = PyNumber_Divmod(py_long, billion);
> if (!divmod)
> goto exit;
If I'm not mistaken, divmod was already unconditionally initialized
by PyNumber_Divmod().
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optimize Unicode strings in Python 3.3
04.05.12 02:45, Victor Stinner написав(ла): * Two steps: compute the length and maximum character of the output string, allocate the output string and then write characters. str%args was using it. * Optimistic approach. Start with a ASCII buffer, enlarge and widen (to UCS2 and then UCS4) the buffer when new characters are written. Approach used by the UTF-8 decoder and by str%args since today. In real today UTF-8 decoder uses two-steps approach. Only after encountering an error it switches to optimistic approach. The optimistic approach uses realloc() to resize the string. It is faster than the PyAccu approach (at least for short ASCII strings), maybe because it avoids the creating of temporary short strings. realloc() looks to be efficient on Linux and Windows (at least Seven). IMHO, realloc() has no relationship to this. The case in the cost of managing of the list and creating of temporary strings. Various notes: * PyUnicode_READ() is slower than reading a Py_UNICODE array. And PyUnicode_WRITE() is slower than writing a Py_UNICODE/PyUCS* array. * Some decoders unroll the main loop to process 4 or 8 bytes (32 or 64 bits CPU) at each step. Note, this is not only CPU-, but OS-depending (LP64 vs LLP64). I am interested if you know other tricks to optimize Unicode strings in Python, or if you are interested to work on this topic. Optimized ASCII decoder (issue 14419) is not only reads 4 or 8 bytes at a time, but writes them all at a time. This is a very specific optimization. More general principle is replacing serial scanning and translating on an one-pass optimistic reading and writing. This improves the efficiency of the memory cache. I'm going to try it in UTF-8 decoder, it will allow to increase the speed of decoding ASCII-only strings up to speed of optimized ASCII decoder. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python program name
IIUC, the program name of the Python executable is set to whatever argv[0] is.
Is there a reason for this, rather than using one of the various OS-specific
APIs [1] for getting the name of the running executable? The reason I ask is
that in a virtual environment (venv), the exe's path is the only thing you have
to go on, and if you don't have that, you can't find the pyvenv.cfg file and
hence the base Python from which the venv was created.
Of course argv[0] is normally set to the executable's path, but there's at least
one test (in test_sys) where Python is spawned (via subprocess) with argv[0] set
to "nonexistent". If run from a venv created from a source build, with no Python
3.3 installed, this test fails because the spawned Python can't locate the
locale encoding, and bails.
It works when run from a source build ("./python ...") because the getpath.c
code to find a prefix looks in the directory implied by argv[0] (in the case of
"nonexistent" => "", i.e. the current directory) for "Modules/Setup", and also
works from a venv if created from an installed Python 3.3 (since the value of
sys.prefix is used as a fallback check, and that value will contain that
Python). However, when run from a venv created from a source build, with no
Python 3.3 installed, the error occurs.
A workaround might be one of these:
1. Use an OS-specific API rather than argv[0] to get the executable's path for
the processing done by getpath.c in all cases, or
2. If the file named by argv[0] doesn't exist, then use the OS-specific API to
find the executable's path, and try with that, or
3. If using the current logic, no prefix is found, then use the OS-specific API
to to find the executable's path, and try with that.
I would prefer to use option 2 and change getpath.c / getpathp.c accordingly.
Does anyone here see problems with that approach?
Regards,
Vinay Sajip
[1] http://stackoverflow.com/a/933996
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: avoid unitialized memory
On 5/4/2012 1:14 AM, benjamin.peterson wrote:
> http://hg.python.org/cpython/rev/b0deafca6c02
> changeset: 76743:b0deafca6c02
> user:Benjamin Peterson
> date:Fri May 04 01:14:03 2012 -0400
> summary:
> avoid unitialized memory
>
> files:
> Modules/posixmodule.c | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
>
> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
> --- a/Modules/posixmodule.c
> +++ b/Modules/posixmodule.c
> @@ -3576,7 +3576,7 @@
> split_py_long_to_s_and_ns(PyObject *py_long, time_t *s, long *ns)
> {
> int result = 0;
> -PyObject *divmod;
> +PyObject *divmod = NULL;
> divmod = PyNumber_Divmod(py_long, billion);
How is that uninitialized if it's being set on the next line?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Another buildslave - Ubuntu again
On 04.05.2012 07:21, Senthil Kumaran wrote: On Thu, May 3, 2012 at 12:46 AM, Antoine Pitrou wrote: Daily code coverage builds would be nice, but that's probably beyond what the current infrastructure can offer. It would be nice if someone wants to investigate that. Code coverage buildbots would indeed be good. I could give a try on this. What kind of infra changes would be required? I presume, it is the server side that you are referring to. I think the setup could be similar to the daily DMG builder. If the slave generates a set of HTML files (say), those can be uploaded just fine. However, we don't have any "make coverage" target currently in the makefile. So if you contribute that, we could then have it run daily. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Another buildslave - Ubuntu again
That page would probably like a good cleanup. I don't even think creating an user is required - it's just good practice, and you probably want that user to have as few privileges as possible. That's indeed the motivation. Buildbot slave operators need to recognize that they are opening their machines to execution of arbitrary code, even though this could only be abused by committers. But suppose a committer loses the laptop, which has his SSH key on it, then anybody getting the key could commit malicious code, which then gets executed by all build slaves. Of course, it would be possible to find out whose key has been used (although *not* from the commit message), and revoke that, but the damage might already be done. Regards, Martin P.S. Another attack vector is through the master: if somebody hacks into the machine running the master, they can also compromise all slaves. Of course, we are trying to make it really hard to break into python.org. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #14127: Add ns= parameter to utime, futimes, and lutimes.
On 05/04/2012 08:04 AM, Larry Hastings wrote:
>
> On 05/03/2012 10:07 PM, Benjamin Peterson wrote:
>>> +if (times && ns) {
>>> +PyErr_Format(PyExc_RuntimeError,
>> Why not a ValueError or TypeError?
>
> Well it's certainly not a TypeError. The 3.2 documentation defines TypeError
> as:
>
> Raised when an operation or function is applied to an object of
> inappropriate type. The associated value is a string giving details about
> the type mismatch.
>
> If someone called os.utime with both times and ns, and the values of each
> would
> have been legal if they'd been passed in in isolation, what would be the type
> mismatch?
What exception do you get otherwise when you call a function with inappropriate
argument combinations?
> Is using RuntimeError some sort of Pythonic faux pas?
RuntimeError is not used very much in the stdlib, and if used, then for somewhat
more dramatic errors.
> Finally, I appreciate the feedback, but... why post it to python-dev? You
> could
> have sent me private email, or posted to the issue (#14127), the latter of
> which
> would have enabled using rich chocolaty Rietveld. I've seen a bunch of
> comments
> on checkins posted here and it all leaves me scratching my head.
It has been argued in the past that python-committers is a better place for the
review comments, but it was declined as being "not public enough". I agree that
python-checkins or private email *definitely* isn't public enough.
Georg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
