Re: [Python-Dev] IO module improvements
Le dimanche 07 février 2010 à 09:54 +0100, Pascal Chambon a écrit : > > Actually, TextIOWrapper is simply not thread-safe for most of its > > operations. I > > think we did the work for simple writing, though, since it's better for > > multi-threaded use of print(). > > Argh, I had the impression that all io streams were theoretically > thread-safe (although it's not documented so indeed). It needs > clarification maybe. It should first be discussed which classes need to be thread-safe. There is nothing about it in PEP 3116, and the first (pure Python) implementation of the io module had no locks anywhere. We later added locks to the Buffered classes because it seemed an obvious requirement for many use cases (for example object databases such as ZODB or Durus). > > You can, but be aware that _pyio is *really* slow... I'm not sure it would > > be a > > service to many users. > > > Hum... would a pure python module, augmented with cython declarations, > offer a speed similar to c modules ? Maybe I shall investigate that > way, because it would be great to have an implemntation which is both > safer and sufficiently quick... There's no obvious answer. I suspect that it won't be as fast as the current C implementation, because some things simply aren't possible or available in Python. But it could be "fast enough". You have to experiment. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Feb 06, 2010, at 02:20 PM, Guido van Rossum wrote: >> Upon further reflection, I agree. __file__ also points to the source in >> Python 2.7. > >Not in the 2.7 svn repo I have access to. It still points to the .pyc >file if it was used. Ah, I was fooled by a missing pyc file. Run it a second time and you're right, it points to the pyc. >And I propose not to disturb this in 2.7, at least not by default. I'm >fine though with a flag or distro-overridable config setting to change >this behavior. Cool. I'm not sure this is absolutely necessary for Debian/Ubuntu, so I'll call YAGNI on it for 2.x (until and unless it isn't ;). >> Do we need an attribute to point to the compiled bytecode file? > >I think we do. Quite unrelated to this discussion I have a use case >for knowing easily whether a module was actually loaded from bytecode >or not -- but I also have a need for __file__ to point to the source. >So having both __file__ and __compiled__ makes sense to me. __compiled__ or __cached__? I like the latter but don't have strong feelings about it either way. >When there is no source code but only bytecode I am file with both >pointing to the bytecode; in that case I presume that the bytecode is >not in a __pyr__ subdirectory. For dynamically loaded extension >modules I think both should be left unset, and some other __xxx__ >variable could point to the .so or .dll file. FWIW the most common use >case for __file__ is probably to find data files relative to it. Since >the data won't be in the __pyr__ directory we couldn't make __file__ >point to the __pyr__/pyc file without much code breakage. The other main use case for having such an attribute on extension modules is diagnostics. I want to be able to find out where on the file system a .so actually lives: Python 2.7a3+ (trunk:78030, Feb 6 2010, 15:18:29) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import _socket >>> _socket.__file__ '/home/barry/projects/python/trunk/build/lib.linux-x86_64-2.7/_socket.so' >(Yes, I am still in favor of the folder-per-folder model.) Cool. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Feb 04, 2010, at 03:00 PM, Glenn Linderman wrote: >When a PEP 3147 (if modified by my suggestion) version of Python runs, >and the directory doesn't exist, and it wants to create a .pyc, it would >create the directory, and put the .pyc there. Sort of just like how it >creates .pyc files, now, but an extra step of creating the repository >directory if it doesn't exist. After the first run, it would exist. It >is described in the PEP, and I quoted that section... "Python will >create a 'foo.pyr' directory"... I'm just suggesting different semantics >for how many directories, and what is contained in them. I've added __pyr_version__ as an open question in the PEP (not yet committed), as is making this default behavior (no -R flag required). -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: >With a single cache directory, we could have an option to force writing >bytecode to a desired location. That might be useful on it's own for >creating runtime bytecode only installations for installers. One important reason for wanting to keep the bytecode cache files colocated with the source files is that I want to be able to continue to manipulate $PYTHONPATH to control how Python finds its modules. With a single system-wide cache directory that won't be easy. E.g. $PYTHONPATH might be hacked to find the source file you expect, but how would that interact with how Python finds its cache files? I'm strongly in favor of keeping the cache files as close to the source they were generated from as possible. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On 07/02/2010 17:48, Barry Warsaw wrote: [snip...] And I propose not to disturb this in 2.7, at least not by default. I'm fine though with a flag or distro-overridable config setting to change this behavior. Cool. I'm not sure this is absolutely necessary for Debian/Ubuntu, so I'll call YAGNI on it for 2.x (until and unless it isn't ;). What are the chances of getting this into 2.x at all? For it to get into the 2.7, likely to be the last major version in the 2.x series, the PEP needs to be approved and the implementation needs to be feature complete by April 3rd (first beta release according to the schedule [1]). Michael Foord [1] http://www.python.org/dev/peps/pep-0373/#release-schedule -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Jan 31, 2010, at 08:10 PM, Silke von Bargen wrote: >Martin v. Löwis schrieb: >> There is also the issue of race conditions with multiple simultaneous >> accesses. The original format for the PEP had race conditions for >> multiple simultaneous writers; ZIP will also have race conditions for >> concurrent readers/writers (as any new writer will have to overwrite >> the central directory, making the zip file temporarily unavailable - >> unless they copy it, in which case we are back to writer/writer >> races). >> >> Regards, >> Martin >> >> >Good point. OTOH the probability for this to happen actually is very small. And yet, when it does happen, it's probably a monster to debug and defend against. Unless we have a convincing cross-platform story for preventing these race conditions, I think a single-file (e.g. zipfile) approach is infeasible. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: >I must admit I quite like the __pyr__ directory approach as well. Since >the interpreter knows the suffix it is looking for, names shouldn't >conflict. Using a single directory allows the name to be less cryptic, >too (e.g. __pycache__). Something else that occurs to me; the name of the directory (under folder-per-folder approach) probably ought to be the same as the name of the module attribute. There's probably no good reason to make it different, and making it the same makes the association stronger. That still gives us plenty of opportunity to bikeshed, but __pycache__ seems reasonable to me (it's the cache of parsing and compiling the .py file). -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Feb 01, 2010, at 08:26 AM, Tim Delaney wrote: >The pyc/pyo files are just an optimisation detail, and are essentially >temporary. Given that, if they were to live in a single directory, to me it >seems obvious that the default location for that should be in the system >temporary directory. I an immediately think of the following advantages: > >1. No one really complains too much about putting things in /tmp unless it >starts taking up too much space. In which case they delete it and if it gets >reused, it gets recreated. IIUC the Filesystem Hierarchy Standard correctly, then these files really should go under /var/cache/python. (Don't ask me where that would be on non-FHS compliant systems Windows). I've explained in other followups why I don't particularly like separating the source from the cache files though, but if you wanted a sick approach: Take the full absolutely path to the .py file, plus the magic number, plus the time stamp and hash that. Cache the pyc file under /var/cache/python/. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Feb 06, 2010, at 04:02 PM, Guido van Rossum wrote: >On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw wrote: >> On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: >> >>>It's an interesting challenge to write the file in such a way that >>>it's safe for a reader and writer to co-exist. Like Brett, I >>>considered an append-only scheme, but one needs to handle the case >>>where the bytecode for a particular magic number changes. At some >>>point you'd need to sweep garbage from the file. All solutions seem >>>unnecessarily complex, and unnecessary since in practice the case >>>should not come up. >> >> I don't think that part's difficult. The byte code's only going to change if >> the source file has changed, and in that case, /all/ the byte code in the >> "fat >> pyc" file will be invalidated, so the whole thing can be deleted by the first >> writer. I'd worked that out in the original fat pyc version of the PEP. > >I'm sorry, but I'm totally against fat bytecode files. They make >things harder for all tools. The beauty of the existing bytecode >format is that it's totally trivial: magic number, source mtime, >unmarshalled code object. You can't beat the beauty of that. Just for the record, I totally agree. I was just explaining something I had figured out in the original version of the PEP, which wasn't published but which Martin had seen an early draft of. When Martin made the suggestion of sibling cache directories, I immediately realized that it was much cleaner, better, and easier to implement than fat files (especially because I already had some nasty complex code that implemented the fat files ;). I'm beginning to be convinced that a folder-per-folder approach is the best take on this yet. >For the traditional "skinny" bytecode files, I believe that the >existing algorithm which writes zeros in the place of the magic number >first, writes the rest of the file, and then goes back to write the >correct magic number, is correct with a single writer and multiple >readers (assuming the readers ignore the file if its magic number is >invalid). The creat(O_EXCL) option ensures that there won't be >multiple writers. No rename() is necessary; POSIX rename() may be >atomic, but it's a directory modification which makes it potentially >slow. Agreed, and the current approach is time and battle tested. I don't think we need to be mucking around with it. My current effort on this PEP will be spent on fleshing out the folder-per-folder approach, understanding the implications of that, and integrating all the other great comments in this thread. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: >The conflict is purely that PEP 3147 proposes the new behavior to be >optional, and adds a flag (-R) and an environment variable >($PYTHONPYR) to change it. I presume Barry is proposing this out of >fear that the new behavior might upset somebody; personally I think it >would be better if the behavior weren't optional. At least not in new >Python releases Good to know! Yes, that's one reason why I made it option, the other being that I suspect most people don't care about the original use case (making sure pyc files from different Python versions don't conflict). However, with a folder-per-folder approach, the side benefit of reducing directory clutter by hiding all the pyc files becomes more compelling. > -- in backports such as a distribution that wants this >feature might make, it may make sense to be more conservative, or at >least to have a way to turn it off. For backports I think the most conservative approach is to require a flag to enable this behavior. If we make this the default for new versions of Python (something I'd support) then tools written for Python >= 3.2 will know this is just how it's done. I worry about existing deployed tools for Python < 2.7 and 3.1. How about this: enable it by default in 3.2 and 2.7. No option to disable it. Allow distro back ports to define a flag or environment variable to enable it. The PEP can even be silent about how that's actually done, and a Debian implementation for Python 2.6 or 3.1 could even use the (now documented :) -X flag. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Feb 07, 2010, at 05:59 PM, Michael Foord wrote: >On 07/02/2010 17:48, Barry Warsaw wrote: >> [snip...] >>> And I propose not to disturb this in 2.7, at least not by default. I'm >>> fine though with a flag or distro-overridable config setting to change >>> this behavior. >>> >> Cool. I'm not sure this is absolutely necessary for Debian/Ubuntu, so I'll >> call YAGNI on it for 2.x (until and unless it isn't ;). Sorry, I was calling YAGNI on any change in behavior of module.__file__. >What are the chances of getting this into 2.x at all? For it to get into >the 2.7, likely to be the last major version in the 2.x series, the PEP >needs to be approved and the implementation needs to be feature complete >by April 3rd (first beta release according to the schedule [1]). I'd like to consult with my Debian/Ubuntu Python maintainer colleagues to see if it's worth getting into 2.7. If it is, and we can get a BDFL pronouncement on the PEP (after the next rounds of updates), then I think it will be feasible to implement in the time remaining. Heck, that's what Pycon sprints are for, no? :) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ is not always an absolute path
On Feb 06, 2010, at 11:22 PM, exar...@twistedmatrix.com wrote: >>I haven't tried to repro this particular example, but the reason is >>that we don't want to have to call getpwd() on every import nor do we >>want to have some kind of in-process variable to cache the current >>directory. (getpwd() is relatively slow and can sometimes fail >>outright, and trying to cache it has a certain risk of being wrong.) > >Assuming you mean os.getcwd(): > >exar...@boson:~$ python -m timeit -s 'def f(): pass' 'f()' >1000 loops, best of 3: 0.132 usec per loop >exar...@boson:~$ python -m timeit -s 'from os import getcwd' 'getcwd()' >100 loops, best of 3: 1.02 usec per loop >exar...@boson:~$ >So it's about 7x more expensive than a no-op function call. I'd call >this pretty quick. Compared to everything else that happens during an >import, I'm not convinced this wouldn't be lost in the noise. I think >it's at least worth implementing and measuring. I'd like to see the effect on command line scripts that are run often and then exit, e.g. Bazaar or Mercurial. Start up time due to import overhead seems to be a constant battle for those types of projects. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ is not always an absolute path
Barry Warsaw python.org> writes: > > >exarkun boson:~$ python -m timeit -s 'from os import getcwd' 'getcwd()' > >100 loops, best of 3: 1.02 usec per loop [...] > > I'd like to see the effect on command line scripts that are run often and then > exit, e.g. Bazaar or Mercurial. Start up time due to import overhead seems to > be a constant battle for those types of projects. If os.getcwd() is only called once when "normalizing" sys.path, and if it just takes one microsecond, I don't really see the point. :-) Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
Barry Warsaw wrote: > On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: > >> How about using an optionally relative cache dir setting to let >> the user decide ? > > Why do we need that level of flexibility? It's very easy to implement (see the code I posted) and gives you a lot of control with a single env variable. Some use cases: 1. PYTHONCACHE=. (store the cache files in the same dir as the .py file) This settings mimics what we've had in Python for decades. Users know about this Python behavior and expect it. It's also the only reasonable way of shipping byte-code only packages. 2. PYTHONCACHE=.pycache (store the cache files in a subdir of the dir where the .py file is stored) When using lots of cache files for multiple Python versions or variants, .py source code directory can easily get cluttered with too many such files. Putting them into a subdir solves this problem. This would be useful for developers running and testing the code with different Python versions. 3. PYTHONCACHE=~/.python/cache (store the cache files in a user dir, outside the Python source file dir) This allows easy removal of all cache files and prevents cluttering up the sys.path dirs with cache files or directories altogether. It's also handy if the source code dirs are not writable by the user importing them. OTOH, every user would create a copy of the cache files (this is what currently happens with setuptools eggs and is very annoying). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 07 2010) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Sun, Feb 7, 2010 at 10:17 AM, Barry Warsaw wrote: > On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: > >>I must admit I quite like the __pyr__ directory approach as well. Since >>the interpreter knows the suffix it is looking for, names shouldn't >>conflict. Using a single directory allows the name to be less cryptic, >>too (e.g. __pycache__). > > Something else that occurs to me; the name of the directory (under > folder-per-folder approach) probably ought to be the same as the name of the > module attribute. There's probably no good reason to make it different, and > making it the same makes the association stronger. I'm not sure I follow. The directory doesn't suddenly become an attribute. Moreover, the directory contains many files (assuming folder-per-folder) and the attribute would point to a single file inside that directory. > That still gives us plenty of opportunity to bikeshed, but __pycache__ seems > reasonable to me (it's the cache of parsing and compiling the .py file). While technically it is a cache, I don't think that emphasizing that point is helpful. For 20 years people have thought of it as "compiled bytecode". Also while on the filesystem it makes sense for it to have "py" in the directory name, that does not make sense for the attribute name. After all we don't go around calling things __pyfile__, __pygetattr__, __pysys__... ;-) I'm still for __compiled__ as the attribute; I don't have a particular preference for the directory name or the naming scheme used inside it, as long as neither starts with '.' (and probably the directory should be __something__). -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Sun, Feb 7, 2010 at 10:44, Barry Warsaw wrote: > On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: > >>The conflict is purely that PEP 3147 proposes the new behavior to be >>optional, and adds a flag (-R) and an environment variable >>($PYTHONPYR) to change it. I presume Barry is proposing this out of >>fear that the new behavior might upset somebody; personally I think it >>would be better if the behavior weren't optional. At least not in new >>Python releases > > Good to know! Yes, that's one reason why I made it option, the other being > that I suspect most people don't care about the original use case (making sure > pyc files from different Python versions don't conflict). However, with a > folder-per-folder approach, the side benefit of reducing directory clutter by > hiding all the pyc files becomes more compelling. > >> -- in backports such as a distribution that wants this >>feature might make, it may make sense to be more conservative, or at >>least to have a way to turn it off. > > For backports I think the most conservative approach is to require a flag to > enable this behavior. If we make this the default for new versions of Python > (something I'd support) then tools written for Python >= 3.2 will know this is > just how it's done. I worry about existing deployed tools for Python < 2.7 > and 3.1. > > How about this: enable it by default in 3.2 and 2.7. No option to disable it. > Allow distro back ports to define a flag or environment variable to enable it. > The PEP can even be silent about how that's actually done, and a Debian > implementation for Python 2.6 or 3.1 could even use the (now documented :) -X > flag. Would you keep the old behavior around as well, or simply drop it? I personally vote for the latter for simplicity and performance reasons (by not having to look in so many places for bytecode), but I can see tool people who magically calculate the location of the bytecode not loving the idea (another reason why giving loaders a method to return all relevant paths is a good idea; no more guessing). -Brett > > -Barry > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 385 progress report
It's been a long time! So for the past few weeks, Mercurial crew member Patrick Mezard has been hunting for the ugly bug in hgsubversion that I'd previously been looking at, and it finally got fixed. A new bug popped up, but then we managed to fix that, too (thanks to the PSF for partially funding our sprint, it was very succesful!). In a joyous moment, I nagged Augie Fackler to actually put a hgsubversion release out there so hopefully more people can start using it, so we now have that, too. Another sponsor for our sprint was Logilab (who provided their brand new office for us to work in), and one of their employees, Andre Espaze, fortunately wanted to help out and managed to write up a patch for the sys.mercurial attribute (now in the pymigr repo). In fact, a few weeks ago I talked to Brett and we figured that we should probably pin down a deadline. We discussed aiming at May 1, and at this time I think that should be feasible. That also seems to coincide with the release of 2.7b2, though, so maybe we need to do it one week later (or sooner?). Anyway, we figured that a weekend would probably be a good time. If we manage to find a good date, I'll put it in the PEP. As for the current state of The Dreaded EOL Issue, there is an extension which seems to be provide all the needed features, but it appears there are some nasty corner cases still to be fixed. Martin Geisler has been working on it over the sprint, but I think there's more work to be done here. Anyone who wants to jump in would be quite welcome (maybe Martin will clarify here what exactly the remaining issues are). The current version of the repository (latest SVN revision is 78055, clone it from hg.python.org) weighs in at about 1.4G, but still needs branch pruning (which will be my primary focus for the coming few weeks). The good part about it now being a year later than, well, last year is that named branches are much more solid than before, and so I feel much better about using those for Python's release branches. Any questions and/or concerns? I will also be at PyCon; I'll be doing a more advanced talk on Mercurial internals on Sunday but I'd also be happy to do some handholding or introductory stuff in an open space. If there's anyone who'd like help converting their SVN repository, I might be able to help there too (during the sprints). For other conversions, I know for a fact that an expert in CVS conversions will be there. Cheers, Dirkjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385 progress report
2010/2/7 Dirkjan Ochtman : > It's been a long time! Thank you very much for staying on this task! I'm still excited. > > In fact, a few weeks ago I talked to Brett and we figured that we > should probably pin down a deadline. We discussed aiming at May 1, and > at this time I think that should be feasible. That also seems to > coincide with the release of 2.7b2, though, so maybe we need to do it > one week later (or sooner?). Anyway, we figured that a weekend would > probably be a good time. If we manage to find a good date, I'll put it > in the PEP. How about a week after, so we have more time to adjust release procedures? > Any questions and/or concerns? Will you do test conversions of the sandbox projects, too? Also I think we should have some document (perhaps the dev FAQ) explaining exactly how to do common tasks in mercurial. For example - A bug fix, which needs to be in 4 branches. - A bug fix, which only belongs in 2.7 and 2.6 or 3.2 and 3.1. - Which way do we merge (What's a subset of what?) -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385 progress report
Hi Dirkjan, On 8/02/2010 8:35 AM, Dirkjan Ochtman wrote: ... In fact, a few weeks ago I talked to Brett and we figured that we should probably pin down a deadline. We discussed aiming at May 1, and at this time I think that should be feasible. That also seems to coincide with the release of 2.7b2, though, so maybe we need to do it one week later (or sooner?). Anyway, we figured that a weekend would probably be a good time. If we manage to find a good date, I'll put it in the PEP. Isn't setting a date premature while outstanding issues remain without a timetable for their resolution? As for the current state of The Dreaded EOL Issue, there is an extension which seems to be provide all the needed features, but it appears there are some nasty corner cases still to be fixed. Martin Geisler has been working on it over the sprint, but I think there's more work to be done here. Anyone who wants to jump in would be quite welcome (maybe Martin will clarify here what exactly the remaining issues are). See http://mercurial.selenic.com/wiki/EOLTranslationPlan#TODO - of particular note: * There are transient errors in the tests which Martin is yet to identify. These tests do not require windows to reproduce or fix. * The mercurial tests do not run on Windows. Given the above, most sane Windows developers would hold off on "live" testing of the extension until at least the first issue is resolved - but the second issue makes it very difficult for them to help resolve that. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Sun, Feb 7, 2010 at 12:23 PM, Brett Cannon wrote: > On Sun, Feb 7, 2010 at 10:44, Barry Warsaw wrote: >> On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: >> >>>The conflict is purely that PEP 3147 proposes the new behavior to be >>>optional, and adds a flag (-R) and an environment variable >>>($PYTHONPYR) to change it. I presume Barry is proposing this out of >>>fear that the new behavior might upset somebody; personally I think it >>>would be better if the behavior weren't optional. At least not in new >>>Python releases >> >> Good to know! Yes, that's one reason why I made it option, the other being >> that I suspect most people don't care about the original use case (making >> sure >> pyc files from different Python versions don't conflict). However, with a >> folder-per-folder approach, the side benefit of reducing directory clutter by >> hiding all the pyc files becomes more compelling. >> >>> -- in backports such as a distribution that wants this >>>feature might make, it may make sense to be more conservative, or at >>>least to have a way to turn it off. >> >> For backports I think the most conservative approach is to require a flag to >> enable this behavior. If we make this the default for new versions of Python >> (something I'd support) then tools written for Python >= 3.2 will know this >> is >> just how it's done. I worry about existing deployed tools for Python < 2.7 >> and 3.1. >> >> How about this: enable it by default in 3.2 and 2.7. No option to disable >> it. >> Allow distro back ports to define a flag or environment variable to enable >> it. >> The PEP can even be silent about how that's actually done, and a Debian >> implementation for Python 2.6 or 3.1 could even use the (now documented :) -X >> flag. > > Would you keep the old behavior around as well, or simply drop it? I > personally vote for the latter for simplicity and performance reasons > (by not having to look in so many places for bytecode), but I can see > tool people who magically calculate the location of the bytecode not > loving the idea (another reason why giving loaders a method to return > all relevant paths is a good idea; no more guessing). For 3.2 I think it's fine to simply drop the old behavior (as long as a good loader API is added at the same time). But for 2.7 I think we ought to be a lot more conservative and not force tools to upgrade, so I think we should keep the old behavior in 2.7 as the default (though distros can change this if they want to, and backport if they need to). -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
Barry Warsaw wrote: On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: With a single cache directory, we could have an option to force writing bytecode to a desired location. That might be useful on it's own for creating runtime bytecode only installations for installers. One important reason for wanting to keep the bytecode cache files colocated with the source files is that I want to be able to continue to manipulate $PYTHONPATH to control how Python finds its modules. With a single system-wide cache directory that won't be easy. E.g. $PYTHONPATH might be hacked to find the source file you expect, but how would that interact with how Python finds its cache files? I'm strongly in favor of keeping the cache files as close to the source they were generated from as possible. Yes, I agree, after thinking about it, it does seems like it may be more complex than I first thought. I think the folder-per-folder option sounds like the best default option at this time. It reduces folder clutter for the python developer and may loosen the link between source files and byte code files just enough that it will be easier to experiment with more flexible modes later. It seems to me that in the long run, (probably no time soon), it might be nice to even do away with on disk byte code altogether unless it's explicitly asked for. As computers get faster, the time it takes to compile byte code may become a smaller and smaller percent of the total run time. That is unless the size of python programs increase at the same rate or faster. To tell the truth in most cases I hardly notice the extra time the first run takes compared to later runs with the precompiled byte code. Yes it may be a few seconds at start up, but after that it's usually not a big part of the execution time. Hmmm, I wonder if there's a threshold in file size where it really doesn't make a significant difference? Regards, Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
