Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator
On 26 June 2014 23:59, Ben Hoyt wrote: > Would love feedback on the PEP, but also of course on the proposal itself. A solid +1 from me. Some specific points: - I'm in favour of it being in the os module. It's more discoverable there, as well as the other reasons mentioned. - I prefer scandir as the name, for the reason you gave (the output isn't the same as an iterator version of listdir) - I'm mildly against windows_wildcard (even though I'm a windows user) - You mention the caching behaviour of DirEntry objects. The limitations should be clearly covered in the final docs, as it's the sort of thing people will get wrong otherwise. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator
On 29 June 2014 12:08, Nick Coghlan wrote: > This is what makes me wary of including lstat, even though Windows > offers it without the extra stat call. Caching behaviour is *really* > hard to make intuitive, especially when it *sometimes* returns data > that looks fresh (as it on first call on POSIX systems). If it matters that much we *could* simply call it cached_lstat(). It's ugly, but I really don't like the idea of throwing the information away - after all, the fact that we currently throw data away is why there's even a need for scandir. Let's not make the same mistake again... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My summary of the scandir (PEP 471)
On 1 July 2014 14:00, Ben Hoyt wrote: > 2) Nick Coghlan's proposal on the previous thread > (https://mail.python.org/pipermail/python-dev/2014-June/135261.html) > suggesting an ensure_lstat keyword param to scandir if you need the > lstat_result value > > I would make one small tweak to Nick Coghlan's proposal to make > writing cross-platform code easier. Instead of .lstat_result being > None sometimes (on POSIX), have it None always unless you specify > ensure_lstat=True. (Actually, call it get_lstat=True to kind of make > this more obvious.) Per (b) above, this means Windows developers > wouldn't accidentally write code which failed on POSIX systems -- it'd > fail fast on Windows too if you accessed .lstat_result without > specifying get_lstat=True. This is getting very complicated (at least to me, as a Windows user, where the basic idea seems straightforward). It seems to me that the right model is the standard "thin wrapper round the OS feature" that acts as a building block - it's typical of the rest of the os module. I think that thin wrapper is needed - even if the various bells and whistles are useful, they can be built on top of a low-level version (whereas the converse is not the case). Typically, such thin wrappers expose POSIX semantics by default, and Windows behaviour follows as closely as possible (see for example stat, where st_ino makes no sense on Windows, but is present). In this case, we're exposing Windows semantics, and POSIX is the one needing to fit the model, but the principle is the same. On that basis, optional attributes (as used in stat results) seem entirely sensible. The documentation for DirEntry could easily be written to parallel that of a stat result: """ The return value is an object whose attributes correspond to the data the OS returns about a directory entry: * name - the object's name * full_name - the object's full name (including path) * is_dir - whether the object is a directory * is file - whether the object is a plain file * is_symlink - whether the object is a symbolic link On Windows, the following attributes are also available * st_size - the size, in bytes, of the object (only meaningful for files) * st_atime - time of last access * st_mtime - time of last write * st_ctime - time of creation * st_file_attributes - Windows file attribute bits (see the FILE_ATTRIBUTE_* constants in the stat module) """ That's no harder to understand (or to work with) than the equivalent stat result. The only difference is that the unavailable attributes can be queried on POSIX, there's just a separate system call involved (with implications in terms of performance, error handling and potential race conditions). The version of scandir with the ensure_lstat argument is easy to write based on one with optional arguments (I'm playing fast and loose with adding attributes to DirEntry values here, just for the sake of an example - the details are left as an exercise) def scandir_ensure(path='.', ensure_lstat=False): for entry in os.scandir(path): if ensure_lstat and not hasattr(entry, 'st_size'): stat_data = os.lstat(entry.full_name) entry.st_size = stat_data.st_size entry.st_atime = stat_data.st_atime entry.st_mtime = stat_data.st_mtime entry.st_ctime = stat_data.st_ctime # Ignore file_attributes, as we'll never get here on Windows yield entry Variations on how you handle errors in the lstat call, etc, can be added to taste. Please, let's stick to a low-level wrapper round the OS API for the first iteration of this feature. Enhancements can be added later, when real-world usage has proved their value. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My summary of the scandir (PEP 471)
tl;dr - I agree with your points and think that the original PEP 471 proposal is fine. The details here are just clarification of why my proposal wasn't just "use PEP 471 as written" in the first place... On 2 July 2014 13:41, Ben Hoyt wrote: > 1) It's a nasty API to actually write code with. If you try to use it, > it gives off a "made only for low-level library authors" rather than > "designed for developers" smell. For example, here's a get_tree_size() > function I use written in both versions (original is the PEP 471 > version with the addition of .full_name): > > def get_tree_size_original(path): > """Return total size of all files in directory tree at path.""" > total = 0 > for entry in os.scandir(path): > if entry.is_dir(): > total += get_tree_size_original(entry.full_name) > else: > total += entry.lstat().st_size > return total > > def get_tree_size_new(path): > """Return total size of all files in directory tree at path.""" > total = 0 > for entry in os.scandir(path): > if hasattr(entry, 'is_dir') and hasattr(entry, 'st_size'): > is_dir = entry.is_dir > size = entry.st_size > else: > st = os.lstat(entry.full_name) > is_dir = stat.S_ISDIR(st.st_mode) > size = st.st_size > if is_dir: > total += get_tree_size_new(entry.full_name) > else: > total += size > return total > > I know which version I'd rather write and maintain! Fair point. But *only* because is_dir isn't guaranteed to be available. I could debate other aspects of your translation to use my API, but it's not relevant as my proposal was flawed in terms of is_XXX. > It seems to me new > users and folks new to Python could easily write the top version, but > the bottom is longer, more complicated, and harder to get right. Given the is_dir point, agreed. > It > would also be very easy to write code in a way that works on Windows > but bombs hard on POSIX. You may have a point here - my Windows bias may be showing. It's already awfully easy to write code that works on POSIX but bombs hard on Windows (deleting open files, for example) so I find it tempting to think "give them a taste of their own medicine" :-) More seriously, it seems to me that the scandir API is specifically designed to write efficient code on platforms where the OS gives information that allows you to do so. Warping the API too much to cater for platforms where that isn't possible seems to have the priorities backwards. Making the API not be an accident waiting to happen is fine, though. And let's be careful, too. My position is that it's not too hard to write code that works on Windows, Linux and OS X but you're right you could miss the problem with platforms that don't even support a free is_dir(). It's *easier* to write Windows-only code by mistake, but the fix to cover the "big three" is pretty simple (if not hasattr, lstat). > 2) It seems like your assumption is that is_dir/is_file/is_symlink are > always available on POSIX via readdir. This isn't actually the case > (this was discussed in the original threads) -- if readdir() returns > dirent.d_type as DT_UNKNOWN, then you actually have to call os.stat() > anyway to get it. So, as the above definition of get_tree_size_new() > shows, you have to use getattr/hasattr on everything: > is_dir/is_file/is_symlink as well as the st_* attributes. Ah, the wording in the PEP says "Linux, Windows, OS X". Superficially, that said "everywhere" to me. It might be worth calling out specifically some examples where it's not available without an extra system call, just to make the point explicit. You're right, though, that blows away the simplicity of my proposal. The original PEP 471 seems precisely right to me, in that case. I was only really arguing for attributes because they seem more obviously static than a method call. And personally I don't care about that aspect. > 3) It's not much different in concept to the PEP 471 version, except > that PEP 471 has a built-in .lstat() method, making the user's life > much easier. This is the sense in which it's the worst of both worlds > -- it's a far less nice API to use, but it still has the same issues > with race conditions the original does. Agreed. My intent was never to remove the race conditions, I see them as the responsibility of the application to consider (many applications simply won't care, and those that do will likely want a specific solution, not a library-level compromise). > So thinking about this again: > > First, based on the +1's to Paul's new solution, I don't think people > are too concerned about the race condition issue (attributes being > different between the original readdir and the os.stat calls). I think > this is probably fair -- if folks care, they can handle it in an > application-specific way. So that means Paul's new solution and the > original PEP 471 appr
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
On 9 July 2014 02:08, Ben Hoyt wrote: > Comments and votes, please! +1 on option 1 (current PEP approach) at the moment, but I would like to see how the error handling would look (suppose the function logs files that can't be statted, and assumes a size of 0 for them). The idea of a multi-level ensure_lstat isn't unreasonable, either, and that helps option 2. The biggest issue *I* see with option 2 is that people won't remember to add the ensure_XXX argument, and that will result in more code that seems to work but fails cross-platform. Unless scandir deliberately fails if you use an attribute that you haven't "ensured", but that would be really unfriendly... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
On 9 July 2014 13:48, Ben Hoyt wrote:
> Okay folks -- please respond: option #1 as per the current PEP 471, or
> option #2 with Ethan's multi-level thing tweaks as per the above?
I'm probably about 50/50 at the moment. What will swing it for me is
likely error handling, so let's try both approaches with some error
handling:
Rules are that we calculate the total size of all files in a tree (as
returned from lstat), with files that fail to stat being logged and
their size assumed to be 0.
Option 1:
def get_tree_size(path):
total = 0
for entry in os.scandir(path):
try:
isdir = entry.is_dir()
except OSError:
logger.warn("Cannot stat {}".format(entry.full_name))
continue
if entry.is_dir():
total += get_tree_size(entry.full_name)
else:
try:
total += entry.lstat().st_size
except OSError:
logger.warn("Cannot stat {}".format(entry.full_name))
return total
Option 2:
def log_err(exc):
logger.warn("Cannot stat {}".format(exc.filename))
def get_tree_size(path):
total = 0
for entry in os.scandir(path, info='lstat', onerror=log_err):
if entry.is_dir:
total += get_tree_size(entry.full_name)
else:
total += entry.lstat.st_size
return total
On this basis, #2 wins. However, I'm slightly uncomfortable using the
filename attribute of the exception in the logging, as there is
nothing in the docs saying that this will give a full pathname. I'd
hate to see "Unable to stat __init__.py"!!!
So maybe the onerror function should also receive the DirEntry object
- which will only have the name and full_name attributes, but that's
all that is needed.
OK, looks like option #2 is now my preferred option. My gut instinct
still rebels over an API that deliberately throws information away in
the default case, even though there is now an option to ask it to keep
that information, but I see the logic and can learn to live with it.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
On 9 July 2014 14:22, Ben Hoyt wrote: >> So maybe the onerror function should also receive the DirEntry object >> - which will only have the name and full_name attributes, but that's >> all that is needed. > > That's an interesting idea -- though enough of a deviation from > os.walk()'s onerror that I'm uncomfortable with it -- I'd rather just > document that the onerror exception .filename is the full path name. But the onerror exception will come from the lstat call, so it'll be a raw OSError (unless scandir modifies it, which may be what you're thinking of). And if so, aren't we at the mercy of what the OS module gives us? That's why I said we can't guarantee it. I looked at the documentation of OSError (in "Built In Exceptions"), and all it says is "the filename" (unqualified). I'd expect that to be "whatever got passed to the underlying OS API" - which may well be an absolute pathname if we're lucky, but who knows? (I'd actually prefer it if OSError guaranteed a full pathname, but that's a bigger issue...) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
On 9 July 2014 16:05, Victor Stinner wrote: > The PEP says that DirEntry should mimic pathlib.Path, so I think that > DirEntry.is_dir() should work as os.path.isir(): if the entry is a > symbolic link, you should follow the symlink to get the status of the > linked file with os.stat(). Would this not "break" the tree size script being discussed in the other thread, as it would follow links and include linked directories in the "size" of the tree? As a Windows user with only a superficial understanding of how symlinks should behave, I don't have any intuition as to what the "right" answer should be. But I would say that the tree size code we've been debating over there (which recurses if is_dir is true and adds in st_size otherwise) should do whatever people would expect of a function with that name, as it's a perfect example of something a Windows user might write and expect it to work cross-platform. If it doesn't much of the worrying over making sure scandir's API is "cross-platform by default" is probably being wasted :-) (Obviously the walk_tree function could be modified if needed, but that's missing the point I'm trying to make :-)) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
On 9 July 2014 14:22, Ben Hoyt wrote: > One issue with option #2 that I just realized -- does scandir yield > the entry at all if there's a stat error? It can't really, because the > caller will except the .lstat attribute to be set (assuming he asked > for type='lstat') but it won't be. Is effectively removing these > entries just because the stat failed a problem? I kind of think it is. > If so, is there a way to solve it with option #2? So the issue is that you need to do a stat but it failed. You have "whatever the OS gave you", but can't get anything more. This is only an issue on POSIX, where the original OS call doesn't give you everything, so it's fine, those POSIX people can just learn to cope with their broken OS, right? :-) More seriously, why not just return a DirEntry that says it's a file with a stat entry that's all zeroes? That seems pretty harmless. And the onerror function will be called, so if it is inappropriate the application can do something. Maybe it's worth letting onerror return a boolean that says whether to skip the entry, but that's as far as I'd bother going. It's a close call, but I think option #2 still wins (just) for me. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
On 9 July 2014 17:35, Ethan Furman wrote: > More specifically, if we go with choice 1 (no built-in error handling, no > mutable DirEntry), how would I implement choice 2? Would I have to write my > own CustomDirEntry object? Having built-in error handling is, I think, a key point. That's where #1 really falls down. But a mutable DirEntry and/or letting onerror manipulate the result is a lot more than just having a hook for being notified of errors. That seems to me to be a step too far, in the current context. Specifically, the tree size example doesn't need it. Do you have a compelling use case that needs a mutable DirEntry? It feels like YAGNI to me. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
On 9 July 2014 21:24, Victor Stinner wrote: > Example where you may sometimes need is_dir(), but not always > --- > for entry in os.scandir(path): > if ignore_entry(entry.name): > # this entry is not interesting, lstat_result is useless here > continue > if entry.is_dir(): # fetch required data if needed > continue > ... > --- That is an extremely good point, and articulates why I've always been a bit uncomfortable with the whole ensure_stat idea. > I don't understand why you are all focused on handling os.stat() and > os.lstat() errors. See for example the os.walk() function which is an > old function (python 2.6!): it doesn't catch erros on isdir(), even if > it has an onerror parameter... It only handles errors on listdir(). > IMO errors on os.stat() and os.lstat() are very rare under very > specific conditions. The most common case is that you can get the > status if you can list files. Personally, I'm only focused on it as a response to others feeling it's important. I'm on Windows, where there are no extra stat calls, so all *I* care about is having an API that deals with the use cases others are concerned about without making it too hard for me to use it on Windows where I don't have to worry about all this. If POSIX users come to a consensus that error handling doesn't need special treatment, I'm more than happy to go back to the PEP version. (Much as previously happened with the race condition debate). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
On 10 July 2014 01:23, Victor Stinner wrote: >> As a Windows user with only a superficial understanding of how >> symlinks should behave, (...) > > FYI Windows also supports symbolic links since Windows Vista. The > feature is unknown because it is restricted to the administrator > account. Try the "mklink" command in a terminal (cmd.exe) ;-) > http://en.wikipedia.org/wiki/NTFS_symbolic_link > > ... To be honest, I never created a symlink on Windows. But since it > is supported, you need to know it to write correctly your Windows > code. I know how symlinks *do* behave, and I know how Windows supports them. What I meant was that, because Windows typically makes little use of symlinks, I have little or no intuition of what feels natural to people using an OS where symlinks are common. As someone (Tim?) pointed out later in the thread, FindFirstFile/FindNextFile doesn't follow symlinks by default (and nor do the dirent entries on Unix). So whether or not it's "natural", the "free" functionality provided by the OS is that of lstat, not that of stat. Presumably because it's possible to build symlink-following code on top of non-following code, but not the other way around. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()
On 15 July 2014 13:19, Ben Hoyt wrote: > Hmmm, perhaps. You suggest .full_name implies it's the absolute path, > which isn't true. I don't mind .path, but it kind of sounds like "the > Path object associated with this entry". I think "full_name" is fine > -- it's not "abs_name". Interesting. I hadn't really thought about it, but I might have assumed full_name was absolute. However, now I see that it's "only as absolute as the directory argument to scandir is". Having said that, I don't think that full_name *implies* that, just that it's a possible mistake people could make. I agree that "path" could be seen as implying a Path object. My preference would be to retain the name full_name, but just make it explicit in the documentation that it is based on the directory name argument. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Exposing the Android platform existence to Python modules
On 1 August 2014 02:54, Glenn Linderman wrote: > I've no idea what you mean by "userland" in your suggestions above or below, > but doesn't the Android environment qualify as a (multi-versioned) platform > independently of its host OS? Seems I've read about an Android > reimplementation for Windows, for example. As long as all the services > expected by Android are faithfully produced, the host OS may be irrelevant > to an Android application... in which case, I would think/propose/suggest > the platform name should change from win32 or linux to Android (and the > Android version be reflected in version parts). Alternatively, if having sys.platform be "linux" makes portability easier because code that does a platform check generally gets the right answer if Android reports as "linux", then why not make sys.linux_distribution report "android"? To put it briefly, either android is the platform, or android is a specific distribution of the linux platform. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pathlib handling of trailing slash (Issue #21039)
On 7 August 2014 02:55, Antoine Pitrou wrote: > pathlib is generally concerned with filesystem operations written in Python, > not arbitrary third-party tools. Also it is probably easy to append the > trailing slash in your command-line invocation, if so desired. I had a use case where I wanted to allow a config file to contain "path: foo" to create a file called foo, and "path: foo/" to create a directory. It was a shortcut for specifying an explicit "directory: true" parameter as well. The fact that pathlib stripped the slash made coding this mildly tricky (especially as I wanted to cater for Windows users writing "foo\\"...) It's not a showstopper, but I agree that semantically, being able to distinguish whether an input had a trailing slash is sometimes useful. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On 9 August 2014 06:08, Steven D'Aprano wrote: > py> with Stopwatch(): > ... sum(carray) # carray is a numpy array of 7500 floats. > ... > 11250.0 > time taken: 52.659770 seconds > py> with Stopwatch(): > ... numpy.sum(carray) > ... > 11250.0 > time taken: 0.161263 seconds > > >> Why have builtin sum at all if its use comes with so many caveats? > > Because sum() is a perfectly reasonable general purpose tool for adding > up small amounts of numbers where high floating point precision is not > required. It has been included as a built-in because Python comes with > "batteries included", and a basic function for adding up a few numbers > is an obvious, simple battery. But serious programmers should be > comfortable with the idea that you use the right tool for the right job. Changing the subject a little, but the Stopwatch function you used up there is "an obvious, simple battery" for timing a chunk of code at the interactive prompt. I'm amazed there's nothing like it in the timeit module... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On 20 August 2014 07:53, Ben Finney wrote: > "Stephen J. Turnbull" writes: > >> Marko Rauhamaa writes: >> > Unix programmers, though, shouldn't be shielded from bytes. >> >> Nobody's trying to do that. But Python users should be shielded from >> Unix programmers. > > +1 QotW That quote is actually almost a "hidden extra Zen of Python" IMO :-) Both parts of it. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On 21 August 2014 23:27, Cameron Simpson wrote: > That's not "ASCII compatible". That's "not all byte codes can be freely used > without thought", and any multibyte coding will have to consider such things > when embedding itself in another coding scheme. I wonder how badly a Unix system would break if you specified UTF16 as the system encoding...? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On 23 August 2014 16:15, Oleg Broytman wrote: > On Sat, Aug 23, 2014 at 06:02:06PM +0900, "Stephen J. Turnbull" > wrote: >> And that's the big problem with Oleg's complaint, too. It's not at >> all clear what he wants > >The first thing is I want to understand why people continue to refer > to Unix was as "broken". Better yet, to persuade them it's not. Generally, it seems to be mostly a reaction to the repeated claims that Python, or Windows, or whatever, is "broken". Unix advocates (not yourself) are prone to declaring anything *other* than the Unix model as "broken", so it's tempting to give them a taste of their own medicine. Sorry for that (to the extent that I was one of the people doing so). Rhetoric aside, none of Unix, Windows or Python are "broken". They just react in different ways to fundamentally difficult edge cases. But expecting Python (a cross-platform language) to prefer the Unix model is putting all the pain on non-Unix users of Python, which I don't feel is reasonable. Let's all compromise a little. Paul PS The key thing *I* think is a problem with the Unix behaviour is that it treats filenames as bytes rather than Unicode. People name files using *characters*. So every filename is semantically text, in the mind of the person who created it. Unix enforces a transformation to bytes, but does not retain the encoding of those bytes. So information about the original author's intent is lost. But that's a historical fact, baked into Unix at a low level. Whether that's "broken" or just "something to deal with" is not important to me. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On 23 August 2014 19:37, Oleg Broytman wrote: > Unix takes the idea that everything is text and a stream of bytes to > its extreme. I don't really understand the idea of "text and a stream of bytes". The two are fundamentally different in my view. But I guess that's why we have to agree to differ - our perspectives are just very different. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows Unicode console support [Was: Bytes path support]
On 24 August 2014 04:27, Nick Coghlan wrote: > One of those areas is the fact that we still use the old 8-bit APIs to > interact with the Windows console. Those are just as broken in a > multilingual world as the other Windows 8-bit APIs, so Drekin came up > with a project to expose the Windows console as a UTF-16-LE stream > that uses the 16-bit APIs instead: > https://pypi.python.org/pypi/win_unicode_console > > I personally hope we'll be able to get the issues Drekin references > there resolved for Python 3.5 - if other folks hope for the same > thing, then one of the best ways to help that happen is to try out the > win_unicode_console module and provide feedback on what does and > doesn't work. This looks very cool, and I plan on giving it a try. But I don't see any issues mentioned there (unless you mean the fact that it's not possible to hook into Python's interactive interpreter directly, but I don't see how that could be fixed in an external module). There's no open issues on the project's github tracker. I'd love to see this go into 3.5, so any more specific suggestions as to what would be needed to move it forwards would be great. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows Unicode console support [Was: Bytes path support]
On 27 August 2014 09:09, Nick Coghlan wrote: > There are two links to CPython issues from the project description: > > http://bugs.python.org/issue1602 > http://bugs.python.org/issue17620 > > Part of the feedback on those was that as much as possible should be > made available as a third party module before returning to the > question of how to update CPython. OK, ta. The only issues I'm seeing are that it doesn't play well with the interactive interpreter, which is a known problem but unfortunately makes it pretty hard for me to do any significant testing (nearly all of the stuff that I do which prints to the screen is in the REPL, or in IPython which has its own custom interpreter loop). If I come up with anything worth commenting on, I will do so (I assume that comments of the form "+1 me too!" are not needed ;-)) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip enhancement
On 27 August 2014 13:58, Neal Becker wrote: > At least, pip should have the ability to alert the user to potential updates, > > pip update > > could list which packages need updating, and offer to perform the update. I > think this would go a long way to helping with this problem. Do you mean something like "pip list --outdated"? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip enhancement
On 27 August 2014 14:46, Skip Montanaro wrote: > it would be great if there was a way for it to tell me where on my > system it found outdated package X. The --verbose flag tells me all > sorts of other stuff I'm not really interested in, but not the > installed location of the outdated package. There's also packaged environments like conda. It would be nice if pip could distinguish between conda-managed packages and ones I installed myself. Really, though, this is what the PEP 376 "INSTALLER" file was intended for. As far as I know, though, it was never implemented (and you'd also need to persuade the Linux vendors, the conda people, etc, to use it as well if it were to be of any practical use). Agreed about reporting the installed location, though. Specific suggestions like this would be good things to add to the pip issue tracker. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows Unicode console support [Was: Bytes path support]
On 27 August 2014 10:46, Paul Moore wrote: > If I come up with anything worth commenting on, I will do so (I assume > that comments of the form "+1 me too!" are not needed ;-)) Nevertheless, here's a "Me, too". I've just been writing some PyPI interrogation scripts, and it's absolutely awful having to deal with random encoding errors in the output. Being able to just print *anything* is a HUGE benefit. This is how sys.stdout should behave - presumably the Unix guys are now all rolling their eyes and saying "but it does - just use a proper OS" :-) Enlightened-ly y'rs, Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
30 August 2014 03:44, Alex Gaynor wrote: > Discussion points: > > * Disabling verification entirely externally to the program, through a CLI > flag > or environment variable. I'm pretty down on this idea, the problem you hit > is > that it's a pretty blunt instrument to swing, and it's almost impossible to > imagine it not hitting things it shouldn't As a data point, I use --no-check-certificates extensively, in wget, curl and some Python programs which have it, like youtube-dl. The reason I do so is typically because the programs do not use the Windows cerificate store, and configuring a second certificate store on a per-program basis is too much of a pain to be worth it (per-program because the hacks such programs use to get round the fact that Windows has no central location like /etc are inconsistent). The key question for me is therefore, does Python's ssl support use the Windows store directly these days? I checked the docs and couldn't find anything explicitly stating this (but all the terminology is foreign to me, so I may have missed it). If it does, programs like youtube-dl will start to "just work" and I won't have the need for a "switch off everything" flag. If a new Python 3.5 installation on a Windows machine will enforce https cert checking and yet will not check the system store (or, I guess, come with an embedded store, but aren't there maintenance issues with doing that?) then I believe a global "don't check" flag will be needed, as not all programs offer a "don't check certificates" mode. And naive users like me may not even know how to code the behaviour for such an option - and the tone of the debate here leads me to believe that it'll be hard for developers to get unbiased advice on how to switch off checking, so it'll end up being patchily implemented. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 30 August 2014 16:22, Alex Gaynor wrote: > The Windows certificate store is used by ``load_default_certs` Cool, in which case this sounds like a good plan. I have no particular opinion on whether there should be a global Python-level "don't check certificates" option, but I would suggest that the docs include a section explaining how a user can implement a "--no-check-certificates" flag in their program if they want to (with appropriate warnings as to the risks, of course!). Better to explain how to do it properly than to say "you shouldn't do that" and have developers implement awkward or incorrect hacks in spite of the advice. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 17:27, Christian Heimes wrote: > It's very simple to trust a self-signed certificate: just download it > and stuff it into the trust store. "Stuff it into the trust store" is the hard bit, though. I have honestly no idea how to do that. Or if it's temporary (which it likely is) how to manage it - delete it when I no longer need it, list what junk I've added over time, etc. And equally relevantly, no idea how to do that in a way that won't clash with my company's policies... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 18:29, Antoine Pitrou wrote: > If an application has special needs that require trusting a self-signed > certificate, then it should expose a configuration setting to let users > specify the cert's location. I can't see how that would be something the application would know. For example, pip allows me to specify an "alternate cert bundle" but not a single additional cert. So IIUC, I can't use my local index that serves https using a self-signed cert. I'd find it hard to argue that it's pip's responsibility to think of that use case - pretty much any program that interacts with a web service *might* need to interact with a self-signed dummy version, if only under test conditions. Or did you mean that Python should provide such a setting that would cover all applications written in Python? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 19:37, Antoine Pitrou wrote: > Well, it's certainly pip's responsibility more than Python's. What would > Python do? Provide a setting that would blindly add a cert for all uses of > httplib? That's more or less my point, pip doesn't have that much better idea than Python. I was talking about putting the cert in my local cert store, so that *I* can decide, and applications don't need to take special care to allow me to handle this case. You said that doing so was bad, but I don't see why. It seems to me that you're saying that I should raise a feature request for pip instead, which seems unreasonable. Am I missing something? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 21:15, Antoine Pitrou wrote: > What do you call your local cert store? I was referring to Christian's comment > It's very simple to trust a self-signed certificate: just download it and > stuff it into the trust store. >From his recent response, I guess he meant the system store, and he agrees that this is a bad option. OK, that's fair, but: a) Is there really no OS-level personal trust store? I'm thinking of Windows here for my own personal use, but the same question applies elsewhere. b) I doubt my confusion over Christian's response is atypical. Based on what he said, if we hadn't had the subsequent discussion, I would probably have found a way to add a cert to "the store" without understanding the implications. While it's not Python's job to educate users, it would be a shame if its default behaviour led people to make ill-informed decisions. Maybe an SSL HOWTO would be a useful addition to the docs, if anyone feels motivated to write one. Regardless, thanks for the education! Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 23:10, Nick Coghlan wrote: > Assuming sslcustomize was in site-packages rather than the standard library > directories, you would also be able to use virtual environments with an > appropriate sslcustomize module to disable cert checking even if the > application you were running didn't support direct configuration. Would this mean that a malicious package could install a custom sslcustomize.py and so add unwanted certs to the system? I guess we have to assume that installed packages are trusted, but I just wanted to be explicit. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On 31 August 2014 22:38, Victor Stinner wrote: > This case is described as the use case #2 in the PEP, so it is supported. As > written in the PEP, if you want to be notified of the signal, set a signal > handler which raises an exception. For example the default signal handler > for SIGINT raises KeyboardInterrupt. Wait - sigint? Does this mean that (unless the application adds a signal handler) keyboard interrupts will be in effect ignored while in a system call? I'm not sure I like that - I'd rather Ctrl-C always interrupted the program. Specifically, in one-off scripts that *don't* take care to handle all errors appropriately and just show the traceback... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.4): Issue #22295: Adopt 'python -m pip' as the preferred invocation
On 6 September 2014 13:47, Skip Montanaro wrote: > Based on the comment in the second issue, it doesn't appear this will > be resolved until 1.7 at the earliest. The second issue is specific to setuptools, where we have some very unpleasant hacks to deal with the setuptools/distribute mess. Having said that I don't have an immediate feel for what proportion of the issues around --user are fixed in 1.6, as I don't personally use --user much. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 24 September 2014 06:13, Tim Golden wrote: > My only real misgiving here is that, for a few years, we'll need *three* > versions installed to build the active branches: 2008 for 2.7; 2010 for 3.4; > and 2014 for 3.5. Am I wrong? Also, will 2014 express edition be able to fully build extensions for Python, 32 and 64 bit? Will it be able to build Python itself? I ask because I'm currently discovering how much fun (basically none) it is to set up the SDK compilers for building extensions, and it would be fantastic if that pain point could be removed. (It's also personally relevant to me now, as my python-dev provided MSDN subscription has run out, so I'm back to only having access to free tools :-)) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 24 September 2014 14:16, Mike Miller wrote: > It has been a supported option for just shy of 15 years on 2.X... most if > not all the bugs (setuptools) were fixed a decade ago, and right now > thousands, if not millions of people are running it under Program Files > right now. I can vouch for several thousand because a company I work for > distributes Python and pip there for its customers all around the world w/o > issue. One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt. Assuming so, that means that if Python is installed there, the standard "pip install XXX" would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use). I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 24 September 2014 22:29, Steve Dower wrote: >> In my experience pip --user works just fine also. We use it on our unmanned >> media players successfully. > > This is good to know. Maybe we aren't as far away as we think. Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice. Reports like this are great for confirming that it's a viable approach. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 25 September 2014 02:08, Antoine Pitrou wrote: >> Indeed. Moving towards having --user as the norm is definitely >> something we want to look at for pip. One of the biggest concerns is >> how well-exercised the whole user site directory area is in practice. > > What do you mean by well-exercised? Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations. There's probably no way to improve that much other than by making --user the default and waiting for reports of any issues, but feedback like Mike's adds a certain level of confidence that there are no significant problems. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 25 September 2014 16:43, Donald Stufft wrote: > Basically people have Python in a ton of different configurations and it’s > hard to figure out if —user will work out of the box in all of them or not. I guess that "Using the python.org Python installer on Windows" is a limited enough subset that we probably could check that --user worked in that situation. The problem is, how do we implement it? A special case so that pip defaults to --user sometimes, but not others? (I'm strongly against that) Leave the default as not --user and document that Windows users with Python in "Program Files" should always specify --user? (I'm against that because it makes the documentation highly confusing, and we've just done a lot of work to simplify it). Basically, I'd like to hold off moving to "Program Files" as a default until *after* we have enough confidence in user installs that we are willing to switch pip to --user as the default behaviour everywhere. And yes, I'm aware that the first "we" in that was "python-dev" and the second was "PyPA". And that expecting python-dev to wait for PyPA to approve the change may well be unacceptable. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 25 September 2014 17:05, Steve Dower wrote:
> So yes, pip can certainly do this, and if it's already running elevated then
> it shouldn't reprompt, but it's not entirely trivial to get this right ("are
> you denied write access to that directory because you're not admin or because
> it's on read-only media?") and it's considerably easier to try it, fail on
> access issues, but provide a flag for the user to force elevation. "pip
> --sudo install ..." would be fine by me :)
I thought one issue with running an elevated command line subprocess
from a non-elevated one, was that the elevated one didn't have access
to the non-elevated console, so it popped up its own independent
console window, which disappeared immediately the process completed
(hence losing any error messages). I definitely recall easy_install
did that at one stage, and it was a real pain. Or is that something
the parent process can affect, and the cmd/easy_install pair just
didn't do so?
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 25 September 2014 18:13, Steve Dower wrote: > Again, this isn't trivial to get right. The design for the elevation model > seems to have focused mainly on GUI rather than console, probably assuming > that people who need to elevate from the console will elevate the shell > itself (this is a guess - I have no insight into how the Windows team > designed this), so there are limitations we have to work within. Yeah, I think my view is that we don't really want to have to support code that complex, so let's keep it simple. To use pip to install into a protected location, pip must be run from an elevated shell. Given that doing so is inconvenient (specifically, even more inconvenient than sudo on Unix) let's ensure that for the normal case the user doesn't need to do that. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 26 September 2014 01:38, Donald Stufft wrote: > Either way I'm fairly commited to making --user the default, the only > question > on my mind is what exactly does that look like (e.g. does root get --user by > default?) and how we get from where we are now to that point. I think that > raising an error here is fairly unfriendly when we can know with pretty good > certainity what the user wanted (and they can explicitly declare that if > they want too). A couple of points come to mind (although they may be more suitable for distutils-sig). 1. Do user installs "leak" into virtualenvs? If so, then in effect --use-system-packages is switched back on again if --user installs become the norm. Which is almost certainly not what is wanted. 2. pip install should default to not being --user when run from within a virtualenv (same logic as the isolated Python case, but much more important that behaviour remains as now, because the whole *point* of virtualenvs is to isolate). Note that both of these points apply both to venv and virtualenvs. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 26 September 2014 14:31, Donald Stufft wrote: > Yea, I think we throw an error when you use —user inside a virtual > environment. So if --user became the default, what would happen? I'd like pip inside a virtualenv to install into the environment without needing a --system flag. Is that the intention? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7
On 26 September 2014 19:01, Steve Dower wrote: > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). > We've produced this package to help library developers build wheels for > Windows, but also to help users unblock themselves when they need to build C > extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: > http://aka.ms/vcpython27 That is fantastic news! Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On 27 September 2014 06:08, Terry Reedy wrote: > Pip on Windows should act like a normal Windows program. If I install > Python for all users, I expect pipped packages to be installed for all users > too, unless I specify otherwise. If installation (for all users) requires > admin privileges, I expect a UAC box to pop up and ask for the admin > password. This is pretty routine, at least with Win7. Most every program I > install does this either on installation or on first running. Some Windows > operations also pop up a box. There are only a few things that require that > I actually login as an admin user. The main problem is that there is little or no prior art on Windows for *console* programs that require elevation. Those few that do need it require you to start the program from an elevated prompt, but that's not a good user experience. But having said that, I agree with your point, the UAC changes are designed specifically to get people used to the "think about what you're doing" approach, and we should be doing the same. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7
On 27 September 2014 14:01, Nick Coghlan wrote: > I personally believe we should treat handling both this and the SDK > compilers properly as a platform-enablement bug for distutils and > ensure they work properly with the currently maintained branches > (including 2.7). +1 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7
On 26 September 2014 19:01, Steve Dower wrote:
> You can install the package without requiring administrative privileges and,
> with the latest version of setuptools (when it releases), use tools such as
> pip, wheel, or a setup.py file to produce binaries on Windows.
>From the setuptools changelog, it seems that setuptools 6.0+ should
cover this. However, I just got the following error:
>.\ve-2.7\Scripts\pip.exe wheel blist
Downloading/unpacking blist
Running setup.py
(path:C:\Work\Projects\buildwheel\ve-2.7\build\blist\setup.py)
egg_info for package blist
warning: no files found matching 'blist.rst'
Building wheels for collected packages: blist
Running setup.py bdist_wheel for blist
Destination directory: c:\work\projects\buildwheel\wheelhouse
Complete output from command
C:\Work\Projects\buildwheel\ve-2.7\Scripts\python.exe -c "import
setuptools;__file__='C:\\Work\\Projects\\buildwheel\\ve-2.7\\build\\blist\\setup.py';exec(compile(open(__file__).read().replace('\r\n',
'\n'), __file__, 'exec'))" bdist_wheel -d
c:\work\projects\buildwheel\wheelhouse:
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-2.7
creating build\lib.win-amd64-2.7\blist
copying blist\_btuple.py -> build\lib.win-amd64-2.7\blist
copying blist\_sorteddict.py -> build\lib.win-amd64-2.7\blist
copying blist\_sortedlist.py -> build\lib.win-amd64-2.7\blist
copying blist\__init__.py -> build\lib.win-amd64-2.7\blist
running build_ext
building 'blist._blist' extension
Traceback (most recent call last):
File "", line 1, in
File "C:\Work\Projects\buildwheel\ve-2.7\build\blist\setup.py", line
52, in
long_description=open('README.rst').read()
File "C:\Apps\Python27\Lib\distutils\core.py", line 152, in setup
dist.run_commands()
File "C:\Apps\Python27\Lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
File "C:\Apps\Python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File
"C:\Work\Projects\buildwheel\ve-2.7\lib\site-packages\wheel\bdist_wheel.py",
line 175, in run
self.run_command('build')
File "C:\Apps\Python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\Apps\Python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Apps\Python27\Lib\distutils\command\build.py", line 127, in run
self.run_command(cmd_name)
File "C:\Apps\Python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\Apps\Python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File
"C:\Work\Projects\buildwheel\ve-2.7\lib\site-packages\setuptools\command\build_ext.py",
line 54, in run
_build_ext.run(self)
File "C:\Apps\Python27\Lib\distutils\command\build_ext.py", line 337, in run
self.build_extensions()
File "C:\Apps\Python27\Lib\distutils\command\build_ext.py", line
446, in build_extensions
self.build_extension(ext)
File
"C:\Work\Projects\buildwheel\ve-2.7\lib\site-packages\setuptools\command\build_ext.py",
line 187, in build_extension
_build_ext.build_extension(self, ext)
File "C:\Apps\Python27\Lib\distutils\command\build_ext.py", line
496, in build_extension
depends=ext.depends)
File "C:\Apps\Python27\Lib\distutils\msvc9compiler.py", line 473, in compile
self.initialize()
File "C:\Apps\Python27\Lib\distutils\msvc9compiler.py", line 383, in
initialize
vc_env = query_vcvarsall(VERSION, plat_spec)
File
"C:\Work\Projects\buildwheel\ve-2.7\lib\site-packages\setuptools\msvc9_support.py",
line 52, in query_vcvarsall
return unpatched['query_vcvarsall'](version, *args, **kwargs)
File "C:\Apps\Python27\Lib\distutils\msvc9compiler.py", line 299, in
query_vcvarsall
raise ValueError(str(list(result.keys(
ValueError: [u'path', u'include', u'lib']
Failed building wheel for blist
Failed to build blist
Cleaning up...
PS 15:36 [00:02] C:\Work\Projects\buildwheel
>.\ve-2.7\Scripts\pip.exe list
pip (1.5.6)
setuptools (6.0.1)
wheel (0.24.0)
This looks suspiciously like http://bugs.python.org/issue7511, which
is worrying. Is this not something I should have expected setuptools
to have patched?
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 10 October 2014 01:29, Victor Stinner wrote: > What about the Python stable ABI? Would it be broken if we use a > different compiler? > > What about third party Python extensions? > > What about external dependencies like gzip, bz2, Tk, Tcl, OpenSSL, etc.? The key point for me is that any supported build on Windows supports the exact same ABI. It is difficult for Windows users to set up a build environment (and changing the compiler will not alter that fact) so Windows users will rely on binary builds. If multiple ABIs exist, users will have the problem of projects shipping only one ABI binary, and if it doesn't match their Python, they are out of luck. It's critical that we don't double the number of binary builds projects need to ship. Having said that, I'm personally not interested in this, as I am happy with MSVC Express. Python 3.5 will be using MSVC 14, where the express edition supports both 32 and 64 bit. The licensing doesn't bother me personally, and the compiler is easy to install for people who want it. Any competing build environment would have to be as easy to install and use as MSVC Express to be worthwhile, IMO. The only advantage[1] to a new compiler would be if it trivially supported cross-compiling on Linux, as that would allow Limux developers to easily ship Windows binaries. Paul [1] I am not commenting on philosophical advantages like licensing here. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 10 October 2014 10:50, Victor Stinner wrote: > Is MinGW fully compatible with MSVS ABI? I read that it reuses the > MSVCRT, but I don't know if it's enough. I guess that a full ABI > compatibility means more than just using the C library, calling > convention and much more. MinGW can be made to build ABI-compatible extensions. Whether this will continue with MSVC 15 I don't know, as it requires a change to add an interface library for the relevant msvcrXX runtime. And the MinGW community is somewhat fragmented these days, with the core project not supporting 64-bit and various external projects doing so. Having said all this, it *is* possible with some effort to use MinGW to build Python extensions. As noted, the numpy developers have done a lot of work on this as some of the libraries they need must be built with mingw. And the state of distutils support for mingw is very sad, as well, IIRC (last time I looked there were a number of open bugs, and very little movement on them). Rather than put effort into more build options for CPython, I think it would be much more beneficial to the Windows community if effort was put into: 1. Improving support for compiling *extensions* with mingw and its 64-bit cousins, in a way that is compatible with a standard MSVC-built Python. This would involve sorting through and applying some of the distutils changes, as well as working with the mingw projects to ensure they will offer support for the MSVC15 runtime ina timely manner. 2. Looking at ways to support cross-compiling Windows extensions from Linux using mingw. I've no idea how practical this would be, but if Linux developers could provide Windows builds without having to maintain a Windows environment, that would be great. As I say, I have no personal interest here, as I use MSVC, but I know the above two options are commonly asked for. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 10 October 2014 15:36, Tres Seaver wrote: > On 10/10/2014 05:26 AM, Larry Hastings wrote: >> IMO the benefit from supporting other compilers on Windows is >> negligible > > Did you miss the OP's point that OpenBLAS cannot be compiled with MSVC, > raising the priority of mingw-buildable extensions for numerical work? The proposal here is to make it possible to build *python* with a different compiler. Being able to compile extensions with mingw is independent. It used to be relatively easy to build extensions with mingw, although Python's support for that has bitrotted to the point where it's not worth it unless you have a strong need (which is why the SciPy folks are working on it). In my view, anyone capable of modifying the Python build process to support other compilers would also be more than capable of improving the distutils support for building extensions with mingw in a way that is compatible with the current MSVC-built Python. While I can't dictate where anyone else chooses to spend their time, it seems to me that working on allowing non-MSVC builds of Python is a sad waste of limited resources while the mingw extension building issues remain. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 10 October 2014 17:28, Mark Lawrence wrote: > There are 55 open issues on the bug tracker with mingw in the title. It's not easy to tell, but on a spot check a fair proportion of them seem to be about distutils/extension builds. And a lot of the rest are related to http://bugs.python.org/issue3871 which is a rejected issue about adding build support for mingw. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 11 October 2014 19:32, Nathaniel Smith wrote: > The bigger problem is that getting a usable DLL at all is a serious > challenge. Some of the issues we deal with: (a) the classic, stable mingw > has no 64-bit support, (b) the only portable way to compile fortran (f2c) > only works for the ancient fortran 77, (c) getting even mingw-w64 to use a > msvc-compatible ABI is not trivial (I have no idea why this is the case, but > it is), (d) mingw-built dlls normally depend on the mingw runtime dlls. > Because these aren't shipped globally with Python, they have to be either > linked statically or else a separate copy of them has to be placed into > every directory that contains any mingw-compiled extension module. These are all genuine and difficult issues. I have some knowledge of the mingw environment, although only as a user, and I can confirm that it is not in an ideal state. As you mention, the core project only offers 32-bit compilers, and the 64-bit versions are separate, not always reliable, projects. It can be the case that getting a successful build relies on using a precise distributed archive of the relevant mingw binaries. By default, mingw uses msvcrt, for essentially licensing reasons (to do with the fact that msvcrt is the only version they can consider as being "shipped with the OS" which is relevant to their use of the GPL). To get it to link to an alternative VC runtime requires a symbol library for that runtime, which needs someone to build it. I don't know whether mingw includes runtime symbol libraries for later than msvcr10[1] - I asked for that to be added when Python switched to VC10, and it was fairly difficult to get that done, even at that point when the mingw community was much less fragmented than now. It is possible to build C extensions with mingw in such a way that they don't depend on the mingw runtime DLLs - at least the distutils support made that happen for the average extension when I last worked on it which was pre-Python 3... I'm pretty sure that C++ needs runtime support DLLs which would be tricky to avoid, and I have no idea what sorts of difficulty Fortran might add (although your comments sound about what I expect). > All the runtime and ABI issues do mean that it would be much easier to use > mingw(-w64) to build extension modules if Python itself were built with > mingw(-w64). Obviously this would in turn make it harder to build extensions > with MSVC, though, which would be a huge transition. I don't know whether > gcc's advantages (support for more modern C, better cross-platform > compatibility, better accessibility to non-windows-experts, etc.) would > outweigh the transition and other costs. As mentioned above, I don't think the mingw environment is that reliable, which would be an issue if Python were built with it. Would it really be a positive step if we had to say "to build Python you need to download this precise personal build from a specific mingw64 spin-off project"? And yes, I have code for which that is *precisely* what I need to do. > As an intermediate step, there are almost certainly things that could be > done to make it easier to use mingw-w64 to build python extensions, e.g. > teaching setuptools about how to handle the ABI issues. Maybe it would even > be possible to ship the mingw runtimes in some globally available location. As I've said a number of times now, I think this is much more likely to be a useful avenue. For example, shipping the appropriate libmsvcrXXX.a and static libraries for the relevant runtimes with distutils, instead of relying on the user having a version of mingw that does so. And testing (and fixing if necessary) the distutils MingwCompiler class with a wider range of mingw builds. Note that where I say distutils here, it would ideally be something that we do in setuptools, so that it won't be tied to the stdlib release cycles. But AFAIK, setuptools doesn't yet include any compiler classes, so that'd be a bigger change. I have no idea what the most appropriate direction to take would be here. By the way, Steve Dower - distutils\cygwinccompiler will need the new MSVC runtime added to get_msvcr() for 3.5/VC15. It won't help unless/until mingw ships the runtime symbol library, of course, but if it's not added to the shipped Python 3.5, it'll be a pain to add later... It doesn't seem to be in your VC14 branch. Paul [1] I just checked and it seems that there's a msvcr110 library shipped with the mingw I have, but nothing later. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 25 October 2014 21:50, Steve Dower wrote: > Ray Donnelly wrote: >> What is it that you >> are afraid of if CPython can be compiled out of the box using >> mingw/MinGW-w64? Why are you fighting so hard against having option. > > I'm afraid of users having numpy crash because they're using an MSVC > CPython instead of a mingw CPython. I'm afraid of users not being able > to use library A and library B at the same time because A requires MSVC > CPython and B requires mingw CPython. (I can produce more examples > if you like, but the general concern is having a fragmented community, > as I said in my previous post.) Precisely. Either developers test with *all* supported compilers, or there is a risk of incompatibilities. Advocates of supporting mingw as a build system for Python generally do *not* suggest that they are willing to test for, and deal with, cross-version compatibility issues. Typically mingw is seen as "another platform" in some sense, by such advocates, having its own set of supporters and maintainers. The possibility of extensions built with a mingw-compiled Python failing when used under a MSVC-built Python is real. It's difficult to say how big that risk is, but it's certainly there. And I see no-one offering to be responsible for such compatibility issues (the mingw supporters generally don't want to set up a MSVC build chain, so it's difficult to see how they could credibly offer). > I'm fighting against "having options" because it will suck up the precious > volunteer time we have and direct it away from where it would be more > useful, which is making it easier to build extensions with other compilers. And claiming that it doesn't suck up developer time ignores the point I made above - *someone* has to deal with any compatibility issues that come up. As a starter, does the wheel format need to include tags to distinguish whether the target Python is MSVC-built and mingw-built? Who will check that? If there is a need, who will work on the code needed to incorporate that change into wheel, pip, and the relevant PEPs? As Steve says, the Python community has a genuine, strong need for people with mingw expertise working on making it easier to build *extensions* using mingw, that work with a MSVC-built CPython. If it were possible to cross-compile compatible extensions on Linux, projects developed on Linux could supply Windows binaries much more easily, which would be a huge benefit to the whole Windows Python community. But the mingw experts don't want to work on that, preferring to develop patches allowing CPython to be built with mingw. No objection from me, it's your free time, use it as you wish, but as a Windows user of Python I can confirm that it's not what I'd like you to be doing as your contribution to Python. > I would love to see extensions for Windows built on all platforms. I see no > value in building Python itself for Windows from different platforms. Exactly. > If other core developers agree with you that a more "pure" build of Python is > worthwhile, then they can go ahead and merge the patches (though I suspect > most would like to see some traction achieved on a fork first). I think it's > important that I (as Windows build manager) make my own position clear, > that's all. Personally, I'm not a core developer, just a long-time member of this list and occasional contributor to discussions. But I am also a core pip developer and a Windows user, and from that perspective I am acutely aware of the common pain points for Python users on Windows. And they are all about binary extensions, and none at all about building Python itself. So in my view, being able to build CPython using mingw is somewhat interesting from a theoretical perspective, but of little or no practical value[1] and disruptive in a number of ways, as mentioned above, to improving the overall experience of Python users on Windows. Paul [1] I note, without trying to make a judgement, that many of the benefits cited for building with mingw are around "being able to use free tools" or similar essentially ideological issues. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 25 October 2014 23:22, Chris Angelico wrote: > On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou wrote: >> My point is that your "Windows build" would not have the same behaviour >> as a MSVC-produced Windows build, and so testing it with it would not >> certify that your code would actually be compatible with genuine >> MSVC builds of CPython, which we will not stop supporting. >> > > So you're saying it's impossible to support two compilers? No, rather that Windows currently only has a single supported compiler (excluding cygwin, which is essentially a different OS). Adding a second compiler doesn't just involve adding support for it - which is all that the people offering mingw patches are doing - but also involves going through the whole CPython ecosystem locating the places where there is an implicit assumption that "all Windows builds use the same compiler" and fixing them. I've already pointed out where this is a question for pip and wheel. Whoever wants to add support for a second compiler needs to be willing to do this part of the job as well. Handwaving arguments that "it's binary compatible" aren't enough. Prove it. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 26 October 2014 13:12, Tony Kelman wrote: > Only cross-compilation and the build system in the above list are relevant > to CPython, but I hope I have convinced you, Paul Moore, etc. that there are > real reasons for some groups of users and developers to prefer MinGW-w64 > over MSVC. Not really, to be honest. I still don't understand why anyone not directly involved in CPython development would need to build their own Python executable on Windows. Can you explain a single specific situation where installing and using the python.org executable is not possible (on the assumption that the mingw build is functionally identical and ABI compatible with the CPython build, the claim being made here)? Note that "not possible" is different from "I don't want to" or "it doesn't fit my views about free software" or similar. Also note that building extensions is different - you have to assume that building extensions using mingw with a mingw-built CPython is just as hard as building them with a MSVC-built CPython (otherwise you've made changes to extension building and you should contribute them independently so that everyone can benefit, not just those who build their own Python with mingw!) > Paul Moore: >> If it were possible to cross-compile compatible extensions on Linux, >> projects developed on Linux could supply Windows binaries much more >> easily, which would be a huge benefit to the whole Windows Python >> community. > > I want to do exactly this in an automated repeatable way, preferably on > a build service. This seems harder to do when CPython cannot itself be > built and handled as a dependency by that same automated, repeatable > build service. I cannot see why you would need to build Python in order to build extensions. After all, if your build service is on Linux, it couldn't run a mingw-built Python anyway. If your build service is a Windows machine, just install the python.org binaries (which is a simple download and install, that can be fully automated, but which is a one-off process anyway). > Unless it becomes possible to cross-compile extensions > using the build machine's own version of Python, which might be the right > approach. This may be where we are getting confused. I see this as the only practical way of cross-compiling Windows extensions on Linux, by using the Linux Python. So being able to cross-compile Python is not relevant. On a tangential note, any work on supporting mingw builds and cross-compilation should probably be done using setuptools, so that it is external to the CPython code. That way (a) it isn't constrained by the CPython release schedules and backward compatibility constraints, and (b) it can be used in older versions of Python (which is pretty much essential if it's to be useful, TBH). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 26 October 2014 17:59, Tony Kelman wrote: > Ensuring compatibility with CPython's > chosen msvcrt has made that work even more difficult for them. Ensuring compatibility with CPython's msvcrt is mandatory unless you want to create a split in the community over which extensions work with which builds. That's precisely the scenario Steve Dower and myself (among others) fear, and want to avoid at all cost. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 26 October 2014 14:28, Ray Donnelly wrote: > I like this idea. To reduce the workload, we should probably pick > Python3 (at least initially)? Aren't the existing patches on the tracker already for Python 3.5+? They should be, as that's the only version that's likely to be a possible target (unless you get someone to agree to allow a change like this as in scope for Pythhon 2.7, which I've seen no indication of). Maybe I'm confused here. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 26 October 2014 23:24, Tony Kelman wrote: > I want, and in many places *need*, an all-MinGW stack. OK, I'm willing to accept that statement. But I don't understand it, and I don't think you've explained why you *need* your CPython interpreter to be compiled with mingw (as opposed to a number of other things you might need around building extensions). You may well "need" a mingw-compiled CPython because no-one has yet fixed the issues around using mingw to build extensions for the python.org python build. But that's my point - I'd rather "they" fixed that issue, rather than perpetuating your need for a non-standard compiler that uses extensions no-one else can use. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 26 October 2014 23:11, Ray Donnelly wrote: > I don't know where this "ABI compatible" thing came into being; Simple. If a mingw-built CPython doesn't work with the same extensions as a MSVC-built CPython, then the community gets fragmented (because you can only use the extensions built for your stack). Assuming numpy needs mingw and ultimately only gets built for a mingw-compiled Python (because the issues building for MSVC-built Python are too hard) and assuming that nobody wants to make the effort to build pywin32 under mingw, then what does someone who needs both numpy and pywin32 do? Avoiding that issue is what I mean by ABI-compatible. (And that's all I mean by it, nothing more subtle or controversial). I view it as critical (because availability of binaries is *already* enough of a problem in the Windows world, without making it worse) that we avoid this sort of fragmentation. I'm not seeing an acknowledgement from the mingw side that they agree. That's my concern. If we both agree, there's nothing to argue about. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 27 October 2014 12:30, Nick Coghlan wrote: >> OK, I'm willing to accept that statement. But I don't understand it, >> and I don't think you've explained why you *need* your CPython >> interpreter to be compiled with mingw (as opposed to a number of other >> things you might need around building extensions). > > I can take a go at an explanation that may make more sense to you. > Consider one of our key requirements for packaging applications for > Fedora: that Fedora builds be *self-hosting*. Given a base Fedora > system, and a source RPM, we need to be able to *build the binary RPM > from source*. (Other Linux distros generally have a similar > requirement) > > Relying on opaque binary blobs downloaded from the internet as part of > the build process is not permitted (modulo a few exceptions for > firmware blobs in device drivers). > > Now consider that this "automatically rebuild the entire system from > source" model is not unique to Linux - you can use it for any system > where your build process is sufficiently automated, and you have a > need for it. However, the *structure* of those kind of automation > tends to differ wildly between POSIX style tooling (gcc, clang) and > MSVC. If you have an existing build automation system for *nix > targets, then cross-compilation via MinGW is likely going to be your > smoothest path to adding Windows binary support. > > At that point, if CPython is one of your dependencies, you're going to > have the choice of allowing the python.org binaries to be pulled in as > opaque pre-built blobs, or else figuring out how to build an ABI > compatible version with MinGW rather than with MSVC. Think of this > more in the case of *embedding* the CPython runtime in a larger > context (e.g. in Tony's case, to make Python software usable with the > Julia runtime), rather than in building a standalone Python > interpreter for general use. > > So, for embedding cases, and for incorporation into POSIX-style build > systems using MinGW-w64 for cross-compilation of Windows binaries, it > may make sense to incorporate the patches that allow building with > MinGW-w64 into mainline CPython (if I recall correctly, we supported > building with Intel's C compiler for a long time, even though we never > shipped anything built with it). Thanks Nick. That explanation makes sense to me. I was aware of this sort of scenario, and as I've said before I don't have any objection per se to making things easier for people with that sort of requirement. But some of the other arguments in this thread seemed to imply more than that. Without specifics, though, I concede that I may be over-interpreting the rhetoric, so that's the part of the debate I'm stepping back from, to avoid descending into FUD. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 26 October 2014 01:05, Ray Donnelly wrote: > Download and run: > http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download Sending this offline because I really don't want to start up another extended debate, but is there a version of this that I can use that ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
Please ignore this. I hit the wrong button. On 27 October 2014 14:18, Paul Moore wrote: > On 26 October 2014 01:05, Ray Donnelly wrote: >> Download and run: >> http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download > > Sending this offline because I really don't want to start up another > extended debate, but is there a version of this that I can use that ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 26 October 2014 23:44, Paul Moore wrote: > On 26 October 2014 23:11, Ray Donnelly wrote: >> I don't know where this "ABI compatible" thing came into being; > > Simple. If a mingw-built CPython doesn't work with the same extensions > as a MSVC-built CPython, then the community gets fragmented (because > you can only use the extensions built for your stack). Assuming numpy > needs mingw and ultimately only gets built for a mingw-compiled Python > (because the issues building for MSVC-built Python are too hard) and > assuming that nobody wants to make the effort to build pywin32 under > mingw, then what does someone who needs both numpy and pywin32 do? > > Avoiding that issue is what I mean by ABI-compatible. (And that's all > I mean by it, nothing more subtle or controversial). > > I view it as critical (because availability of binaries is *already* > enough of a problem in the Windows world, without making it worse) > that we avoid this sort of fragmentation. I'm not seeing an > acknowledgement from the mingw side that they agree. That's my > concern. If we both agree, there's nothing to argue about. I have just done some experiments with building CPython extensions with mingw-w64. Thanks to Ray for helping me set this up. The bad news is that the support added to the old 32-bit mingw to support linking to alternative C runtime libraries (specifically -lmsvcr100) has bitrotted, and no longer functions correctly in mingw-w64. As a result, not only can mingw-w64 not build extensions that are compatible with python.org Python, it can't build extensions that function at all [1]. They link incompatibly to *both* msvcrt and msvcr100. This is a bug in mingw-w64. I have reported it to Ray, who's passed it onto one of the mingw-w64 developers. But as things stand, mingw builds will definitely produce binary extensions that aren't compatible with python.org Python. Paul [1] Note, that's if you just use --compiler=mingw32 as supported by distutils. Looking at how the numpy folks build, they seem to hack their own version of the distutils C compiler classes. I don't know whether that's just to work around this bug, or whether they do it for other reasons as well (but I suspect the latter). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 27 October 2014 18:47, Case Van Horsen wrote: > I've managed to build gmpy2 (which requires GMP, MPFR, and MPC > libraries) using msys2. I've detailed the steps (hacking) at: > > https://code.google.com/p/gmpy/source/browse/trunk/msys2_build.txt Thanks for this. I don't have the time to read your notes right now, but I will do so. > One of the hacks I made addresses the linking bug. The extension > does run with the both the 32-bit and 64-bit versions of CPython 2.7, > 3.2, 3.3, and 3.4. Did you report the linking bug to the mingw-w64 project? They key thing here is that without gcc -lmsvcrt100 foo.c working (i.e., not resulting in linking with msvcrt), building Python extensions will always need hacks to workaround that bug. > It is possible, just not easy. Anything that makes is easier would > be very helpful. With the bug fixed, the steps should be as trivial as: 1. Using python.org Python, with gcc on your PATH. 2. Install any dependencies (e.g., gmp) where gcc can see them. 3. python setup.py build_ext --compiler=mingw32 bdist_wheel (or whatever setup.py invocation suits you, as long as you set compiler=mingw32). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 27 October 2014 20:45, Greg Ewing wrote: > Nick Coghlan wrote: >> >> That assumption will allow MinGW-w64 to link with the appropriate >> MSVCRT versions for extention building without anything breaking. > > > If that works, then the same technique should allow CPython > itself to be built in a VS-compatible way with mingw, > shouldn't it? Yes. > Those objecting to a mingw-built python seem to be assuming > that such a thing will necessarily be incompatible with > VS builds, but I don't see why that has to be the case. No, we've been trying to establish whether the patches to build with mingw were intended to produce such a compatible build. It's not clear, but so far it seems that apparently that is *not* the intent (and worse, mingw-w64 may not even be able to build viable executables that link with msvcr100 without some heavy hacking, although that's still somewhat unclear). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 27 October 2014 21:19, Steve Dower wrote: >> No, we've been trying to establish whether the patches to build with mingw >> were >> intended to produce such a compatible build. It's not clear, but so far it >> seems >> that apparently that is *not* the intent (and worse, mingw-w64 may not even >> be >> able to build viable executables that link with msvcr100 without some heavy >> hacking, although that's still somewhat unclear). > > Unless there is also opposition to moving to VC14, I'd rather see the mingw > projects invest in linking to those libraries. I believe they'll have a much > easier > time of it than worrying about VC10, and the investment will be worth more in > the future as the public API of the CRT stops changing. I think the point is that anything other than msvcrt is extra work, because using msvcrt is coded into the guts of gcc (which in turn is because msvcrt is apparently OK to consider as "part of the OS" in GPL legality terms). So whether it's the vc10 libraries or the vc14 ones is irrelevant - and mingw ships with the vc10 link library, so it's easier to discuss the problem in terms of vc10. But yes, vc14 would be the long term target. Of course if the vc14 libs were deemed as "shipped with the OS" and/or were named msvcrt.dll, then that would be different. But I assume that's not what will happen. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 28 October 2014 14:46, Tony Kelman wrote: > Patches should be done well and accompanied with proper documentation > so new functionality is fully reproducible. If that's what's holding > up review, comments in the bug threads indicating as much would be > helpful. Typically that tends to be expressed as a terse "I can't get this working". That's not ideal, but is an indication. When the response is "but it's easy" (as it sometimes is) communication degenerates quite fast. There's an example here in this thread - it took me a *long* time, with help from a couple of people, to work out how to get a mingw toolchain I could use to try things out. Even though the premise of the whole discussion was "it's easy to set up a mingw toolchain"... > In my opinion the MSVC toolchain makes that problem worse, as it's far > harder for unix developers to have any familiarity with how things work. > But you do have involvement and core developers from Microsoft which is > obviously incredibly important. Maybe even mandatory for Python on Windows > to be viable in your eyes. One problem I've seen a lot on other projects is that when Unix developers set up a comfortable, Unix-like environment on Windows using something like mingw, they frequently aren't aware of crucial differences between Unix and Windows and tend to write software that even though it's Windows-native, *feels* Unixy to Windows users (don't ask me to be more specific :-)). That's always going to happen, and the people with Windows experience have to take up the slack by keeping an eye out for such things, but setting the bar marginally higher, to "you have to at least be willing to download and use a native Windows compiler" can at least remind said Unix developers that they are in a different environment. That's *not* a criticism of anyone in the Python community, btw. Typically the experience of Windows users is very well respected by python core and package developers. But I tend to think that's partly *because* we didn't take the "Unix-like toolchain" approach by default. > This is not the case at all in the scientific community. NumPy and SciPy > put in a lot of extra work to come up with something that is compatible > with the MSVC build of CPython because they have to, not because they're > "happy to" jump through the necessary hoops. The situation today with > NumPy AIUI is they already have to build with a custom hacked MinGW > toolchain that only one person knows how to use. Evidently until very > recently they were using a many-years-old 32-bit-only build of GCC 3.x. > Do python-dev and numpy-discussion not talk to one another? I get that > not everyone uses numpy, or Windows, but it sounds like there's very > little understanding, or even awareness, of the issues they face. Sadly, no. The numpy developers have always been a pretty much separate world. We're seeing a bit more interaction these days, but it's very limited and far from the level that's needed. The fault (if that's the right word) probably lies on both sides. It's certainly not purely the responsibility of the core team to build communication links. As this thread has demonstrated, python-dev (and distutils-sig, which is another place that desparately needs more numpy interaction) is not the most welcoming of places for someone who is 100% focused on the scientific stack, but conversely the scientific types do tend to do their own thing, and sometimes when they do surface, they can dive in with little appreciation of the wider picture. But we can get along, and we can make progress (albeit not always as fast as either party might like). If all this thread has done is raise awareness of each others' concerns, it's still been a win IMO. > I'm going to move the "extensions with MinGW-w64" part of this conversation > over to numpy-discussion, since they have a need for this today and are > already using workarounds and processes that I don't think anyone is fully > satisfied with. I do find this combination interesting, worth working on, > and possible to make work well, but not completely in isolation as it does > not address my embedding use case. Please keep distutils-sig in the loop as well. I can't promise we'll be able to help, but we should at least make sure we're not hindering you, and we can make you aware of when your solutions won't work outside your area of interest. Now the door is open, let's not close it again. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 29 October 2014 15:31, Nathaniel Smith wrote: >> You can use Express editions of Visual Studio. > > IIUC, the express edition compilers are 32-bit only, and what you actually > want are the "SDK compilers": > https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows > > These are freely downloadable by anyone, no msdn subscription required, but > only if you know where to find them! > > AFAICT the main obstacle to using MSVC to build python extensions (assuming > it can handle your code at all) is not anything technical, but rather that > there's no clear and correct tutorial on how to do it, and lots of confusion > and misinformation circulating. Would it help if I wrote a document explaining how to set up the MS compilers (free and paid for) to allow building of Python extensions? There are a few provisos: 1. A lot of it will be pretty trivial: "If you have Vistal Studio (full edition), install it. Done." 2. It will be out of date very fast as the situation for Python 3.5+ will be trivial across the board. 3. I don't have anywhere particularly authoritative to host it (maybe the Python Wiki?) and it could easily get lost in the huge swamp of variously outdated, over-complicated, or otherwise alternative documents available. Ideally I'd like someone to suggest an "official" location I could use. I don't want to do this if it won't be useful, as it'll take me a bit of effort to confirm the process for the only non-trivial scenario (64-bit Python 3.3/3.4 with free tools). But if people think it would help, that's fine, I volunteer. Paul PS Even if I don't get positive feedback, I may just say "to heck with it" and do it anyway, because it *is* so trivial :-) I just won't promise. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 29 October 2014 20:26, Donald Stufft wrote: > This sounds like something good for packaging.python.org Yeah, I wondered about that. I'll work up a patch for that. But the more I think about it, it really is trivial: - For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works. - For Python 3.5+, install the current Visual Studion Express and everything just works. (I propose to deem Python 2.7 without setuptools as "unsupported") The only things I can see that need expansion are: 1. The precise versions to use (trivial to add, I'm just to lazy to check right now). 2. Where to get VS 2010 Express (as it's no longer supported or available from MS). 3. How to set up the SDK environment for 64-bit Python 3.2-3.4. Before I do a write-up, I want to set up clean VMs with these configurations, so that I can confirm the details. But I don't expect any problems, as I've done them all before. Paul. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 29 October 2014 22:19, Ethan Furman wrote:
>> Yeah, I wondered about that. I'll work up a patch for that. But the
>> more I think about it, it really is trivial:
>
> I am reminded of an interview question I was once asked which was prefaced
> with: "Here's an easy one..."
>
> My reply was, if you know the answer, it's easy!
Yeah, I know what you mean. My take on this is that I agree it's not
easy if you don't know and can't get access to the information, but if
you can, there's very little to it.
>> - For non-free MSVC, install the appropriate version, and everything just
>> works.
>> - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
>> package and everything just works as long as you're using setuptools.
>> - For 32 bit Python 3.2-3.4, install Visual Studio Express and
>> everything just works.
>> - For 64 bit Python 3.2-3.4, install the SDK, set some environment
>> variables, and everything just works.
>> - For Python 3.5+, install the current Visual Studion Express and
>> everything just works.
>
>
> I would suggest
> - where to get these packages
Conceded. Working out how to point people at stuff on MSDN is a
challenge, things seem to move around. Maybe Steve Dower could help
here with canonical URLs for some of them (IIRC, he provided one for
the VS compilers for Python 2.7 package when it was announced).
For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)
And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly.
> - where to get any dependencies
There are none. I could state that explicitly, I guess.
> - any options to [not] specify during install
I'll have to go through the installs again just to be sure, but I'm
pretty certain "Select the default for everything" is correct.
> - what environment variables to what value
None, except for the SDK which I did say I needed to test out and
cover in more detail.
> - where one should be at when one starts the compile process
I don't understand this. It's just "pip wheel foo" to build a wheel
for foo (which will be downloaded), or "pip wheel ." or "python
setup.py bdist_wheel" as you prefer for a local package. That's
standard distutils/setuptools/pip extension building. I don't propose
to cover that, just how to *set up* the environment.
With the sole exception of the SDK case, once installed, everything
just works everywhere, nothing to set up, no special environment to
configure. Start up a new cmd or powershell console (or use the one
you're already in) and go. Maybe Unix users expect more complexity
because it's not that simple on Unix? But I thought it was - install
the appropriate OS packages and that's it?
> and, of course, a gotchas section for uncommon but frustrating things that
> might go wrong.
Hmm, I see your point here, but I'm not sure what I might include. You
*can* get in a mess [1] but generally I don't as I'm an experienced
Windows user. I also don't want to offer to debug and fix everyone's
problems in getting things set up, so offering to collect and document
"common issues" that I've seen is impractical. Maybe a section
entitled "Common Issues and Their Solutions", with some placeholder
text saying "If you have any issues installing any of the compiler
packages mentioned, please document what went wrong and how to fix it,
and submit a PR!" would do?
> And thanks for doing this!
No problem!
Paul
[1] I once spent a *long* time fighting failed installs of the Windows
SDK. Turns out it was because I was trying to install a 32-bit SDK on
a 64-bit machine (doh!), and the installer really doesn't like that
:-( About all I could say to document that is "Read the instructions
properly" and "I'm sorry that the MS installers fail really badly when
faced with relatively obvious idiot-user errors, but I can't do
anything about it :-("
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 29 October 2014 23:02, Ethan Furman wrote: > On 10/29/2014 03:46 PM, Paul Moore wrote: >> >> On 29 October 2014 22:19, Ethan Furman wrote: >>> >>> >>>- where one should be at when one starts the compile process >> >> >> I don't understand this. It's just "pip wheel foo" to build a wheel >> for foo (which will be downloaded), or "pip wheel ." or "python >> setup.py bdist_wheel" as you prefer for a local package. > > > Hmmm... That looks like it's for installing/compiling somebody else's > package. Is that last command sufficient to prepare one's own wheel for > uploading to PyPI, or there something else to do? Oh, I see what you're thinking. I explicitly *don't* want to get into general packaging issues - they are covered elsewhere. The point here is that if you have the right compiler set up, you can read any generic packaging instructions and it doesn't matter whether there's C code involved. That's it. But yes, "python setup.py bdist_wheel" is how you build a wheel for upload. That's true whether your project includes C extensions or not. I'm also not expecting to explain to people how to build any dependencies (for example, PyYAML depends on the C compiler being able to find libyaml). That *is* harder on Windows than on Linux, because there's no "system location" equivalent to /usr/local/include and /usr/local/lib. But again, I consider this to be outside the scope of the document, because it's specific to the particular project you're building. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 29 October 2014 23:22, Nathaniel Smith wrote:
>> Yeah, I know what you mean. My take on this is that I agree it's not
>> easy if you don't know and can't get access to the information, but if
>> you can, there's very little to it.
>
> That's great, but yeah. In case it helps as a data point, I consider
> myself a reasonably technical guy, probably more than your average
> python package author -- 15 years of free software hacking, designed a
> pretty popular X remote display protocol, helped invent DVCS, have
> written patches to gcc and the kernel, numpy co-maintainer, etc. etc.
Yeah, that counts :-)
> For the last ~12 months I've had an underlined and circled todo item
> saying "make wheels for https://pypi.python.org/pypi/zs/";, and every
> time I've sat down and spent a few hours trying I've ended up utterly
> defeated with a headache. Distutils is spaghetti, and Windows is
> spaghetti (esp. if you've never used it for more than checking email
> on a friend's laptop), and the toolchain setup is spaghetti, and then
> suddenly people are talking about vcvarsall.bats and I don't know what
> all. I honestly would totally benefit from one of those
> talk-your-grandparents-through-it tutorials ("here's a screenshot of
> the dialogue box, and I've drawn an arrow pointing at the 'ok' button.
> You should press the 'ok' button.")
OK. That needs to be fixed. How about this. I'll work with you to get
the wheel build process set up in such a way that you can maintain it,
and we'll document what you needed to know in order to do it. Then I
can put that documentation into the packaging user guide for other
people to work with. It can wait till you have the time, but when you
do, shout and I'll help.
I know nothing about zs, but I just tried building it. MSVC 2010
doesn't support "static inline" (used twice in zs\pycrc-crc64xz.h) but
when I removed the "inline" from that it built fine for me. So it
should be a good example, as there are no particular complexities to
deal with.
- For non-free MSVC, install the appropriate version, and everything just
works.
- For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
package and everything just works as long as you're using setuptools.
- For 32 bit Python 3.2-3.4, install Visual Studio Express and
everything just works.
- For 64 bit Python 3.2-3.4, install the SDK, set some environment
variables, and everything just works.
>
> I think the SDK covers both 32 and 64 bit? If so then leaving visual
> studio out of things entirely is probably simpler.
The SDK is the most complex environment to set up, so avoiding it
unless you need it is a good thing. But maybe it's worth explaining
that if you need it anyway, this is how you set it up to cover some of
the other cases as well.
> I'm also unsure how you even get both 32- and 64-bit versions of
> python to coexist on the same system.
Just install them into different directories. The default is the same
for 32 and 64 bit, so you need to override the default for (at least)
one of them, but that's all.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 29 October 2014 23:49, Steve Dower wrote: > "For the paid versions, I'm going to assume that anyone who paid for a > compiler and doesn't know where their copy is, probably can't be > helped ;-)" > > You could link to visualstudio.com for the trial versions, and maybe to a > page/post about the PSF's grants process if such a page exists. That doesn't help for VS 2010 and VS 2008, which are the relevant ones right now, unfortunately. In thew brave new world of Python 3.5+ that might well do, though. (Although the feedback from Unix users is that they really want a direct link to the exact edition they need - i.e., a permalink to VC++ Express Edition for Python 3.5+. I'm not sure how well the MS website structure does that, though. > "And of course there's the awkward problem that VS 2010 Express is > unsupported and therefore no longer available from *any* official > location. I can't solve that, sadly" > > These are still at > http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs#DownloadFamilies_4, > which is the main download page. Hopefully they don't go away before 3.5, > but I have no control over that unfortunately. Wow! I didn't know that. That's great, solves the biggest issue I had. > The link I posted for 2.7 was aka.ms/vcpython27, which is a redirect that I > own and can update if necessary. Cool. As long as it's stable and reliable, it's exactly what I want. Many thanks, Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OneGet provider for Python
On 15 November 2014 00:12, Vincent Povirk wrote: > My end goal is to be able to package a Python application such that an > end-user on Windows (who doesn't know anything about Python) can > easily install it, without either of us having to think about how all > the dependencies are going to get there. That sounds awesome. > If anyone has questions or concerns about this, please let me know. > Keep in mind that I am not subscribed to the mailing list and will > have to be CC'd. You should probably discuss this on distutils-sig, as that is the main list for anything to do with packaging on Python. As Terry pointed out, you can monitor the lists without subscribing, but it would probably be better to subscribe, as people can tend to forget to cc the original author if a thread gets long. > If anyone has questions about OneGet generally, you should probably > ask them directly (see https://github.com/OneGet/oneget), as I am not > a definitive source of information on the project. Unless I'm misreading their install instructions, OneGet needs Windows 8.1 or Windows Server 2012, as it's based on WMF 5. That would exclude people using Windows 7 or earlier (or even the original Windows 8), which is a big proportion of Windows users. While that's not something you can do much about, it is something that's going to impact the goal of making it easy for Windows users to handle managing Python. > Incidentally, it would be really useful if python.org provided stable > url's that always redirected to the latest .msi installers, for > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on > scraping the web site) for this. https://www.python.org/ftp/python/$ver/python-$ver.msi https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi As far as I know these are stable, supported URLs. Although I don't think they are documented anywhere :-( Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OneGet provider for Python
On 15 November 2014 10:54, Nathaniel Smith wrote: > On 15 Nov 2014 10:10, "Paul Moore" wrote: >> >> > Incidentally, it would be really useful if python.org provided stable >> > url's that always redirected to the latest .msi installers, for >> > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >> > scraping the web site) for this. >> >> https://www.python.org/ftp/python/$ver/python-$ver.msi >> https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi > > Right, but what's the URL for "the latest 2.7.x release" or "the latest > 3.x.x release"? I don't know. As I said, it would be nice if these URLs (and "latest" ones as you mentioned) were documented. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OneGet provider for Python
On 15 November 2014 15:17, Benjamin Peterson wrote: > On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote: >> On 15 Nov 2014 10:10, "Paul Moore" wrote: >> > >> > > Incidentally, it would be really useful if python.org provided stable >> > > url's that always redirected to the latest .msi installers, for >> > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >> > > scraping the web site) for this. >> > >> > https://www.python.org/ftp/python/$ver/python-$ver.msi >> > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi >> >> Right, but what's the URL for "the latest 2.7.x release" or "the latest >> 3.x.x release"? > > The website has an API you know. Um, no. Where can I find out about it? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OneGet provider for Python
On 15 November 2014 15:40, Paul Moore wrote: > On 15 November 2014 15:17, Benjamin Peterson wrote: >> On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote: >>> On 15 Nov 2014 10:10, "Paul Moore" wrote: >>> > >>> > > Incidentally, it would be really useful if python.org provided stable >>> > > url's that always redirected to the latest .msi installers, for >>> > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >>> > > scraping the web site) for this. >>> > >>> > https://www.python.org/ftp/python/$ver/python-$ver.msi >>> > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi >>> >>> Right, but what's the URL for "the latest 2.7.x release" or "the latest >>> 3.x.x release"? >> >> The website has an API you know. > > Um, no. Where can I find out about it? I don't know if this got lost in the other messages in this thread, but *is* there a stable URL for "the latest Python 3.4 MSI for Windows amd64" (or similar)? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OneGet provider for Python
On 17 November 2014 19:23, Ned Deily wrote: > Paul Moore wrote: >> I don't know if this got lost in the other messages in this thread, >> but *is* there a stable URL for "the latest Python 3.4 MSI for Windows >> amd64" (or similar)? > > AFAIK, no, there is no such stable URL that directly downloads the > latest installer(s) for a platform; the closest is probably > https://www.python.org/downloads/windows/ which would require scraping. > I'm not sure we would want to encourage such a thing; I'm happy enough with just the direct links to the exact versions (3.4.1 etc). I have to update my automatic build script whenever a new minor version comes out, which is a bit of a pain but as you say, having to deliberately decide to upgrade the version it installs is not a bad thing. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 21 November 2014 13:53, Chris Angelico wrote: > On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger > wrote: >> Also, the proposal breaks a reasonably useful pattern of calling >> next(subiterator) inside a generator and letting the generator terminate >> when the data stream ends. Here is an example that I have taught for >> years: >> >> def izip(iterable1, iterable2): >> it1 = iter(iterable1) >> it2 = iter(iterable2) >> while True: >> v1 = next(it1) >> v2 = next(it2) >> yield v1, v2 > > Is it obvious to every user that this will consume an element from > it1, then silently terminate if it2 no longer has any content? It is to me, certainly. I'm mostly with Raymond on this. The main exception is that there does seem to be a trend towards more "tricky" uses of this feature, typically around ideas such as a stop() function to break out of generator expressions. I do think this is bad practice and shouldn't be allowed. But given that it's possible to write bad code in plenty of ways, and changing the language "to protect people from themselves" is not a good idea, I don't think the benefits justify the change [1]. Paul [1] Probably simply telling people not to try to cram over-complex code into a genexp would address the worst aspects of the problem... ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 21 November 2014 15:58, Steven D'Aprano wrote: >> > def izip(iterable1, iterable2): >> > it1 = iter(iterable1) >> > it2 = iter(iterable2) >> > while True: >> > v1 = next(it1) >> > v2 = next(it2) >> > yield v1, v2 >> >> Is it obvious to every user that this will consume an element from >> it1, then silently terminate if it2 no longer has any content? > > "Every user"? Of course not. But it should be obvious to those who think > carefully about the specification of zip() and what is available to > implement it. > > zip() can't detect that the second argument is empty except by calling > next(), which it doesn't do until after it has retrieved a value from > the first argument. If it turns out the second argument is empty, what > can it do with that first value? It can't shove it back into the > iterator. It can't return a single value, or pad it with some sentinel > value (that's what izip_longest does). Since zip() is documented as > halting on the shorter argument, it can't raise an exception. So what > other options are there apart from silently consuming the value? Interestingly, although I said "yes, it's obvious", I'd missed this subtlety. But I don't consider it "unexpected" or a "gotcha", just subtle. I certainly don't consider it to be the *wrong* behaviour - on the contrary, I'd be more surprised to get a RuntimeError when the second iterator was shorter. What I understand to be the recommended alternative: def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: try: # Is it OK to cover both statements with one try...except? # I think it should be, but it's a pattern I try to avoid v1 = next(it1) v2 = next(it2) except StopIteration: return yield v1, v2 looks less obvious to me, and obscures the intent a little. (Note that I understand this is only one example and that others present a much more compelling case in favour of the explicit return form) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 23 November 2014 at 15:25, Chris Angelico wrote: > Thank you, it's nice to have a successful one to counterbalance the > "failure" of PEP 463. (Which, incidentally, never actually got a > resolution. It's still showing up as 'Draft' status.) I think it's worth pointing out that both this and PEP 463 were complex an contentious topics, and the discussion in both cases was well-focused and civil. I don't think the fact that you were the author of both PEPs is unrelated to this. Thanks for taking on *both* of these PEPs and handling them so well. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 24 November 2014 at 10:20, Nick Coghlan wrote: > Another aspect that can be somewhat annoying is the terminology > conflict between branches in the git sense and bookmarks vs named > branches in the Mercurial sense. This is probably the thing that hurts me most in git/hg translation. In git, I routinely create branches, play with them, then throw them away (either once they get integrated, or when I decide they are a bad idea). It seems to me to be *much* harder to get that lightweight behaviour in hg - if I create a PR for a hg project that isn't accepted, how do I delete it from my clone? In a way that means it won't reappear when I push/pull from my other PC, where I forgot to *also* delete it? Does the answer to that depend on whether I used a bookmark or a branch (and relatedly, which should I use)? Maybe it's also more complicated than it seems on git, but cleaning up experiments seems much easier to me on git. Not that we're getting offtopic here at all, of course... ;-) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 26 November 2014 at 16:24, Isaac Schwabacher wrote: > This actually leads to a good example of why the PEP is necessary: [...] Oh! If that's the current behaviour, then it probably needs to go into the PEP as a motivating example. It's far more convincing than most of the other arguments I've seen. Just one proviso - is it fixable in contextlib *without* a language change? If so, then it loses a lot of its value. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 26 November 2014 at 17:19, Guido van Rossum wrote: > It's hard to use as an example because the behavior of contextlib is an > integral part of it -- currently for me the example boils down to "there is > a bug in contextlib" Hmm, fair point. I was assuming that the bug in contextlib can't be fixed with the current language behaviour (and I'd personally be OK with the example simply adding a comment "this can't be fixed without changing Python as proposed in the PEP"). But I'm not sure how true that is, so maybe it's not quite as compelling as it seemed to me at first. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On 29 November 2014 at 23:27, Donald Stufft wrote: > In previous years there was concern about how well supported git was on > Windows > in comparison to Mercurial. However git has grown to support Windows as a > first > class citizen. In addition to that, for Windows users who are not well > aquanted > with the Windows command line there are GUI options as well. I have little opinion on the PEP as a whole, but is the above statement true? From the git website, version 2.2.0 is current, and yet the downloadable Windows version is still 1.9.4. That's a fairly significant version lag for a "first class citizen". I like git, and it has a number of windows-specific extensions that are really useful (more than Mercurial, AFAIK), but I wouldn't say that the core product supported Windows on an equal footing to Linux. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On 30 November 2014 at 16:08, Donald Stufft wrote: >> On Nov 30, 2014, at 7:31 AM, Paul Moore wrote: >> >> On 29 November 2014 at 23:27, Donald Stufft wrote: >>> In previous years there was concern about how well supported git was on >>> Windows >>> in comparison to Mercurial. However git has grown to support Windows as a >>> first >>> class citizen. In addition to that, for Windows users who are not well >>> aquanted >>> with the Windows command line there are GUI options as well. >> >> I have little opinion on the PEP as a whole, but is the above >> statement true? From the git website, version 2.2.0 is current, and >> yet the downloadable Windows version is still 1.9.4. That's a fairly >> significant version lag for a "first class citizen". >> >> I like git, and it has a number of windows-specific extensions that >> are really useful (more than Mercurial, AFAIK), but I wouldn't say >> that the core product supported Windows on an equal footing to Linux. >> >> Paul > > I think so yes. I may be wrong, however while 1.9.4 may be the latest > downloadable version of git for Windows, there is no downloadable > version of the Linux clients at all, they just tell you to go use > your package manager which for instance is version 1.7 on Debian. On > OS X the latest version is 2.0.1. OTOH, presumably you can build your own copy of git from source on Linux/OSX. I haven't tried this on Windows but it looks pretty difficult (you start by downloading the msysgit development environment and go from there). Also, if it's easy to produce binaries for 2.2.0 on Windows, why haven't the msysgit project (still an external project, to an extent, AFAICT) done so? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 19 December 2014 at 08:26, Maciej Fijalkowski wrote: > I would like to add that "not doing anything" is not a good strategy > either, because you accumulate bugs that get fixed upstream (I'm > pretty sure all the problems from cpython got fixed in upstream > libffi, but not all libffi fixes made it to cpython). Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
On 3 January 2015 at 23:34, Steve Dower wrote: > I've put together a short post showing where I've been taking the Windows > installer for Python 3.5, since I know there are interested people on this > list who will have valuable feedback. > > http://stevedower.id.au/blog/the-python-3-5-installer/ > > Nothing is merged in yet and everything can still change, so I'm keen to hear > whatever feedback people have. I've tried to make improvements fairly for > first-time users through to sysadmins, but if I've missed something big I'd > like to hear about it before we get too close to alpha 1. Overall, this looks good. One question - will it be possible to install both 32-bit and 64-bit Python on the same machine? Currently, you need a custom install to do this (as the default directory doesn't include the architecture) and IIRC there's some oddness around install order. It would be nice if installing both versions were a supported option, both for the "default" install and in custom installs. Also, what happens now with setting PATH? Is Python (and the scripts directory) added to PATH by default? If so, what happens when you install 2 versions of Python? In case it's not clear, I'm thinking of the impact on build machines, which often have multiple versions, in both 32- and 64-bit forms, installed simultaneously (but can also be used as a "normal" development machine, and for that purpose will want a selected Python version as the default one. Also, how does the launcher py.exe fit into the picture? Is it still installed into the Windows directory? What about for a user install? Are Python scripts associated with the launcher, and if so, how does it pick up the version you selected as default? (Sorry, that was more than one question :-)) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
On 4 January 2015 at 22:56, Steve Dower wrote: >> Also, what happens now with setting PATH? Is Python (and the scripts >> directory) added to PATH by default? If so, what happens when you >> install 2 versions of Python? > > Yes, and in general the later installed version will win and system-wide > installs always beat per-user installs. As I mentioned above, using py.exe > with a parameter or shebang line is the most reliable way to get the version > you want. Hmm, that's unfortunate. Normally I leave "Add to PATH" off for all but one installation. I certainly don''t want all versions on there - I've had too many issues with running something like py.test and finding it's running Python 3.3 because I didn't install it into Python 3.4. I'm pretty sure we'll get bug reports "I installed Python 3.5.1 but I'm still getting Python 3.5" from the per-user behaviour. And saying "don't do that" isn't a fix, nor is blaming Microsoft, really. Unfortunately, I don't see a good solution here. >> Also, how does the launcher py.exe fit into the picture? Is it still >> installed into the Windows directory? What about for a user install? >> Are Python scripts associated with the launcher, and if so, how does >> it pick up the version you selected as default? > > py.exe is more important than ever. It's still installed into the Windows > directory for all-user installs, and into the Python directory for > just-for-me. It's installed in a way that will make upgrades more reliable > (so if you install 3.6 and then 3.5, you'll keep the newer launcher) and all > the file associations go straight to the launcher. The biggest problem here is that py.exe doesn't help in the slightest with script wrappers like pip.exe, virtualenv.exe, py.test.exe, ipython.exe ... I've actually drifted away from using py.exe because of this. Having just the interpreter special cased isn't really good enough. (I know there's py -m pip, but it's not easy to get people to use this...) I think it's really important to only have one Python on your PATH, so just dumping everything there by default, particularly if user installs get pushed to the end and don't have precedence, is a bad experience. > The default Python for the launcher seems to be 2.7 if it's there and the > latest version if not (though I could be wrong). Shebang lines are the best > way to choose a specific version. Yes, that's a very bad default IMO. It was probably reasonable at the time, but it should be fixed now - at least to a simple "the latest version". Getting a user who's just installed Python 3.5 (with a system Python 2.7 present) to edit an ini file in C:\Windows\System32 just to use his new version is *not* a good user experience. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
On 5 January 2015 at 01:20, Steve Dower wrote: > I think this means the best way to make multiple versions work properly is to > rename python.exe to python3.5.exe, then install the launcher as python.exe > and python3.exe (with some logic to look at its own name) so it can resolve > the right version. Maybe we can even extend the launcher to resolve launchers > in Scripts (pip.exe, etc.) and have consistent rules there too? This was a big debate over on distutils-sig. I advocated "py -m pip" for portable use, but got basically nowhere. People want "pip" to work, for compatibility with Unix (and hence ease of documentation). If we can't get PATH sorted out, we need to do something about this, IMO. I don't know what (believe me, I tried to find a solution) unfortunately. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
On 5 January 2015 at 08:40, Donald Stufft wrote: >> The biggest problem here is that py.exe doesn't help in the slightest >> with script wrappers like pip.exe, virtualenv.exe, py.test.exe, >> ipython.exe ... I've actually drifted away from using py.exe because >> of this. Having just the interpreter special cased isn't really good >> enough. (I know there's py -m pip, but it's not easy to get people to >> use this...) > > Won’t script wrappers use whatever version they were installs against? Or do > you mean the “installed” version might be 3.5 for ``pip.exe`` even though > there’s a 3.5.1 for ``pip.exe`` on the PATH? What I mean is that if you have Python 3.4 and Python 3.5 installed, and pip.exe is in the Scripts directory of each, then which pip.exe you get (and hence which Python you install into) if you just type "pip install xxx" depends on your PATH. Steve is in essence saying that it's not possible to sanely manage PATH as part of the new installer, but that py.exe makes that unnecessary. My point is that while py handles the interpreter, it doesn't handle things like pip (unless we change the standard usage instructions to say "py -m pip" is the supported approach on Windows). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
tl;dr We should have "Add this Python to PATH" as a user choice on the initial installer screen, applicable to whichever install type the user chooses. Details as to why are below. On 5 January 2015 at 17:09, Steve Dower wrote: > Paul Moore wrote: >> Steve is in essence saying that it's not possible to sanely manage PATH as >> part >> of the new installer, but that py.exe makes that unnecessary. > > It's actually not possible to sanely manage PATH from any installer - it > really needs to be handled by a user directly (though I can't ever bring > myself to tell anyone to use the UI for it). The old installers were less > susceptible because they didn't support per-user installs, but you'd still > get the "last install Python wins" behaviour. Agreed. But the current behaviour has "Add Python to PATH" as an explicit option for the user to choose (I can't recall whether the default is still not to do so, or if it was changed in 3.4, but that's a minor issue). People (the minority) who install multiple Pythons should (IMO) only set this flag on for one Python at most - and I don't feel obliged to include any special behaviour for anyone who doesn't follow that rule. The new installers, with their "One-Click" installs, don't give that choice. So they are left having to juggle the multiple path entry problem, without enough input from the user on what he/she wants. For myself, I'll probably have to use the custom install to get the control I want. But that's suboptimal, as I probably *want* to use the new locations, and having to override the "Custom" default of C:\Python35 to specify "Program Files" is a nuisance. And I bet I'll install a 32-bit Python in "C:\Program Files" rather than "C:\Program Files (x86)" at least once, and probably more than once... Maybe all that's needed is to have an extra checkbox on the first page saying "Add this Python to PATH" (doesn't matter whether it's on or off by default). We can warn if it's set on when there's another Python already on PATH, and warn more vigorously (possibly even an error) if the other Python will take precedence over the one we're installing. But I don't think you can avoid giving the user the choice of whether to put Python on PATH. There are too many bad corner cases to let the installer guess.[1] >> My point is that while py handles the interpreter, it doesn't handle things >> like >> pip (unless we change the standard usage instructions to say "py -m pip" is >> the >> supported approach on Windows). > > Or we make pip3.5.exe launch against 3.5 and pip.exe and pip3.exe behave like > the launcher, regardless of where they run from. That would be a change to pip (the wrapper generation code for wheels) and setuptools (the equivalent code for sdist installs). The launcher behaviour is more complex to handle, but could be done (at least in the case of wheels - that uses the distil wrapper code which doesn't work like the launcher but Vinay could comment on whether that would be something that could be added). I'm not against this, it's been mentioned in the past for pip but no-one has had the time or inclination to work on it. The alternative is to special-case pip and I'm strongly -1 on that - this is a general issue for all wrappers. > For example, say I have Python 3.5 and Python 3.6 installed and > PATH=C:\Python35;C:\Python35\Scripts;C:\Python36;C:\Python36\Scripts;... > (yes, those are the 'wrong' way around). If I just type "pip" it will run > C:\Python35\Scripts\pip.exe, which may or may not be what I expect. As I say, I don't really care much about supporting this case - my view is that the user should have only added *one* of the Python installations to PATH. > If pip behaved like the launcher, it would find the latest installed version > (3.6) and launch pip3.6.exe, which would always run against > C:\Python36\python.exe. (How does it do this? No idea yet. Technical > details...) One hard problem - if pip is only installed in Python 3.5, it would not be able to import pip from the Python 3.6 interpreter because it's simply not present. I'm not sure this is soluble. > This might also mean you could write "pip -3.5 install ..." to run against a > specific version, but that doesn't have to be supported if it's likely to > break scripts by stealing an argument. Since we'll install "pip", "pip3" and > "pip3.5", the options are already there. > > Would we need "pip3.5-32" as well? This could get out of hand pretty quickly. > Perhaps the launcher could include "usepy -3.
Re: [Python-Dev] New Windows installer for Python 3.5
On 5 January 2015 at 21:47, Steve Dower wrote: > Paul Moore wrote: >> tl;dr We should have "Add this Python to PATH" as a user choice on the >> initial >> installer screen, applicable to whichever install type the user chooses. >> Details >> as to why are below. > > I agree. I'll work this up before alpha 1. (FWIW, it defaults to unselected.) Cool. Would you make the new installer keep "Add to PATH" as unselected by default? I got the impression that the "One Click" installs would add to PATH, making it reasonable to have the checkbox selected by default. > Displaying a warning about other Pythons already being on the path at this > point is trickier, but I can probably figure something out. No big deal if you can't. It's a "nice to have" but not essential. >> For myself, I'll probably have to use the custom install to get the control I >> want. But that's suboptimal, as I probably *want* to use the new locations, >> and >> having to override the "Custom" default of C:\Python35 to specify "Program >> Files" is a nuisance. And I bet I'll install a 32-bit Python in "C:\Program >> Files" rather than "C:\Program Files (x86)" at least once, and probably more >> than once... > > Inclined to agree, and it is possible to make the default on the customise > page switch with the Install as Administrator checkbox until it's been > manually edited. My main concern is people who want it the old way, but they > may not actually exist. Perhaps making the old way trickier is the best way > to flush them out sooner... Having said this, if I can choose "Add to PATH" without needing a custom install, then probably the only reason I'd be likely to want a custom install would be to change the path, so the value of the default is not such a big deal for me personally. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
On 6 January 2015 at 13:56, Nick Coghlan wrote: >> If we can't get PATH sorted out, we need to do something about this, >> IMO. I don't know what (believe me, I tried to find a solution) >> unfortunately. > > I personally believe we should aim to make the "Windows" section in > the above link universal (i.e. working across *nix, Windows, and > virtual environments). That's the only one that can feasibly be made > universal - all the other alternatives are already known to be > incompatible with particular scenarios. Making it the universally recommended approach is one thing. Getting people to actually work that way is another thing entirely :-( Also, the "Windows recommendation" (currently) doesn't work properly with virtualenv - there's no way to get py.exe to use "the currently active virtualenv" (i.e., the default version from PATH). It should be possible to fix this - the difficulty is likely to be designing a viable interface. As a first cut, I think py.exe should default to (in the following order): 1. The first version of Python on PATH that "looks like a virtualenv" (need to consider both venv and virtualenv styles). 2. The default version of Python set in the ini file. 3. The latest version of Python installed on the machine (no longer preferring Python 2 over Python 3!) A #! line in a script being run will override all of these, of course, as will explicit command line flags. Does this seem reasonable? I could work on a patch if it does. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile Python on Windows (OpenSSL)
On 15 January 2015 at 22:26, Zachary Ware wrote: > Extension building in general is still a mess on Windows, I hope the > links above are enough to get you going again! For building extensions, I have a powershell script that, starting with a clean machine, downloads and installs everything needed to build extensions for Python 2.7-3.4 (Python, 32 and 64-bit, SDK compilers and Visual C for Python 2.7, and some support packages). It's available at https://github.com/pfmoore/pybuild It's pretty fragile (largely because the SDK installs are pretty fragile, but also because it doesn't check if things it wants to install are already there), but it's good for setting up a new VM from scratch. It isn't designed for building Python, and I've no idea how well it would work for that. But you might be able to pick out some parts of it that would be useful (if nothing else, it includes direct download URLs for the various components needed). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any grammar experts?
On 26 January 2015 at 18:12, Antoine Pitrou wrote:
> I have to agree with Barry. While the following is obvious even without
> having ever used it:
>
> a, b, *c = range(5)
>
> the following makes me scratch my head and I can't seem to guess what
> the *intent* is - which is very problematic when e.g. reviewing code:
>
> [*x for x in it]
> {**x for x in it}
>
> (and other similar things)
>
> I also think the multiple-starargs function calls are completely
> overboard:
>
> f(**someargs, **someotherargs)
>
> (I might add I've never felt any need for those)
These cases made little sense to me when I first saw them here, but a
quick read of the PEP made them feel reasonably straightforward - *x
is essentially splicing in a list, and a sequence of **x is a sort of
"if you don't find it here, look here" fallback mechanism. Also, real
examples would presumably have clearer variable names, and maybe even
an explanatory comment. And if they don't, they should do (and a code
review that said "this is confusing, add a comment" would seem
entirely appropriate to me).
There *are* some nastily non-intuitive corner cases (for example, if
from_env={'a':12} and from_config={'a':13}, I don't have any sort of
intuition as to what a would be in f(**from_env, **from_config). I'd
go with 12 because the PEP links multiple **-unpackings with
collections.ChainMap, but I wouldn't dare rely on that guess).
My feeling is that the PEP is essentially fine, but the
"Disadvantages" section needs expansion to note (in a reasonable
amount of detail) that it's possible to write very obfuscated code
with these constructs. It should also call out the corner cases and
note that the behaviour, although following from the rules, isn't
obvious. Personally, I don't think the resulting disadvantages are so
bad that the PEP needs to be rejected, it's just a matter of being
open about the potential for unclear code.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] (no subject)
On 10 February 2015 at 00:29, Neil Girdhar wrote:
>> > function(**kw_arguments, **more_arguments)
>> If the key "key1" is in both dictionaries, more_arguments wins, right?
>
>
> There was some debate and it was decided that duplicate keyword arguments
> would remain an error (for now at least). If you want to merge the
> dictionaries with overriding, then you can still do:
>
> function(**{**kw_arguments, **more_arguments})
>
> because **-unpacking in dicts overrides as you guessed.
Eww. Seriously, function(**{**kw_arguments, **more_arguments}) feels
more like a Perl "executable line noise" construct than anything I'd
ever want to see in Python. And taking something that doesn't work and
saying you can make it work by wrapping **{...} round it just seems
wrong.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] (no subject)
On 10 February 2015 at 01:48, Donald Stufft wrote: > I am really really -1 on the comprehension syntax. [... omitting because gmail seems to have messed up the quoting ...] > I don’t think * means “loop” anywhere else in Python and I would never > “guess” that [*item for item in iterable] meant that. It’s completely non > intuitive. Anywhere else you see *foo it’s unpacking a tuple not making an > inner loop. That means that anywhere else in Python *item is the same thing > as item[0], item[1], item[2], …, but this PEP makes it so just inside of a > comprehension it actually means “make a second, inner loop” instead of what > I think anyone who has learned that syntax would expect, which is it should > be equivalent to [(item[0], item[1], item[2], …) for item in iterable]. I agree completely with Donald here. The comprehension syntax has consistently been the part of the proposal that has resulted in confused questions from reviewers, and I don't think it's at all intuitive. Is it allowable to vote on parts of the PEP separately? If not, then the comprehension syntax is enough for me to reject the whole proposal. If we can look at parts in isolation, I'm OK with saying -1 to the comprehension syntax and then we can look at whether the other parts of the PEP add enough to be worth it (the comprehension side is enough of a distraction that I haven't really considered the other bits yet). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 370 - per-user scripts directory on Windows
tl;dr - Having a shared per-user scripts directory on Windows (%APPDATA%/Python/Scripts) causes a number of potential issues when users install the same package in multiple Python versions. I'd like to suggest that this be changed to a versioned directory (%APPDATA%/PythonXY/Scripts) to match all other Python directories on Windows. This issue originally started as a discussion on the pip tracker around making --user the default for pip. See https://github.com/pypa/pip/issues/1668 for details. However, because the user scripts directory is shared across all Python versions, this poses a number of issues: 1. If we "pip install" something like pytest, which version gets the name "py.test.exe" works on a "last install wins" basis, which is not ideal. 2. The above behaviour (and more generally the shared site scripts directory) does not respect the user's "Make this my default Python" choice from the installer, in potentially surprising ways. 3. Uninstalling a package which has been installed in 2 pythons will likely fail, because either the checksum of an installed script does not match what's in the RECORD file, or because a file that should be present has already been uninstalled via a different Python version [1]. I suggest that the per-user scripts directory should be versioned, in exactly the same way as every other Python-related directory on Windows. So we would have %APPDATA%\PythonXY\Scripts alongside %APPDATA%\PythonXY\site-packages. This is a familiar pattern for Windows users, and should not cause significant confusion. It would be good to get this added for Python 3.5, *before* making user installs the default (needed for 3.5 as the default install location for 3.5 will need admin privileges to write). Some additional points: 1. This is a Windows-specific proposal, because Unix has always had scripts installed in a common location (e.g., /usr/local/bin) and commonly-understood solutions already exist on that platform (versioned command names, python -m, system package management, etc). However, these solutions would be a significant change for Windows users. 2. Packages which recommend a user install specifically so that their commands are available regardless of which Python is on PATH would have to find an alternative solution. I doubt there are many such packages, but this should be considered. 3. This proposal does not affect the Python launcher for Windows. People using the launcher exclusively may be currently using "py -m pip" (or whatever) to launch applications, in which case the change of user script directory won't affect them. 4. I am not aware of any tools that rely on the location of the user scripts directory. Such scripts should be using sysconfig in any case. 5. The problems mentioned exist in current Python versions, but haven't been flagged up, presumably because very few people use the user install location - precisely because it's not the default in pip. Changing the pip default would cause more people to encounter the issues, but I don't propose backporting this change, as it's not a bug fix. If it becomes a major issue for pip users on older versions, we can either reconsider this, or pip can decide on its own workaround. Comments? If this is acceptable, I would be willing to prepare a patch for Python 3.5 implementing this. Paul [1] Checking of checksums is not currently implemented, AFAIK, but it is the purpose of the checksums in the RECORD file, and could be implemented in future versions of pip. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows
On 10 February 2015 at 13:34, Nick Coghlan wrote: > I was actually surprised it didn't already work that way, given that almost > everything else on Windows is version specific. That's actually a point I missed making. While I wouldn't class myself as "typical", when I went looking for the user-scripts directory on my machine, I automatically looked in %APPDATA%\PythonXY. I suspect that many people would assume this was how it worked anyway, and be surprised by the current behaviour. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
