subject:"\[Python\-Dev\] Status of C compilers for Python on Windows"


 On Oct 29, 2014, at 11:37 AM, Steve Dower steve.do...@microsoft.com wrote:
 
 Antoine Pitrou wrote:
 On Wed, 29 Oct 2014 11:07:53 -0400
 Tres Seaver tsea...@palladion.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 On 10/29/2014 10:31 AM, R. David Murray wrote:
 
 If you are writing code targeted for Windows, I think you are very 
 likely to have an MSDN subscription of some sort if your package 
 includes C code.  I'm sure it's not 100%, though.
 
 My experience with distributing distributions-with-extensions 
 indicates that the vast majority of Windows developers who are 
 downstream users for those distributions *cannot* build them from 
 source:  if there is no MSI / bdist_win (maybe now wheel), they won't use 
 the project.
 
 (Note that having an MSDN subscription is not the same as knowing 
 how to configure which compiler such that it can bulid extensions 
 against an installed Python binary).
 
 I don't think you have to configure anything. Just install the right Visual 
 Studio version and it's done.
 
 The tricky part here is the *right* Visual Studio version. As many have 
 noted, there were bugs/missing components in some of the older Express 
 editions (like the 64-bit compiler integration), and even the compiler pack 
 we released recently doesn't actually help building Python itself, though 
 it's good for extensions. In general, VS 2008 Professional and VS 2010 
 Professional are easiest to set up if you can get them, the C++ Express 
 editions are fairly easy to set up, and the Windows SDK is difficult but 
 possible (and won't currently build CPython either, as the build depends on 
 VS - I'm also fixing this for 3.5).
 
 My ideal target (Utopia refined to be achievable) is for python-dev to handle 
 the Python binaries themselves (already done) and to make them easy to bundle 
 with applications/etc (I'm working on this for 3.5), and for PyPI to include 
 pre-built wheels for Windows where necessary.
 
 Note that I don't require package developers to build their own wheels, 
 though I think that they are generally the right people to be doing it. It 
 would be nice to create a culture of delegation, so that someone could be a 
 trusted builder for a range of packages (for example, I'd love it if 
 Continuum/Enthought/similar could provide their builds of numpy/scipy as 
 wheels and those projects would be willing to have them be the official PyPI 
 downloads). This is roughly how the python.org installers are handled, and 
 while it may cause some lag between source and binary releases, I think the 
 release cycles are slow enough that most users would only notice that pip 
 install numpy now works.

For the record, a long term “down the road” kind of thing I want to do is have 
PyPI have a build farm which can build Wheels. So that people upload a source 
distribution to PyPI and it kicks off Wheel builds on the various platforms 
automatically.

What this looks like is a complete unknown, all I have is the general idea.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 29 October 2014 15:31, Nathaniel Smith n...@pobox.com wrote:
 You can use Express editions of Visual Studio.

 IIUC, the express edition compilers are 32-bit only, and what you actually
 want are the SDK compilers:
 https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows

 These are freely downloadable by anyone, no msdn subscription required, but
 only if you know where to find them!

 AFAICT the main obstacle to using MSVC to build python extensions (assuming
 it can handle your code at all) is not anything technical, but rather that
 there's no clear and correct tutorial on how to do it, and lots of confusion
 and misinformation circulating.

Would it help if I wrote a document explaining how to set up the MS
compilers (free and paid for) to allow building of Python extensions?

There are a few provisos:

1. A lot of it will be pretty trivial: If you have Vistal Studio
(full edition), install it. Done.
2. It will be out of date very fast as the situation for Python 3.5+
will be trivial across the board.
3. I don't have anywhere particularly authoritative to host it (maybe
the Python Wiki?) and it could easily get lost in the huge swamp of
variously outdated, over-complicated, or otherwise alternative
documents available. Ideally I'd like someone to suggest an official
location I could use.

I don't want to do this if it won't be useful, as it'll take me a bit
of effort to confirm the process for the only non-trivial scenario
(64-bit Python 3.3/3.4 with free tools). But if people think it would
help, that's fine, I volunteer.

Paul

PS Even if I don't get positive feedback, I may just say to heck with
it and do it anyway, because it *is* so trivial :-) I just won't
promise.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-29 Thread Tony Kelman


Stephen J. Turnbull:

the pain of using Windows is what drives me away from all of them.


Enough that you are not able to make the software you write usable on
Windows? I see your point and agree with it - I don't even like Windows
much at all, but supporting it is important for plenty of reasons.

Steve Dower:

FWIW, I'm happy to talk specifics off list


Off-list, on-list, either way. I've read the blog post about the CRT
refactoring 
http://blogs.msdn.com/b/vcblog/archive/2014/06/10/the-great-crt-refactoring.aspx

but have not yet experimented with the early CTP versions of Visual Studio
14. Is there any plan to promote these runtimes to installed-by-default
components of the operating system in future service packs or new releases
of Windows? The fact that the redistributables are a separate download and
not always there by default means MinGW developers are not quite so
optimistic about them suddently solving all problems. In the embedded-
Python-for-IJulia pull request on github for example, it turns out the
Python msi installer does not include the necessary runtimes and wouldn't
work by itself on a brand new computer.

It would also help if there were a preview version of just the runtime
libraries available. More far-fetched and not likely to happen would be a
smaller command line tools for Visual Studio type package, like Apple
provides for Xcode, with only the compiler, tools, and libraries needed
to build C/C++ libraries or Python extensions but not the whole IDE.

Nick Coghlan:

* wanting to build for Windows within the scope of an existing POSIX
based build automation system
* folks that prefer to use only open source software themselves
wanting to make their projects available to Windows users


Don't forget: * MSVC is not capable of compiling the code I need to use.
That's the crux of the scientific-software problem. If it were possible to
use MSVC for everything, I'd probably suck it up and spend my time writing
CMake build systems for every project I wanted to get to work on Windows.
Most software I work with is a library first, Python extension second, if
at all.

Nick Coghlan:

Lots of folks are happy with POSIX emulation layers on Windows, as
they're OK with basically works rather than works like any other
native application. Basically works isn't sufficient for many
Python-on-Windows use cases though, so the core ABI is a platform
native one, rather than a POSIX emulation.


To clarify here: MinGW does not use a POSIX layer at all. Cygwin does,
MSYS does (it's based on Cygwin with relatively minor changes), but when
we're discussing the MinGW.org or MinGW-w64 compilers, there is no POSIX.
There is a GCC runtime library, which can be linked statically if you
choose, though it's not always a good idea to do so. Cygwin or MSYS provide
a POSIX build environment where you can run bash, make, etc, but with MinGW
you are not targeting or depending on that build environment with the
compiled binaries.

The distinction in runtime libraries is that MinGW links to msvcrt.dll,
which is from an old version of Visual Studio but is installed by default
on every version of Windows since XP. The python.org installers are
compiled with a particular more recent version of Visual Studio, so
they (and hence all extensions used with them) depend on the specific
corresponding msvcr##.dll.

Most MinGW developers and users would rather avoid the dll hell and issues
stemming from choosing a particular MSVC runtime and having to stick with
it for everything. NumPy was able to coax a custom MinGW-w64 build into
linking to CPython's designated runtime and statically link all the GCC
libraries (these are 2 separate issues however). I do think some sensitivity
to the requirements of the existing Windows CPython/PyPI environment from
the MinGW-w64 side would be valuable. If NumPy's needs can be met with
fewer modifications and without having to rebuild a custom GCC from source,
I think that would be beneficial for everyone. We'll see how many of the
MinGW-w64 developers I can get to agree with me on that. Any changes that
may need to be upstreamed to GCC will have their own set of challenges.


Adding FORTRAN dependencies to the mix (which
*none* of the platform vendors really support properly)


I'm not sure what you mean by platform vendors? Are you talking Conda,
Enthought, etc? To the best of my knowledge, they're using the Intel
toolchain for Fortran, at least on Windows and probably elsewhere. So
I suspect you'll find a libifcoremd.dll somewhere in those distributions.
That imposes some cost requirements on reproducing their processes that
are a bit high for average users to meet.

Donald Stufft:

So that people upload a source distribution to PyPI and it kicks off
Wheel builds on the various platforms automatically.

What this looks like is a complete unknown, all I have is the general
idea.


I do something similar today for C, C++, and Fortran libraries using
the MinGW-w64 cross-compiler. It looks like

Re: [Python-Dev] Status of C compilers for Python on Windows

This sounds like something good for packaging.python.org


 On Oct 29, 2014, at 4:05 PM, Paul Moore p.f.mo...@gmail.com wrote:
 
 On 29 October 2014 15:31, Nathaniel Smith n...@pobox.com wrote:
 You can use Express editions of Visual Studio.
 
 IIUC, the express edition compilers are 32-bit only, and what you actually
 want are the SDK compilers:
 https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
 
 These are freely downloadable by anyone, no msdn subscription required, but
 only if you know where to find them!
 
 AFAICT the main obstacle to using MSVC to build python extensions (assuming
 it can handle your code at all) is not anything technical, but rather that
 there's no clear and correct tutorial on how to do it, and lots of confusion
 and misinformation circulating.
 
 Would it help if I wrote a document explaining how to set up the MS
 compilers (free and paid for) to allow building of Python extensions?
 
 There are a few provisos:
 
 1. A lot of it will be pretty trivial: If you have Vistal Studio
 (full edition), install it. Done.
 2. It will be out of date very fast as the situation for Python 3.5+
 will be trivial across the board.
 3. I don't have anywhere particularly authoritative to host it (maybe
 the Python Wiki?) and it could easily get lost in the huge swamp of
 variously outdated, over-complicated, or otherwise alternative
 documents available. Ideally I'd like someone to suggest an official
 location I could use.
 
 I don't want to do this if it won't be useful, as it'll take me a bit
 of effort to confirm the process for the only non-trivial scenario
 (64-bit Python 3.3/3.4 with free tools). But if people think it would
 help, that's fine, I volunteer.
 
 Paul
 
 PS Even if I don't get positive feedback, I may just say to heck with
 it and do it anyway, because it *is* so trivial :-) I just won't
 promise.
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 29 October 2014 20:26, Donald Stufft don...@stufft.io wrote:
 This sounds like something good for packaging.python.org

Yeah, I wondered about that. I'll work up a patch for that. But the
more I think about it, it really is trivial:

- For non-free MSVC, install the appropriate version, and everything just works.
- For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
package and everything just works as long as you're using setuptools.
- For 32 bit Python 3.2-3.4, install Visual Studio Express and
everything just works.
- For 64 bit Python 3.2-3.4, install the SDK, set some environment
variables, and everything just works.
- For Python 3.5+, install the current Visual Studion Express and
everything just works.

(I propose to deem Python 2.7 without setuptools as unsupported)

The only things I can see that need expansion are:

1. The precise versions to use (trivial to add, I'm just to lazy to
check right now).
2. Where to get VS 2010 Express (as it's no longer supported or
available from MS).
3. How to set up the SDK environment for 64-bit Python 3.2-3.4.

Before I do a write-up, I want to set up clean VMs with these
configurations, so that I can confirm the details. But I don't expect
any problems, as I've done them all before.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows


 On Oct 29, 2014, at 6:09 PM, Paul Moore p.f.mo...@gmail.com wrote:
 
 On 29 October 2014 20:26, Donald Stufft don...@stufft.io wrote:
 This sounds like something good for packaging.python.org
 
 Yeah, I wondered about that. I'll work up a patch for that. But the
 more I think about it, it really is trivial:
 
 - For non-free MSVC, install the appropriate version, and everything just 
 works.
 - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
 package and everything just works as long as you're using setuptools.
 - For 32 bit Python 3.2-3.4, install Visual Studio Express and
 everything just works.
 - For 64 bit Python 3.2-3.4, install the SDK, set some environment
 variables, and everything just works.
 - For Python 3.5+, install the current Visual Studion Express and
 everything just works.
 
 (I propose to deem Python 2.7 without setuptools as unsupported)
 
 The only things I can see that need expansion are:
 
 1. The precise versions to use (trivial to add, I'm just to lazy to
 check right now).
 2. Where to get VS 2010 Express (as it's no longer supported or
 available from MS).
 3. How to set up the SDK environment for 64-bit Python 3.2-3.4.
 
 Before I do a write-up, I want to set up clean VMs with these
 configurations, so that I can confirm the details. But I don't expect
 any problems, as I've done them all before.
 
 Paul.

I think it’d be good even if considered trivial. I know I certainly have no
idea which pieces I needed to do that.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-29 Thread Ethan Furman


On 10/29/2014 03:09 PM, Paul Moore wrote:

On 29 October 2014 20:26, Donald Stufft don...@stufft.io wrote:

This sounds like something good for packaging.python.org


Yeah, I wondered about that. I'll work up a patch for that. But the
more I think about it, it really is trivial:


I am reminded of an interview question I was once asked which was prefaced with: 
Here's an easy one...

My reply was, if you know the answer, it's easy!



- For non-free MSVC, install the appropriate version, and everything just works.
- For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
package and everything just works as long as you're using setuptools.
- For 32 bit Python 3.2-3.4, install Visual Studio Express and
everything just works.
- For 64 bit Python 3.2-3.4, install the SDK, set some environment
variables, and everything just works.
- For Python 3.5+, install the current Visual Studion Express and
everything just works.


I would suggest
  - where to get these packages
  - where to get any dependencies
  - any options to [not] specify during install
  - what environment variables to what value
  - where one should be at when one starts the compile process

and, of course, a gotchas section for uncommon but frustrating things that 
might go wrong.

And thanks for doing this!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 29 October 2014 22:19, Ethan Furman et...@stoneleaf.us wrote:
 Yeah, I wondered about that. I'll work up a patch for that. But the
 more I think about it, it really is trivial:

 I am reminded of an interview question I was once asked which was prefaced
 with: Here's an easy one...

 My reply was, if you know the answer, it's easy!

Yeah, I know what you mean. My take on this is that I agree it's not
easy if you don't know and can't get access to the information, but if
you can, there's very little to it.

 - For non-free MSVC, install the appropriate version, and everything just
 works.
 - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
 package and everything just works as long as you're using setuptools.
 - For 32 bit Python 3.2-3.4, install Visual Studio Express and
 everything just works.
 - For 64 bit Python 3.2-3.4, install the SDK, set some environment
 variables, and everything just works.
 - For Python 3.5+, install the current Visual Studion Express and
 everything just works.


 I would suggest
   - where to get these packages

Conceded. Working out how to point people at stuff on MSDN is a
challenge, things seem to move around. Maybe Steve Dower could help
here with canonical URLs for some of them (IIRC, he provided one for
the VS compilers for Python 2.7 package when it was announced).

For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)

And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly.

   - where to get any dependencies

There are none. I could state that explicitly, I guess.

   - any options to [not] specify during install

I'll have to go through the installs again just to be sure, but I'm
pretty certain Select the default for everything is correct.

   - what environment variables to what value

None, except for the SDK which I did say I needed to test out and
cover in more detail.

   - where one should be at when one starts the compile process

I don't understand this. It's just pip wheel foo to build a wheel
for foo (which will be downloaded), or pip wheel . or  python
setup.py bdist_wheel as you prefer for a local package. That's
standard distutils/setuptools/pip extension building. I don't propose
to cover that, just how to *set up* the environment.

With the sole exception of the SDK case, once installed, everything
just works everywhere, nothing to set up, no special environment to
configure. Start up a new cmd or powershell console (or use the one
you're already in) and go. Maybe Unix users expect more complexity
because it's not that simple on Unix? But I thought it was - install
the appropriate OS packages and that's it?

 and, of course, a gotchas section for uncommon but frustrating things that
 might go wrong.

Hmm, I see your point here, but I'm not sure what I might include. You
*can* get in a mess [1] but generally I don't as I'm an experienced
Windows user. I also don't want to offer to debug and fix everyone's
problems in getting things set up, so offering to collect and document
common issues that I've seen is impractical. Maybe a section
entitled Common Issues and Their Solutions, with some placeholder
text saying If you have any issues installing any of the compiler
packages mentioned, please document what went wrong and how to fix it,
and submit a PR! would do?

 And thanks for doing this!

No problem!

Paul

[1] I once spent a *long* time fighting failed installs of the Windows
SDK. Turns out it was because I was trying to install a 32-bit SDK on
a 64-bit machine (doh!), and the installer really doesn't like that
:-( About all I could say to document that is Read the instructions
properly and I'm sorry that the MS installers fail really badly when
faced with relatively obvious idiot-user errors, but I can't do
anything about it :-(
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-29 Thread Ethan Furman


On 10/29/2014 03:46 PM, Paul Moore wrote:

On 29 October 2014 22:19, Ethan Furman et...@stoneleaf.us wrote:


   - where one should be at when one starts the compile process


I don't understand this. It's just pip wheel foo to build a wheel
for foo (which will be downloaded), or pip wheel . or  python
setup.py bdist_wheel as you prefer for a local package.


Hmmm...  That looks like it's for installing/compiling somebody else's package.  Is that last command sufficient to 
prepare one's own wheel for uploading to PyPI, or there something else to do?


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows


 On Oct 29, 2014, at 7:02 PM, Ethan Furman et...@stoneleaf.us wrote:
 
 On 10/29/2014 03:46 PM, Paul Moore wrote:
 On 29 October 2014 22:19, Ethan Furman et...@stoneleaf.us wrote:
 
   - where one should be at when one starts the compile process
 
 I don't understand this. It's just pip wheel foo to build a wheel
 for foo (which will be downloaded), or pip wheel . or  python
 setup.py bdist_wheel as you prefer for a local package.
 
 Hmmm...  That looks like it's for installing/compiling somebody else's 
 package.  Is that last command sufficient to prepare one's own wheel for 
 uploading to PyPI, or there something else to do?
 

Generally for uploading to PyPI you do ``python setup.py bdist_wheel``, though 
I don’t think there’d be any bad thing if you used pip wheel.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-29 Thread Nathaniel Smith

On Wed, Oct 29, 2014 at 10:46 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 29 October 2014 22:19, Ethan Furman et...@stoneleaf.us wrote:
 Yeah, I wondered about that. I'll work up a patch for that. But the
 more I think about it, it really is trivial:

 I am reminded of an interview question I was once asked which was prefaced
 with: Here's an easy one...

 My reply was, if you know the answer, it's easy!

 Yeah, I know what you mean. My take on this is that I agree it's not
 easy if you don't know and can't get access to the information, but if
 you can, there's very little to it.

That's great, but yeah. In case it helps as a data point, I consider
myself a reasonably technical guy, probably more than your average
python package author -- 15 years of free software hacking, designed a
pretty popular X remote display protocol, helped invent DVCS, have
written patches to gcc and the kernel, numpy co-maintainer, etc. etc.

For the last ~12 months I've had an underlined and circled todo item
saying make wheels for https://pypi.python.org/pypi/zs/;, and every
time I've sat down and spent a few hours trying I've ended up utterly
defeated with a headache. Distutils is spaghetti, and Windows is
spaghetti (esp. if you've never used it for more than checking email
on a friend's laptop), and the toolchain setup is spaghetti, and then
suddenly people are talking about vcvarsall.bats and I don't know what
all. I honestly would totally benefit from one of those
talk-your-grandparents-through-it tutorials (here's a screenshot of
the dialogue box, and I've drawn an arrow pointing at the 'ok' button.
You should press the 'ok' button.)

 - For non-free MSVC, install the appropriate version, and everything just
 works.
 - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
 package and everything just works as long as you're using setuptools.
 - For 32 bit Python 3.2-3.4, install Visual Studio Express and
 everything just works.
 - For 64 bit Python 3.2-3.4, install the SDK, set some environment
 variables, and everything just works.

I think the SDK covers both 32 and 64 bit? If so then leaving visual
studio out of things entirely is probably simpler.

I'm also unsure how you even get both 32- and 64-bit versions of
python to coexist on the same system.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 29 October 2014 23:22, Nathaniel Smith n...@pobox.com wrote:
 Yeah, I know what you mean. My take on this is that I agree it's not
 easy if you don't know and can't get access to the information, but if
 you can, there's very little to it.

 That's great, but yeah. In case it helps as a data point, I consider
 myself a reasonably technical guy, probably more than your average
 python package author -- 15 years of free software hacking, designed a
 pretty popular X remote display protocol, helped invent DVCS, have
 written patches to gcc and the kernel, numpy co-maintainer, etc. etc.

Yeah, that counts :-)

 For the last ~12 months I've had an underlined and circled todo item
 saying make wheels for https://pypi.python.org/pypi/zs/;, and every
 time I've sat down and spent a few hours trying I've ended up utterly
 defeated with a headache. Distutils is spaghetti, and Windows is
 spaghetti (esp. if you've never used it for more than checking email
 on a friend's laptop), and the toolchain setup is spaghetti, and then
 suddenly people are talking about vcvarsall.bats and I don't know what
 all. I honestly would totally benefit from one of those
 talk-your-grandparents-through-it tutorials (here's a screenshot of
 the dialogue box, and I've drawn an arrow pointing at the 'ok' button.
 You should press the 'ok' button.)

OK. That needs to be fixed. How about this. I'll work with you to get
the wheel build process set up in such a way that you can maintain it,
and we'll document what you needed to know in order to do it. Then I
can put that documentation into the packaging user guide for other
people to work with. It can wait till you have the time, but when you
do, shout and I'll help.

I know nothing about zs, but I just tried building it. MSVC 2010
doesn't support static inline (used twice in zs\pycrc-crc64xz.h) but
when I removed the inline from that it built fine for me. So it
should be a good example, as there are no particular complexities to
deal with.

 - For non-free MSVC, install the appropriate version, and everything just
 works.
 - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
 package and everything just works as long as you're using setuptools.
 - For 32 bit Python 3.2-3.4, install Visual Studio Express and
 everything just works.
 - For 64 bit Python 3.2-3.4, install the SDK, set some environment
 variables, and everything just works.

 I think the SDK covers both 32 and 64 bit? If so then leaving visual
 studio out of things entirely is probably simpler.

The SDK is the most complex environment to set up, so avoiding it
unless you need it is a good thing. But maybe it's worth explaining
that if you need it anyway, this is how you set it up to cover some of
the other cases as well.

 I'm also unsure how you even get both 32- and 64-bit versions of
 python to coexist on the same system.

Just install them into different directories. The default is the same
for 32 and 64 bit, so you need to override the default for (at least)
one of them, but that's all.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 29 October 2014 23:49, Steve Dower steve.do...@microsoft.com wrote:
For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)

You could link to visualstudio.com for the trial versions, and maybe to a
page/post about the PSF's grants process if such a page exists.

That doesn't help for VS 2010 and VS 2008, which are the relevant ones
right now, unfortunately. In thew brave new world of Python 3.5+ that
might well do, though. (Although the feedback from Unix users is that
they really want a direct link to the exact edition they need - i.e.,
a permalink to VC++ Express Edition latest for Python 3.5+. I'm not
sure how well the MS website structure does that, though.

And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly

These are still at
http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs#DownloadFamilies_4,
which is the main download page. Hopefully they don't go away before 3.5,
but I have no control over that unfortunately.

Wow! I didn't know that. That's great, solves the biggest issue I had.

The link I posted for 2.7 was aka.ms/vcpython27, which is a redirect that I
own and can update if necessary.

Cool. As long as it's stable and reliable, it's exactly what I want.

Many thanks,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-29 Thread Steve Dower

For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)

You could link to visualstudio.com for the trial versions, and maybe to a 
page/post about the PSF's grants process if such a page exists.

And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly

These are still at 
http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs#DownloadFamilies_4,
 which is the main download page. Hopefully they don't go away before 3.5, but 
I have no control over that unfortunately.

The link I posted for 2.7 was aka.ms/vcpython27, which is a redirect that I own 
and can update if necessary.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Mooremailto:p.f.mo...@gmail.com
Sent: ‎10/‎29/‎2014 15:48
To: Ethan Furmanmailto:et...@stoneleaf.us
Cc: Python Devmailto:python-dev@python.org
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

On 29 October 2014 22:19, Ethan Furman et...@stoneleaf.us wrote:
 Yeah, I wondered about that. I'll work up a patch for that. But the
 more I think about it, it really is trivial:

 I am reminded of an interview question I was once asked which was prefaced
 with: Here's an easy one...

 My reply was, if you know the answer, it's easy!

Yeah, I know what you mean. My take on this is that I agree it's not
easy if you don't know and can't get access to the information, but if
you can, there's very little to it.

 - For non-free MSVC, install the appropriate version, and everything just
 works.
 - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
 package and everything just works as long as you're using setuptools.
 - For 32 bit Python 3.2-3.4, install Visual Studio Express and
 everything just works.
 - For 64 bit Python 3.2-3.4, install the SDK, set some environment
 variables, and everything just works.
 - For Python 3.5+, install the current Visual Studion Express and
 everything just works.


 I would suggest
   - where to get these packages

Conceded. Working out how to point people at stuff on MSDN is a
challenge, things seem to move around. Maybe Steve Dower could help
here with canonical URLs for some of them (IIRC, he provided one for
the VS compilers for Python 2.7 package when it was announced).

For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)

And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly.

   - where to get any dependencies

There are none. I could state that explicitly, I guess.

   - any options to [not] specify during install

I'll have to go through the installs again just to be sure, but I'm
pretty certain Select the default for everything is correct.

   - what environment variables to what value

None, except for the SDK which I did say I needed to test out and
cover in more detail.

   - where one should be at when one starts the compile process

I don't understand this. It's just pip wheel foo to build a wheel
for foo (which will be downloaded), or pip wheel . or  python
setup.py bdist_wheel as you prefer for a local package. That's
standard distutils/setuptools/pip extension building. I don't propose
to cover that, just how to *set up* the environment.

With the sole exception of the SDK case, once installed, everything
just works everywhere, nothing to set up, no special environment to
configure. Start up a new cmd or powershell console (or use the one
you're already in) and go. Maybe Unix users expect more complexity
because it's not that simple on Unix? But I thought it was - install
the appropriate OS packages and that's it?

 and, of course, a gotchas section for uncommon but frustrating things that
 might go wrong.

Hmm, I see your point here, but I'm not sure what I might include. You
*can* get in a mess [1] but generally I don't as I'm an experienced
Windows user. I also don't want to offer to debug and fix everyone's
problems in getting things set up, so offering to collect and document
common issues that I've seen is impractical. Maybe a section
entitled Common Issues and Their Solutions, with some placeholder
text saying If you have any issues installing any of the compiler
packages mentioned, please document what went wrong and how to fix it,
and submit a PR! would do?

 And thanks for doing this!

No problem!

Paul

[1] I once spent a *long* time fighting failed installs of the Windows
SDK. Turns out it was because I was trying to install a 32-bit SDK on
a 64-bit machine (doh!), and the installer really doesn't like that
:-( About all I could say to document that is Read the instructions
properly and I'm sorry that the MS installers fail really badly

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-29 Thread Terry Reedy


On 10/29/2014 4:05 PM, Paul Moore wrote:

On 29 October 2014 15:31, Nathaniel Smith n...@pobox.com wrote:

You can use Express editions of Visual Studio.


IIUC, the express edition compilers are 32-bit only, and what you actually
want are the SDK compilers:
https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows

These are freely downloadable by anyone, no msdn subscription required, but
only if you know where to find them!

AFAICT the main obstacle to using MSVC to build python extensions (assuming
it can handle your code at all) is not anything technical, but rather that
there's no clear and correct tutorial on how to do it, and lots of confusion
and misinformation circulating.


Would it help if I wrote a document explaining how to set up the MS
compilers (free and paid for) to allow building of Python extensions?



There are a few provisos:

1. A lot of it will be pretty trivial: If you have Vistal Studio
(full edition), install it. Done.
2. It will be out of date very fast as the situation for Python 3.5+
will be trivial across the board.
3. I don't have anywhere particularly authoritative to host it (maybe
the Python Wiki?) and it could easily get lost in the huge swamp of
variously outdated, over-complicated, or otherwise alternative
documents available. Ideally I'd like someone to suggest an official
location I could use.


There is already a section in the devguide for building Python itself, 
with mostly the same info, except it may not be up to date.



I don't want to do this if it won't be useful, as it'll take me a bit
of effort to confirm the process for the only non-trivial scenario
(64-bit Python 3.3/3.4 with free tools). But if people think it would
help, that's fine, I volunteer.


The devguide needs to be kept up to date.  If you open a tracker issue, 
put me as nosy to review and commit.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-29 Thread Terry Reedy


On 10/29/2014 11:37 AM, Steve Dower wrote:


My ideal target (Utopia refined to be achievable) is for python-dev
to handle the Python binaries themselves (already done) and to make
them easy to bundle with applications/etc (I'm working on this for
3.5), and for PyPI to include pre-built wheels for Windows where
necessary.

Note that I don't require package developers to build their own
wheels, though I think that they are generally the right people to be
doing it. It would be nice to create a culture of delegation, so that
someone could be a trusted builder for a range of packages


Christoph Gohlke, Laboratory for Fluorescence Dynamics, University of 
California, Irvine now provides 32 and 64 bit builds for 2.6/7, 3.2/3/4 
(10 total) for about 300 'scientific open-source extension packages' an 
his site.

http://www.lfd.uci.edu/~gohlke/pythonlibs/

Perhaps the PyPI folks could discuss incorporating wheel versions on 
PyPI (I have no idea what is involved, but I presume it could be antomated.)


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-28 Thread Tony Kelman


Stephen J. Turnbull:

Python is open source.  Nobody is objecting to somebody else doing
this.[1]  The problem here is simply that some somebody elses are
trying to throw future work over the wall into python-dev space.


If that's how it's seen at this point, then it sounds like the logical
course of action for CPython-with-MinGW is to demonstrate feasibility
in a fork. Get what currently works as a set of 80-something patches
and PKGBUILD instructions into a single repository that is usable by a
wider variety of people, in more distributions, etc. Set up as much CI as
possible so every patch gets tested in as many configurations as we can.
Experiment with extension compatibility and find out what is actually
achievable, with or without needing to modify MinGW-w64 in the process.

There are people, though evidently not much of python-dev, who have a
need and desire to make this happen. It seems python-dev would rather
have us spend zero time proposing changes that allow CPython itself to
be built differently than today, and all of our time on MinGW extensions.
I personally find 3 of the 4 combinations of how one could build CPython
and how one could build extensions interesting and worth looking into for
different reasons (the one that's uninteresting to me is the currently
used all-MSVC method, due to its many limitations I listed earlier).
Ray for example may only care about using MinGW for everything. Python's
a big community with lots of room for different people to work on
different aspects of the same set of problems.

For the combination of MSVC Python and MinGW extensions that most of you
have recommended we focus on, it would be more productive to engage with
setuptools, distutils-sig, and likely numpy as well, instead of python-dev.
My experience lies more in getting troublesome C codebases to build with
MinGW than it does with the messy intricacies and backwards-compatibility
requirements of Python extensions and package management however, so my
ability to contribute on that side of things is more limited. I'll just
note that every project I've ever had a patch for which improved
functionality with a new compiler (whether GCC, MSVC, clang, icc or ifort,
etc.) reacted with review and thanks for the patches, not why do you want
to do this? pushback. If potential contributors have a desire to get it
working in the first place, then they will also be invested in helping
keep it working on an ongoing basis.

Sincerely,
Tony

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-28 Thread Stephen J. Turnbull

Tony Kelman writes:

  If potential contributors have a desire to get it working in the
  first place, then they will also be invested in helping keep it
  working on an ongoing basis.

Sure -- as long as it works for them, though, such potential
contributors don't necessarily care if it works for anybody else.  My
experience (in other projects) is that allowing that level of
commitment to be sufficient for inclusion in the maintained code base
frequently results in bug reports from third parties who try to use
the new feature and discover it *doesn't* work for them.  The core
maintainers then have a very unpleasant choice: to say we don't
support that usage, or to deal with the problem on a continuing basis
(because the same issues tend to regress repeatedly as the said
invested contributors continue to modify their code on the same
works for us basis).

  If that's how it's seen at this point, then it sounds like the
  logical course of action for CPython-with-MinGW is to demonstrate
  feasibility in a fork. Get what currently works as a set of
  80-something patches and PKGBUILD instructions into a single
  repository that is usable by a wider variety of people, in more
  distributions, etc. Set up as much CI as possible so every patch
  gets tested in as many configurations as we can.  Experiment with
  extension compatibility and find out what is actually achievable,
  with or without needing to modify MinGW-w64 in the process.

Sounds good to me.  You seem to think that's excessive, though:

  There are people, though evidently not much of python-dev, who have a
  need and desire to make this happen.

Well, for starters, most of python-dev would rather avoid any contact
whatsoever with Windows.  I think part of the problem is that Windows
developers *of* Python are *very* rare (fingers of one hand rare).
Sure, there are many Windows-based Python developers, and quite of few
of them do a fair amount to contribute to Python on Windows -- but
they do that work in *Python*, not at a level where they deal with the
extension ABI.  Even those who do work on C extensions have so far
been happy to work with the VC build (except for the recurrent issue
of getting one's hands on the toolchain).

So why should python-dev have a desire to do that work?  It should be
evident by now that our belief is that the large majority of Windows
users is well-served by the current model (produce an official binary
and a plethora of compatible extensions, which provides strong
incentive to third parties to ensure that their extensions and
alternative builds of core are also compatible).

I think the repeated query, why isn't this model good enough for
you? is being misinterpreted.  I suppose that some who ask that
really want to know, because if you have what they consider a good
reason, they'd be willing to help.  Others are asking but by you
they mean the world at large, in particular, what benefit is there to
the large number of users well-served by the current model?

  It seems python-dev would rather have us spend zero time proposing
  changes that allow CPython itself to be built differently than
  today, and all of our time on MinGW extensions.

Of course they would.  (Third person because I'm not competent to do
the work anyway.)  It's quite clear that one thing the two sides agree
on is that ensuring that MinGW and VC interpreter and extension builds
can mix and match is quite a bit of work.  They quite naturally don't
want to do that work, and don't see much point in discussing it if the
(apparently) few people who need it aren't going to supply the
resources.

That's quite a reasonable solution, really: *both* communities avoid
spending effort on mix and match, and because it's a fork, it's a
different name, and people won't expect them to mix and match.

  I personally find 3 of the 4 combinations of how one could build
  CPython and how one could build extensions interesting and worth
  looking into for different reasons (the one that's uninteresting to
  me is the currently used all-MSVC method, due to its many
  limitations I listed earlier).  Ray for example may only care about
  using MinGW for everything. Python's a big community with lots of
  room for different people to work on different aspects of the same
  set of problems.

There's a reason we call it core.  One of python-dev's more
important responsibilities is to ensure that the aspects work and
play well together.  Aspects people tend to deprecate that
responsibility (until somebody else's aspect treads on their blue
suede shoes).  That's as it should be, IMO -- but so is python-dev's
reluctance to admit new aspects until their impact on core
responsibilities is made clear.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-28 Thread Tony Kelman


Stephen J. Turnbull:

Sure -- as long as it works for them, though, such potential
contributors don't necessarily care if it works for anybody else.  My
experience (in other projects) is that allowing that level of
commitment to be sufficient for inclusion in the maintained code base
frequently results in bug reports from third parties who try to use
the new feature and discover it *doesn't* work for them.


Good point. I definitely care whether patches work for everyone else.
Patches should be done well and accompanied with proper documentation
so new functionality is fully reproducible. If that's what's holding
up review, comments in the bug threads indicating as much would be
helpful. Any fork will also have to be thoroughly documented if it's
to have any chance of working.


Sounds good to me.  You seem to think that's excessive, though:


No, just hearing the words come out of my mouth they sound a little nuts.
Maybe not, there are after all half a dozen or more incompatible alternate
Python implementations floating around. I think most of them started as
from-scratch rewrites rather than source forks, but maybe that's irrelevant.


Well, for starters, most of python-dev would rather avoid any contact
whatsoever with Windows.  I think part of the problem is that Windows
developers *of* Python are *very* rare (fingers of one hand rare).


In my opinion the MSVC toolchain makes that problem worse, as it's far
harder for unix developers to have any familiarity with how things work.
But you do have involvement and core developers from Microsoft which is
obviously incredibly important. Maybe even mandatory for Python on Windows
to be viable in your eyes.


Even those who do work on C extensions have so far
been happy to work with the VC build (except for the recurrent issue
of getting one's hands on the toolchain).

It should be evident by now that our belief is that the large majority
of Windows users is well-served by the current model


This is not the case at all in the scientific community. NumPy and SciPy
put in a lot of extra work to come up with something that is compatible
with the MSVC build of CPython because they have to, not because they're
happy to jump through the necessary hoops. The situation today with
NumPy AIUI is they already have to build with a custom hacked MinGW
toolchain that only one person knows how to use. Evidently until very
recently they were using a many-years-old 32-bit-only build of GCC 3.x.
Do python-dev and numpy-discussion not talk to one another? I get that
not everyone uses numpy, or Windows, but it sounds like there's very
little understanding, or even awareness, of the issues they face.


They quite naturally don't want to do that work, and don't see much
point in discussing it if the (apparently) few people who need it aren't
going to supply the resources.


I'm going to move the extensions with MinGW-w64 part of this conversation
over to numpy-discussion, since they have a need for this today and are
already using workarounds and processes that I don't think anyone is fully
satisfied with. I do find this combination interesting, worth working on,
and possible to make work well, but not completely in isolation as it does
not address my embedding use case.


but so is python-dev's
reluctance to admit new aspects until their impact on core
responsibilities is made clear.


Okay. I'll table the discussion with python-dev for now then.

-Tony

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-28 Thread Paul Moore

On 28 October 2014 14:46, Tony Kelman kel...@berkeley.edu wrote:
 Patches should be done well and accompanied with proper documentation
 so new functionality is fully reproducible. If that's what's holding
 up review, comments in the bug threads indicating as much would be
 helpful.

Typically that tends to be expressed as a terse I can't get this
working. That's not ideal, but is an indication. When the response is
but it's easy (as it sometimes is) communication degenerates quite
fast. There's an example here in this thread - it took me a *long*
time, with help from a couple of people, to work out how to get a
mingw toolchain I could use to try things out. Even though the premise
of the whole discussion was it's easy to set up a mingw toolchain...

 In my opinion the MSVC toolchain makes that problem worse, as it's far
 harder for unix developers to have any familiarity with how things work.
 But you do have involvement and core developers from Microsoft which is
 obviously incredibly important. Maybe even mandatory for Python on Windows
 to be viable in your eyes.

One problem I've seen a lot on other projects is that when Unix
developers set up a comfortable, Unix-like environment on Windows
using something like mingw, they frequently aren't aware of crucial
differences between Unix and Windows and tend to write software that
even though it's Windows-native, *feels* Unixy to Windows users (don't
ask me to be more specific :-)). That's always going to happen, and
the people with Windows experience have to take up the slack by
keeping an eye out for such things, but setting the bar marginally
higher, to you have to at least be willing to download and use a
native Windows compiler can at least remind said Unix developers that
they are in a different environment.

That's *not* a criticism of anyone in the Python community, btw.
Typically the experience of Windows users is very well respected by
python core and package developers. But I tend to think that's partly
*because* we didn't take the Unix-like toolchain approach by
default.

 This is not the case at all in the scientific community. NumPy and SciPy
 put in a lot of extra work to come up with something that is compatible
 with the MSVC build of CPython because they have to, not because they're
 happy to jump through the necessary hoops. The situation today with
 NumPy AIUI is they already have to build with a custom hacked MinGW
 toolchain that only one person knows how to use. Evidently until very
 recently they were using a many-years-old 32-bit-only build of GCC 3.x.
 Do python-dev and numpy-discussion not talk to one another? I get that
 not everyone uses numpy, or Windows, but it sounds like there's very
 little understanding, or even awareness, of the issues they face.

Sadly, no. The numpy developers have always been a pretty much
separate world. We're seeing a bit more interaction these days, but
it's very limited and far from the level that's needed. The fault (if
that's the right word) probably lies on both sides. It's certainly not
purely the responsibility of the core team to build communication
links. As this thread has demonstrated, python-dev (and distutils-sig,
which is another place that desparately needs more numpy interaction)
is not the most welcoming of places for someone who is 100% focused on
the scientific stack, but conversely the scientific types do tend to
do their own thing, and sometimes when they do surface, they can dive
in with little appreciation of the wider picture. But we can get
along, and we can make progress (albeit not always as fast as either
party might like).

If all this thread has done is raise awareness of each others'
concerns, it's still been a win IMO.

 I'm going to move the extensions with MinGW-w64 part of this conversation
 over to numpy-discussion, since they have a need for this today and are
 already using workarounds and processes that I don't think anyone is fully
 satisfied with. I do find this combination interesting, worth working on,
 and possible to make work well, but not completely in isolation as it does
 not address my embedding use case.

Please keep distutils-sig in the loop as well. I can't promise we'll
be able to help, but we should at least make sure we're not hindering
you, and we can make you aware of when your solutions won't work
outside your area of interest. Now the door is open, let's not close
it again.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-28 Thread Glenn Linderman


On 10/28/2014 6:45 AM, Stephen J. Turnbull wrote:

because it's a fork, it's a different name
I think this is an important point, and first brought to this discussion 
here. A fork is _not_ called Python, but something else... but if it is 
kept extremely compatible and up-to-date in the hopes of reintegration 
once it proves that it solves a problem, and that there are sufficient 
users, then such hopes seem reasonable.


And targeting the new future compatible MSVCRT sounds like the right 
approach, although it won't solve today's problems today, but it may 
solve tomorrow's problems for a long time into the future... if the 
MinGW people can be convinced to support that new MSVCRT as well.


In addition to all the components that are enabled by MinGW 
(particularly Fortran based extensions), one must remember that the 
current Windows Python also has extensions that interface to MSVC 
libraries that have never been ported to MinGW or Linux, and may never 
be. So an incompatible MinGW-built fork will lose some of those 
extensions... they may not be important to the folks that need MinGW, 
but that would be where  why the community would be split if the MinGW 
fork is not compatible with (some version of MSVC).  Of course, the 
current MSVC-based community is _already_ having issues with 
incompatible versions of MSVC (not limiting that community to Python, 
but broader users of MSVC)... enough problems that even M$ has noticed 
that their incompatibilities are problematical, and are attempting to 
address... not just for Python, but for many other systems and libraries 
as well. So gathering around and supporting their attempts to achieve 
that, by using their new system early, when feedback can still have a 
chance of improving that new future compatible system, will benefit 
everyone... but that time is limited, from what Steve Dower reports of 
the schedule... hoping to be ready for Python 3.5.


So now is an excellent time for this discussion to be happening, and if 
some work can be done to fork, pull the patches together, and tweak them 
to work with the new MSVC, in the Python 3.5 timeframe, you can have a 
phenomenal result, even if it is still a fork at that time.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-28 Thread Stephen J. Turnbull

Tony Kelman writes:

  No, just hearing the words come out of my mouth they sound a little
  nuts.  Maybe not, there are after all half a dozen or more
  incompatible alternate Python implementations floating around. I
  think most of them started as from-scratch rewrites rather than
  source forks, but maybe that's irrelevant.

Well, they have different names and they clearly are not intended to
be ABI compatible, so noone expects that.  OTOH, there clearly is an
expectation among many (and not just in the Windows world, cf. all of
the distros that provide whole stacks of everything for each version
of Python) that downloaded packages will just work without
incompatibility.

   Well, for starters, most of python-dev would rather avoid any contact
   whatsoever with Windows.  I think part of the problem is that Windows
   developers *of* Python are *very* rare (fingers of one hand rare).
  
  In my opinion the MSVC toolchain makes that problem worse, as it's far
  harder for unix developers to have any familiarity with how things
  work.

I've used Cygwin, I've used MinGW, and I've used VC.  Sure, the former
two are GCC-based so I have a lot of muscle memory for command-line
switches.  But that's not very important; the pain of using Windows is
what drives me away from all of them.

  But you do have involvement and core developers from Microsoft
  which is obviously incredibly important. Maybe even mandatory for
  Python on Windows to be viable in your eyes.

No, I don't think that's true.  What I think *is* true is that most
developers on Windows do have access to Microsoft tools, so we do need
to provide compatibility with them.  As you say, the VC toolchain is
not all things to all men, but what's visible to python-dev makes it
more important than Cygwin or MinGW.  See Paul Moore's post about
communications between the scientific Python community and python-dev
for what I mean by visible.

   It should be evident by now that our belief is that the large
   majority of Windows users is well-served by the current model
  
  This is not the case at all in the scientific community. NumPy and
  SciPy put in a lot of extra work to come up with something that is
  compatible with the MSVC build of CPython because they have to, not
  because they're happy to jump through the necessary hoops.

Agreed.  This is well-known to python-dev, and AFAICS it *is* a
concern for us.  However, as Paul points out, a bridge needs to be
built.  Your posts have been a contribution to building that bridge,
for sure, but more work on the bridge is needed.

  Do python-dev and numpy-discussion not talk to one another?

Exactly the issue here.  To resolve this, we need to talk more.
Unfortunately, I'm not one to help build the bridge as I haven't
developed on Windows at all since about 2003.

  I'm going to move the extensions with MinGW-w64 part of this
  conversation over to numpy-discussion,

As far as I can tell, that's a good idea right now.  They have the
need, they have the expertise, both of which are somewhat lacking
here.

  Okay. I'll table the discussion with python-dev for now then.

I hope you'll be able to come pick it back up at some point.  You
might want to interact with Steve Dower off-list, as he's spearheading
the effort to move the official builds to the stable ABI version of
MSVC.  Once that's in place, the MinGW guys will have a stationary
target which is up to date to shoot at.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Nick Coghlan

On 27 October 2014 09:44, Paul Moore p.f.mo...@gmail.com wrote:
 I view it as critical (because availability of binaries is *already*
 enough of a problem in the Windows world, without making it worse)
 that we avoid this sort of fragmentation. I'm not seeing an
 acknowledgement from the mingw side that they agree. That's my
 concern. If we both agree, there's nothing to argue about.

I think there's consensus on this front. From Ray: MinGW-w64 assumes
the very old msvcrt.dll files from
Windows XP SP3 and XP64 specifically to avoid this mess.

That assumption will allow MinGW-w64 to link with the appropriate
MSVCRT versions for extention building without anything breaking.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Nick Coghlan

On 27 October 2014 09:37, Paul Moore p.f.mo...@gmail.com wrote:
 On 26 October 2014 23:24, Tony Kelman kel...@berkeley.edu wrote:
 I want, and in many places *need*, an all-MinGW stack.

 OK, I'm willing to accept that statement. But I don't understand it,
 and I don't think you've explained why you *need* your CPython
 interpreter to be compiled with mingw (as opposed to a number of other
 things you might need around building extensions).

I can take a go at an explanation that may make more sense to you.
Consider one of our key requirements for packaging applications for
Fedora: that Fedora builds be *self-hosting*. Given a base Fedora
system, and a source RPM, we need to be able to *build the binary RPM
from source*. (Other Linux distros generally have a similar
requirement)

Relying on opaque binary blobs downloaded from the internet as part of
the build process is not permitted (modulo a few exceptions for
firmware blobs in device drivers).

Now consider that this automatically rebuild the entire system from
source model is not unique to Linux - you can use it for any system
where your build process is sufficiently automated, and you have a
need for it. However, the *structure* of those kind of automation
tends to differ wildly between POSIX style tooling (gcc, clang) and
MSVC. If you have an existing build automation system for *nix
targets, then cross-compilation via MinGW is likely going to be your
smoothest path to adding Windows binary support.

At that point, if CPython is one of your dependencies, you're going to
have the choice of allowing the python.org binaries to be pulled in as
opaque pre-built blobs, or else figuring out how to build an ABI
compatible version with MinGW rather than with MSVC. Think of this
more in the case of *embedding* the CPython runtime in a larger
context (e.g. in Tony's case, to make Python software usable with the
Julia runtime), rather than in building a standalone Python
interpreter for general use.

So, for embedding cases, and for incorporation into POSIX-style build
systems using MinGW-w64 for cross-compilation of Windows binaries, it
may make sense to incorporate the patches that allow building with
MinGW-w64 into mainline CPython (if I recall correctly, we supported
building with Intel's C compiler for a long time, even though we never
shipped anything built with it).

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 27 October 2014 12:30, Nick Coghlan ncogh...@gmail.com wrote:
 OK, I'm willing to accept that statement. But I don't understand it,
 and I don't think you've explained why you *need* your CPython
 interpreter to be compiled with mingw (as opposed to a number of other
 things you might need around building extensions).

 I can take a go at an explanation that may make more sense to you.
 Consider one of our key requirements for packaging applications for
 Fedora: that Fedora builds be *self-hosting*. Given a base Fedora
 system, and a source RPM, we need to be able to *build the binary RPM
 from source*. (Other Linux distros generally have a similar
 requirement)

 Relying on opaque binary blobs downloaded from the internet as part of
 the build process is not permitted (modulo a few exceptions for
 firmware blobs in device drivers).

 Now consider that this automatically rebuild the entire system from
 source model is not unique to Linux - you can use it for any system
 where your build process is sufficiently automated, and you have a
 need for it. However, the *structure* of those kind of automation
 tends to differ wildly between POSIX style tooling (gcc, clang) and
 MSVC. If you have an existing build automation system for *nix
 targets, then cross-compilation via MinGW is likely going to be your
 smoothest path to adding Windows binary support.

 At that point, if CPython is one of your dependencies, you're going to
 have the choice of allowing the python.org binaries to be pulled in as
 opaque pre-built blobs, or else figuring out how to build an ABI
 compatible version with MinGW rather than with MSVC. Think of this
 more in the case of *embedding* the CPython runtime in a larger
 context (e.g. in Tony's case, to make Python software usable with the
 Julia runtime), rather than in building a standalone Python
 interpreter for general use.

 So, for embedding cases, and for incorporation into POSIX-style build
 systems using MinGW-w64 for cross-compilation of Windows binaries, it
 may make sense to incorporate the patches that allow building with
 MinGW-w64 into mainline CPython (if I recall correctly, we supported
 building with Intel's C compiler for a long time, even though we never
 shipped anything built with it).

Thanks Nick. That explanation makes sense to me. I was aware of this
sort of scenario, and as I've said before I don't have any objection
per se to making things easier for people with that sort of
requirement. But some of the other arguments in this thread seemed to
imply more than that. Without specifics, though, I concede that I may
be over-interpreting the rhetoric, so that's the part of the debate
I'm stepping back from, to avoid descending into FUD.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 26 October 2014 01:05, Ray Donnelly mingw.andr...@gmail.com wrote:
 Download and run:
 http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download

Sending this offline because I really don't want to start up another
extended debate, but is there a version of this that I can use that
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

Please ignore this. I hit the wrong button.

On 27 October 2014 14:18, Paul Moore p.f.mo...@gmail.com wrote:
 On 26 October 2014 01:05, Ray Donnelly mingw.andr...@gmail.com wrote:
 Download and run:
 http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download

 Sending this offline because I really don't want to start up another
 extended debate, but is there a version of this that I can use that
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 26 October 2014 23:44, Paul Moore p.f.mo...@gmail.com wrote:
 On 26 October 2014 23:11, Ray Donnelly mingw.andr...@gmail.com wrote:
 I don't know where this ABI compatible thing came into being;

 Simple. If a mingw-built CPython doesn't work with the same extensions
 as a MSVC-built CPython, then the community gets fragmented (because
 you can only use the extensions built for your stack). Assuming numpy
 needs mingw and ultimately only gets built for a mingw-compiled Python
 (because the issues building for MSVC-built Python are too hard) and
 assuming that nobody wants to make the effort to build pywin32 under
 mingw, then what does someone who needs both numpy and pywin32 do?

 Avoiding that issue is what I mean by ABI-compatible. (And that's all
 I mean by it, nothing more subtle or controversial).

 I view it as critical (because availability of binaries is *already*
 enough of a problem in the Windows world, without making it worse)
 that we avoid this sort of fragmentation. I'm not seeing an
 acknowledgement from the mingw side that they agree. That's my
 concern. If we both agree, there's nothing to argue about.

I have just done some experiments with building CPython extensions
with mingw-w64. Thanks to Ray for helping me set this up.

The bad news is that the support added to the old 32-bit mingw to
support linking to alternative C runtime libraries (specifically
-lmsvcr100) has bitrotted, and no longer functions correctly in
mingw-w64. As a result, not only can mingw-w64 not build extensions
that are compatible with python.org Python, it can't build extensions
that function at all [1]. They link incompatibly to *both* msvcrt and
msvcr100.

This is a bug in mingw-w64. I have reported it to Ray, who's passed it
onto one of the mingw-w64 developers. But as things stand, mingw
builds will definitely produce binary extensions that aren't
compatible with python.org Python.

Paul

[1] Note, that's if you just use --compiler=mingw32 as supported by
distutils. Looking at how the numpy folks build, they seem to hack
their own version of the distutils C compiler classes. I don't know
whether that's just to work around this bug, or whether they do it for
other reasons as well (but I suspect the latter).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Case Van Horsen

On Mon, Oct 27, 2014 at 10:48 AM, Paul Moore p.f.mo...@gmail.com wrote:
 The bad news is that the support added to the old 32-bit mingw to
 support linking to alternative C runtime libraries (specifically
 -lmsvcr100) has bitrotted, and no longer functions correctly in
 mingw-w64. As a result, not only can mingw-w64 not build extensions
 that are compatible with python.org Python, it can't build extensions
 that function at all [1]. They link incompatibly to *both* msvcrt and
 msvcr100.

 This is a bug in mingw-w64. I have reported it to Ray, who's passed it
 onto one of the mingw-w64 developers. But as things stand, mingw
 builds will definitely produce binary extensions that aren't
 compatible with python.org Python.

 Paul

 [1] Note, that's if you just use --compiler=mingw32 as supported by
 distutils. Looking at how the numpy folks build, they seem to hack
 their own version of the distutils C compiler classes. I don't know
 whether that's just to work around this bug, or whether they do it for
 other reasons as well (but I suspect the latter).
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/casevh%40gmail.com

I've managed to build gmpy2 (which requires GMP, MPFR, and MPC
libraries) using msys2. I've detailed the steps (hacking) at:

https://code.google.com/p/gmpy/source/browse/trunk/msys2_build.txt

One of the hacks I made addresses the linking bug. The extension
does run with the both the 32-bit and 64-bit versions of CPython 2.7,
3.2, 3.3, and 3.4.

It is possible, just not easy. Anything that makes is easier would
be very helpful.

casevh
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 27 October 2014 18:47, Case Van Horsen cas...@gmail.com wrote:
 I've managed to build gmpy2 (which requires GMP, MPFR, and MPC
 libraries) using msys2. I've detailed the steps (hacking) at:

 https://code.google.com/p/gmpy/source/browse/trunk/msys2_build.txt

Thanks for this. I don't have the time to read your notes right now,
but I will do so.

 One of the hacks I made addresses the linking bug. The extension
 does run with the both the 32-bit and 64-bit versions of CPython 2.7,
 3.2, 3.3, and 3.4.

Did you report the linking bug to the mingw-w64 project? They key
thing here is that without gcc -lmsvcrt100 foo.c working (i.e., not
resulting in linking with msvcrt), building Python extensions will
always need hacks to workaround that bug.

 It is possible, just not easy. Anything that makes is easier would
 be very helpful.

With the bug fixed, the steps should be as trivial as:

1. Using python.org Python, with gcc on your PATH.
2. Install any dependencies (e.g., gmp) where gcc can see them.
3. python setup.py build_ext --compiler=mingw32 bdist_wheel

(or whatever setup.py invocation suits you, as long as you set
compiler=mingw32).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Ray Donnelly

On Sun, Oct 26, 2014 at 11:52 PM,  mar...@v.loewis.de wrote:

 Zitat von Tony Kelman kel...@berkeley.edu:

 A maintainer has volunteered. Others will help. Can any core developers
 please begin reviewing some of his patches?


 Unfortunately, every attempt to review these patches has failed for me,
 every time. In the last iteration of an attempt to add mingw64 support,
 I had asked contributors to also provide instructions on how to use these
 patches, and haven't received any instructions that actually worked.

 I'm hesitant to add code that I cannot verify as actually working.

 I guess anybody else reviewing these patches ran into similar problems
 (I know some other core developers have tried reviewing them as well,
 others have stated here that they are unable to review the patches).


https://mail.python.org/pipermail/python-dev/2014-October/136756.html

 Regards,
 Martin



 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Nathaniel Smith

On Mon, Oct 27, 2014 at 5:48 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 26 October 2014 23:44, Paul Moore p.f.mo...@gmail.com wrote:
 On 26 October 2014 23:11, Ray Donnelly mingw.andr...@gmail.com wrote:
 I don't know where this ABI compatible thing came into being;

 Simple. If a mingw-built CPython doesn't work with the same extensions
 as a MSVC-built CPython, then the community gets fragmented (because
 you can only use the extensions built for your stack). Assuming numpy
 needs mingw and ultimately only gets built for a mingw-compiled Python
 (because the issues building for MSVC-built Python are too hard) and
 assuming that nobody wants to make the effort to build pywin32 under
 mingw, then what does someone who needs both numpy and pywin32 do?

 Avoiding that issue is what I mean by ABI-compatible. (And that's all
 I mean by it, nothing more subtle or controversial).

 I view it as critical (because availability of binaries is *already*
 enough of a problem in the Windows world, without making it worse)
 that we avoid this sort of fragmentation. I'm not seeing an
 acknowledgement from the mingw side that they agree. That's my
 concern. If we both agree, there's nothing to argue about.

 I have just done some experiments with building CPython extensions
 with mingw-w64. Thanks to Ray for helping me set this up.

 The bad news is that the support added to the old 32-bit mingw to
 support linking to alternative C runtime libraries (specifically
 -lmsvcr100) has bitrotted, and no longer functions correctly in
 mingw-w64. As a result, not only can mingw-w64 not build extensions
 that are compatible with python.org Python, it can't build extensions
 that function at all [1]. They link incompatibly to *both* msvcrt and
 msvcr100.

 This is a bug in mingw-w64. I have reported it to Ray, who's passed it
 onto one of the mingw-w64 developers. But as things stand, mingw
 builds will definitely produce binary extensions that aren't
 compatible with python.org Python.

IIUC, getting mingw-w64 to link against msvcr100 instead of msvcrt
requires a custom mingw-w64 build, because by default mingw-w64's
internal runtime libraries (libgcc etc.) are linked against msvcrt. So
by the time you're choosing compiler switches etc., it's already too
late -- your switches might affect how *your* code is built, but your
code will still be linked against pre-existing runtime libraries that
are linked against msvcrt.

It's possible to hack the mingw-w64 build process to build the runtime
libraries against msvcr100 (or whatever) instead of msvcrt, but this
is still not a panacea -- the different msvcr* libraries are, of
course, incompatible with each other, and IIUC the mingw-w64
developers have never tried to make their libraries work against
anything except msvcrt. For example, mingw-w64's gfortran runtime uses
a symbol that's only available in msvcrt, not msvcr90 or msvcrt100:
  http://sourceforge.net/p/mingw-w64/mailman/message/31768118/

So my impression is that these issues are all fixable, but they will
require real engagement with mingw-w64 upstream.

 [1] Note, that's if you just use --compiler=mingw32 as supported by
 distutils. Looking at how the numpy folks build, they seem to hack
 their own version of the distutils C compiler classes. I don't know
 whether that's just to work around this bug, or whether they do it for
 other reasons as well (but I suspect the latter).

numpy.distutils is a massive pile of hacks to handle all kinds of
weird things including recursive builds, fortran, runtime capability
detection (like autoconf), and every random issue anyone ran into at
some point in the last 10 years and couldn't be bothered filing a
proper upstream bug report. Basically no-one knows what it actually
does -- the source is your only hope :-).

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Greg Ewing


Nick Coghlan wrote:

That assumption will allow MinGW-w64 to link with the appropriate
MSVCRT versions for extention building without anything breaking.


If that works, then the same technique should allow CPython
itself to be built in a VS-compatible way with mingw,
shouldn't it?

Those objecting to a mingw-built python seem to be assuming
that such a thing will necessarily be incompatible with
VS builds, but I don't see why that has to be the case.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 27 October 2014 20:45, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
 Nick Coghlan wrote:

 That assumption will allow MinGW-w64 to link with the appropriate
 MSVCRT versions for extention building without anything breaking.


 If that works, then the same technique should allow CPython
 itself to be built in a VS-compatible way with mingw,
 shouldn't it?

Yes.

 Those objecting to a mingw-built python seem to be assuming
 that such a thing will necessarily be incompatible with
 VS builds, but I don't see why that has to be the case.

No, we've been trying to establish whether the patches to build with
mingw were intended to produce such a compatible build. It's not
clear, but so far it seems that apparently that is *not* the intent
(and worse, mingw-w64 may not even be able to build viable executables
that link with msvcr100 without some heavy hacking, although that's
still somewhat unclear).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Steve Dower

Greg Ewing wrote:
 Nick Coghlan wrote:
 That assumption will allow MinGW-w64 to link with the appropriate
 MSVCRT versions for extention building without anything breaking.

 If that works, then the same technique should allow CPython itself to be built
 in a VS-compatible way with mingw, shouldn't it?

 Those objecting to a mingw-built python seem to be assuming that such a thing
 will necessarily be incompatible with VS builds, but I don't see why that has 
 to
 be the case.

That's true, and a good point that I missed. However, the main (practical) 
desire for building CPython with something other than VS seems to be to avoid 
having to be compatible with VS.

It's entirely possible that having two alternative builds of CPython would 
force everyone to be more compatible, but I think it's more likely to simply 
end up being two different worlds. Maybe I'm being unnecessarily cynical :)

Cheers,
Steve

 --
 Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Steve Dower

 Paul Moore wrote:
 On 27 October 2014 20:45, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
 Nick Coghlan wrote:

 That assumption will allow MinGW-w64 to link with the appropriate
 MSVCRT versions for extention building without anything breaking.


 If that works, then the same technique should allow CPython itself to
 be built in a VS-compatible way with mingw, shouldn't it?
 
 Yes.
 
 Those objecting to a mingw-built python seem to be assuming that such
 a thing will necessarily be incompatible with VS builds, but I don't
 see why that has to be the case.
 
 No, we've been trying to establish whether the patches to build with mingw 
 were
 intended to produce such a compatible build. It's not clear, but so far it 
 seems
 that apparently that is *not* the intent (and worse, mingw-w64 may not even be
 able to build viable executables that link with msvcr100 without some heavy
 hacking, although that's still somewhat unclear).

Unless there is also opposition to moving to VC14, I'd rather see the mingw 
projects invest in linking to those libraries. I believe they'll have a much 
easier time of it than worrying about VC10, and the investment will be worth 
more in the future as the public API of the CRT stops changing.

Unfortunately, I'm not able to help out more than I've already offered 
(researching answers to specific questions). Largely because I have enough 
work-outside-work going on, but also because my employer won't like me getting 
involved with GPL'd software at all.

Cheers,
Steve

 Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Ray Donnelly

On Mon, Oct 27, 2014 at 8:54 PM, Steve Dower steve.do...@microsoft.com wrote:
 Greg Ewing wrote:
 Nick Coghlan wrote:
 That assumption will allow MinGW-w64 to link with the appropriate
 MSVCRT versions for extention building without anything breaking.

 If that works, then the same technique should allow CPython itself to be 
 built
 in a VS-compatible way with mingw, shouldn't it?

 Those objecting to a mingw-built python seem to be assuming that such a thing
 will necessarily be incompatible with VS builds, but I don't see why that 
 has to
 be the case.

 That's true, and a good point that I missed. However, the main (practical) 
 desire for building CPython with something other than VS seems to be to avoid 
 having to be compatible with VS.

I've no idea where you get that impression from, no one has expressed
anything even approximating that. For me it's to avoid using closed
source software for my hobbyist programming and to help to create a
vibrant Open Source distribution for Windows, because I quite like
Windows; it's got a lot going for it. For others it's to ensure that
everything they care about (CPython with Fortran for example) works
together properly and reliably. I expect that avoiding compatibility
couldn't be further from any of our wishes.


 It's entirely possible that having two alternative builds of CPython would 
 force everyone to be more compatible, but I think it's more likely to simply 
 end up being two different worlds. Maybe I'm being unnecessarily cynical :)

 Cheers,
 Steve

 --
 Greg
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 27 October 2014 21:19, Steve Dower steve.do...@microsoft.com wrote:
 No, we've been trying to establish whether the patches to build with mingw 
 were
 intended to produce such a compatible build. It's not clear, but so far it 
 seems
 that apparently that is *not* the intent (and worse, mingw-w64 may not even 
 be
 able to build viable executables that link with msvcr100 without some heavy
 hacking, although that's still somewhat unclear).

 Unless there is also opposition to moving to VC14, I'd rather see the mingw
 projects invest in linking to those libraries. I believe they'll have a much 
 easier
 time of it than worrying about VC10, and the investment will be worth more in
 the future as the public API of the CRT stops changing.

I think the point is that anything other than msvcrt is extra work,
because using msvcrt is coded into the guts of gcc (which in turn is
because msvcrt is apparently OK to consider as part of the OS in GPL
legality terms). So whether it's the vc10 libraries or the vc14 ones
is irrelevant - and mingw ships with the vc10 link library, so it's
easier to discuss the problem in terms of vc10. But yes, vc14 would be
the long term target.

Of course if the vc14 libs were deemed as shipped with the OS and/or
were named msvcrt.dll, then that would be different. But I assume
that's not what will happen.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Devin Jeanpierre

On Sun, Oct 26, 2014 at 3:41 PM, Paul Moore p.f.mo...@gmail.com wrote:
 Not really, to be honest. I still don't understand why anyone not
 directly involved in CPython development would need to build their own
 Python executable on Windows.

Late Python bugfix releases are source-only, so if you suffer from a
bug and need to get it fixed, you need to build Python from source.

https://www.python.org/download/releases/2.6.9/ has no windows binary
and includes several security fixes.

-- Devin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Stephen J. Turnbull

R. David Murray writes:
  On Sun, 26 Oct 2014 00:19:44 +0200, Antoine Pitrou solip...@pitrou.net 
  wrote:

   My point is that your Windows build would not have the same behaviour
   as a MSVC-produced Windows build, and so testing it with it would not
   certify that your code would actually be compatible with genuine
   MSVC builds of CPython, which we will not stop supporting.
  
  While true, I don't think that matters for Chris' point.

[...]

  If I could use a more linux-like toolchain to build CPython on windows,
  I would doubtless do much more testing on windows for stuff where I
  think windows might behave differently (and I might look at more Windows
  bugs...though frankly there are plenty of bugs for me to look at without
  looking at Windows bugs).
  
  This is not necessarily a compelling argument for MinGW support.
  However, it *is* a valid argument, IMO.

Nobody claims that the there are not arguments, even compelling
arguments, for MinGW support (more generally, support for alternative
toolchains).

But there are *also* compelling arguments for *supporting* *both*
those no need to worry about mixed ABIs situations and *mixed*
situations.  And that becomes Python Dev's problem if the patches are
added to core Python.  Currently, they're somebody else's problem, and
that's as it should be at this stage.

Python is open source.  Nobody is objecting to somebody else doing
this.[1]  The problem here is simply that some somebody elses are
trying to throw future work over the wall into python-dev space.
There is nothing wrong with that, either -- that's why there is a
stdlib, for example -- but the python-dev concerns about platform
fragmentation are genuine (even if not applicable to all potential
users of the alternative toolchains), and substantial resources will
be needed to do the testing required to meet python-dev's requirement
that such code be *binary* compatible with other binaries downloaded
for Windows, as well as for maintenance of the code itself.


Footnotes: 
[1]  Some *do* question whether there's a need for anybody to do this,
and that's bogus.  I just wanna is good enough reason to do it.  The
issue here is that it's not good enough reason for python-dev to do
the support and maintenance going forward.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Steve Dower

Ray Donnelly wrote:
 On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 25 October 2014 23:22, Chris Angelico ros...@gmail.com wrote:
 On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou solip...@pitrou.net wrote:
 My point is that your Windows build would not have the same behaviour
 as a MSVC-produced Windows build, and so testing it with it would not
 certify that your code would actually be compatible with genuine
 MSVC builds of CPython, which we will not stop supporting.


 So you're saying it's impossible to support two compilers?

 No, rather that Windows currently only has a single supported compiler
 (excluding cygwin, which is essentially a different OS). Adding a
 second compiler doesn't just involve adding support for it - which is
 all that the people offering mingw patches are doing - but also
 involves going through the whole CPython ecosystem locating the places
 where there is an implicit assumption that all Windows builds use the
 same compiler and fixing them. I've already pointed out where this is
 a question for pip and wheel. Whoever wants to add support for a
 second compiler needs to be willing to do this part of the job as
 well.

 Handwaving arguments that it's binary compatible aren't enough. Prove it.
 
 The msvcrt.dlls that MinGW-w64 depends on are those dating back to
 Windows XP SP3 / XP64. Ironically, the official Windows CPython
 doesn't come with any such crt guarantees and you must ensure that the
 same msvcr??.dll is used for *all* extensions. This puts considerable
 strain on extension developers to use the correct (or any) version of
 Visual Studio to build their extensions for CPython on Windows.

We're well aware of this, and it's a big part of why I'm currently migrating 
CPython to build with VC14, which will not have the same binary compatibility 
issues. For VC14, the entire CRT has been cleaned up and mostly hidden behind 
calls into DLLs, so provided the calling conventions match (which they must or 
everything would crash very quickly), it should be relatively easy to build 
compatible extensions with MinGW-w64.

 Also, where are the publicly accessible specifications and other technical
 descriptions that MinGW-w64 would need to implement strong binary
 compatibility with MSVC? As a random example, those for C++ name
 mangling and the PDB file format would be very helpful.

C++ name mangling is always an implementation detail and it changes from 
version to version. Luckily, CPython is entirely in C, so that doesn't matter. 
PDBs are another red herring - you can build a loadable PE file without PDBs.

The full source code for the MSVCRT is available with any version of Visual 
Studio (including the free editions, last time I checked), so feel free to 
check whatever you need to ensure compatibility. I've suggested to the VC team 
that they could get in touch with the MinGW projects and offer to help them 
improve compatibility with MSVC, but unfortunately I don't think anyone will 
take me up on that. I'm happy to research what I can to answer specific 
questions, but there's very little that isn't already publicly available other 
than direct access to the devs.

Cheers,
Steve

 Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Tony Kelman

Thanks all for the responses. Clearly this is a subject about which
people feel strongly, so that's good at least. David Murray's guidance
in particular points to the most likely path to get improvements to
really happen.

Steve Dower:
 Building CPython for Windows is not something that needs solving.

Not in your opinion, but numerous packagers of MinGW-based native or
cross-compiled package sets would love to include Python. The fact
that they currently can't, without many patches, is a problem.

 The culture on Windows is to redistribute binaries, not source,

There are many cultures using Windows. Including open-source ones.

 and both the core team and a number of redistributors have this figured
 out (and it will only become easier with VC14 and Python 3.5).

With MSVC. It doesn't work with MinGW, it likely doesn't work with Clang.
MSVC is not the only compiler on Windows. There are many use cases for
preferring other compilers. Have you read this wiki page for example?
https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows

In my personal experience, having recently gotten Julia to compile using
MSVC for the first time, MSVC as a compiler is highly deficient for many
needs especially in the scientific software community:
- C99 (getting better recently, but still not done)
- ATT syntax assembly
- C++11 features (also mostly okay now, but not if you're using an older
  MSVC version with Python 2.7, which many people still have to do)
- 128-bit integer intrinsics
- cannot cross-compile from anything that isn't Windows
- build systems foreign relative to shell/makefile systems used by most
  open-source projects, few projects have time to maintain 2 separate build
  systems (cmake helps but takes a lot of initial effort to convert to)
- no free-as-in-beer Fortran compiler available

I have none of these problems when I use MinGW-w64. Hence the desire to
be able to curate an all-MinGW software stack. It's not a matter of open-
source ideology for me, it's brass tacks can I do the work I need to do.
With MSVC I can't, with MinGW-w64 I can. Not being able to include CPython
in an all-MinGW stack hurts, a lot.

Only cross-compilation and the build system in the above list are relevant
to CPython, but I hope I have convinced you, Paul Moore, etc. that there are
real reasons for some groups of users and developers to prefer MinGW-w64
over MSVC.

 I'd rather see this effort thrown behind compiling extensions,
 including cross compilation.

There are patches awaiting review that improve this as well. Efforts to
improve CPython's build system and the handling of extensions are not
completely independent, in many cases the patches are written by the same
set of MinGW users. One of these sets of patches is not inherently evil,
you understandably have less interest in them but it's still disappointing
to see so little movement on either.

 Having different builds of CPython out there will only fragment the
 community and hurt extension authors far more than it may seem to help.

The community of people developing and using open-source projects, either
CPython or otherwise, is already highly fragmented. Ignoring it makes it
worse. python.org does not have to distribute or endorse MinGW-compiled
builds of CPython. If the build option never gets incorporated, then it
will continue to be reverse-engineered.

Guido van Rossum:
 Here's the crux of the matter. We want compiled extension modules
 distributed via PyPI to work with the binaries distributed from python.org.

Absolutely. I don't think additional options in the build system would
change this.

R. David Murray:
 And, at this point, we would NEED A BUILDBOT.  That is, a machine that
 has whatever tools are required installed such that tests added to the
 test suite to test MinGW support in distutils would run, so we can be
 sure we don't break anything when making other changes.

That's not too hard. I've done this for other projects. AppVeyor works if
your build is short enough, and I've done cross-compilation from Travis
CI for other projects. Or Jenkins, or a Vagrant VM. I don't know PSF's
infrastructure, but I can offer guidance if it would help.

Steve Dower:
 I'm afraid of users having numpy crash because they're using an MSVC
 CPython instead of a mingw CPython. I'm afraid of users not being able
 to use library A and library B at the same time because A requires MSVC
 CPython and B requires mingw CPython. (I can produce more examples if you
 like, but the general concern is having a fragmented community, as I said
 in my previous post.)

A valid fear. Mixing C runtimes can cause problems, I've seen this myself.
Correct me if I'm wrong, but this is nearly as much of an issue if someone
wants to use a different version of MSVC to compile CPython than the version
used to build the official binaries. It requires care, but you can't deny
that there are use cases where people will want and need to do such things.
Is possible fragmentation a good

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Ray Donnelly

On Sun, Oct 26, 2014 at 1:12 PM, Tony Kelman kel...@berkeley.edu wrote:
 Thanks all for the responses. Clearly this is a subject about which
 people feel strongly, so that's good at least. David Murray's guidance
 in particular points to the most likely path to get improvements to
 really happen.

 Steve Dower:
 Building CPython for Windows is not something that needs solving.

 Not in your opinion, but numerous packagers of MinGW-based native or
 cross-compiled package sets would love to include Python. The fact
 that they currently can't, without many patches, is a problem.

 The culture on Windows is to redistribute binaries, not source,

 There are many cultures using Windows. Including open-source ones.

 and both the core team and a number of redistributors have this figured
 out (and it will only become easier with VC14 and Python 3.5).

 With MSVC. It doesn't work with MinGW, it likely doesn't work with Clang.
 MSVC is not the only compiler on Windows. There are many use cases for
 preferring other compilers. Have you read this wiki page for example?
 https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows

 In my personal experience, having recently gotten Julia to compile using
 MSVC for the first time, MSVC as a compiler is highly deficient for many
 needs especially in the scientific software community:
 - C99 (getting better recently, but still not done)
 - ATT syntax assembly
 - C++11 features (also mostly okay now, but not if you're using an older
   MSVC version with Python 2.7, which many people still have to do)
 - 128-bit integer intrinsics
 - cannot cross-compile from anything that isn't Windows
 - build systems foreign relative to shell/makefile systems used by most
   open-source projects, few projects have time to maintain 2 separate build
   systems (cmake helps but takes a lot of initial effort to convert to)
 - no free-as-in-beer Fortran compiler available

 I have none of these problems when I use MinGW-w64. Hence the desire to
 be able to curate an all-MinGW software stack. It's not a matter of open-
 source ideology for me, it's brass tacks can I do the work I need to do.
 With MSVC I can't, with MinGW-w64 I can. Not being able to include CPython
 in an all-MinGW stack hurts, a lot.

 Only cross-compilation and the build system in the above list are relevant
 to CPython, but I hope I have convinced you, Paul Moore, etc. that there are
 real reasons for some groups of users and developers to prefer MinGW-w64
 over MSVC.

 I'd rather see this effort thrown behind compiling extensions,
 including cross compilation.

 There are patches awaiting review that improve this as well. Efforts to
 improve CPython's build system and the handling of extensions are not
 completely independent, in many cases the patches are written by the same
 set of MinGW users. One of these sets of patches is not inherently evil,
 you understandably have less interest in them but it's still disappointing
 to see so little movement on either.

 Having different builds of CPython out there will only fragment the
 community and hurt extension authors far more than it may seem to help.

 The community of people developing and using open-source projects, either
 CPython or otherwise, is already highly fragmented. Ignoring it makes it
 worse. python.org does not have to distribute or endorse MinGW-compiled
 builds of CPython. If the build option never gets incorporated, then it
 will continue to be reverse-engineered.

 Guido van Rossum:
 Here's the crux of the matter. We want compiled extension modules
 distributed via PyPI to work with the binaries distributed from
 python.org.

 Absolutely. I don't think additional options in the build system would
 change this.

 R. David Murray:
 And, at this point, we would NEED A BUILDBOT.  That is, a machine that
 has whatever tools are required installed such that tests added to the
 test suite to test MinGW support in distutils would run, so we can be
 sure we don't break anything when making other changes.

 That's not too hard. I've done this for other projects. AppVeyor works if
 your build is short enough, and I've done cross-compilation from Travis
 CI for other projects. Or Jenkins, or a Vagrant VM. I don't know PSF's
 infrastructure, but I can offer guidance if it would help.

 Steve Dower:
 I'm afraid of users having numpy crash because they're using an MSVC
 CPython instead of a mingw CPython. I'm afraid of users not being able
 to use library A and library B at the same time because A requires MSVC
 CPython and B requires mingw CPython. (I can produce more examples if you
 like, but the general concern is having a fragmented community, as I said
 in my previous post.)

 A valid fear. Mixing C runtimes can cause problems, I've seen this myself.
 Correct me if I'm wrong, but this is nearly as much of an issue if someone
 wants to use a different version of MSVC to compile CPython than the version
 used to build the official binaries. It requires care,

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread R. David Murray

On Sun, 26 Oct 2014 06:12:45 -0700, Tony Kelman kel...@berkeley.edu wrote:
 Steve Dower:
  Building CPython for Windows is not something that needs solving.
 
 Not in your opinion, but numerous packagers of MinGW-based native or
 cross-compiled package sets would love to include Python. The fact
 that they currently can't, without many patches, is a problem.

If this includes (or would likely include) a significant portion of the
Scientific Computing community, I would think that would be a compelling
use case.  We'd need to be educated more about the reasons why this
approach works better than remaining compatible with MSVC CPython so we
could evaluate the risks and reward intelligently.  (I wonder...things
are going to fragment anyway if you (cpython) don't do anything might
be an argument, if true...but wouldn't make the consequences any
easier to deal with :(

But as has been discussed, it seems better to focus first on fixing the
issues on which we are all in agreement (building extensions with MinGW).

 R. David Murray:
  And, at this point, we would NEED A BUILDBOT.  That is, a machine that
  has whatever tools are required installed such that tests added to the
  test suite to test MinGW support in distutils would run, so we can be
  sure we don't break anything when making other changes.
 
 That's not too hard. I've done this for other projects. AppVeyor works if
 your build is short enough, and I've done cross-compilation from Travis
 CI for other projects. Or Jenkins, or a Vagrant VM. I don't know PSF's
 infrastructure, but I can offer guidance if it would help.

When I say we need a buildbot, what I mean is that we need someone
willing to donate the resources and the *time and expertise* to setting
up and maintaining something that integrates with our existing buildbot
setup.  You set up a buildbot slave, request an id and password from
Antoine, keep the thing running, and respond in a timely fashion to
requests for help resolving issues that arise on the buildbot (both
buildbot issues and help-me-diagnose-this-failure issues).  After the
initial setup the load isn't generally heavy (I haven't had to do
anything with the OSX buildbot running on the machine in my dinning room
for months and months now, for example).

So your guidance would have to go to someone who was volunteering to
take on this task...there isn't anyone on the existing core team who
would have time to do it (if I'm wrong, I'm sure someone will speak up).
On the other hand, you don't have to be a committer to run a buildbot,
and there *are* people on the core-mentorship list who have expressed
interest in helping out with our automated testing infrastructure,
including (if I understand correctly) adding some level of integration
to other CI systems (which might just be messages to the IRC
channel)[*].  So that could be a fruitful avenue to explore.

--David

[*] This is an area in which I have an interest, but it hasn't gotten
high enough on my todo list yet for me to figure out exactly what the
current state of things is so I can help it along.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Ray Donnelly

On Sun, Oct 26, 2014 at 2:28 PM, Ray Donnelly mingw.andr...@gmail.com wrote:
 On Sun, Oct 26, 2014 at 1:12 PM, Tony Kelman kel...@berkeley.edu wrote:
 Thanks all for the responses. Clearly this is a subject about which
 people feel strongly, so that's good at least. David Murray's guidance
 in particular points to the most likely path to get improvements to
 really happen.

 Steve Dower:
 Building CPython for Windows is not something that needs solving.

 Not in your opinion, but numerous packagers of MinGW-based native or
 cross-compiled package sets would love to include Python. The fact
 that they currently can't, without many patches, is a problem.

 The culture on Windows is to redistribute binaries, not source,

 There are many cultures using Windows. Including open-source ones.

 and both the core team and a number of redistributors have this figured
 out (and it will only become easier with VC14 and Python 3.5).

 With MSVC. It doesn't work with MinGW, it likely doesn't work with Clang.
 MSVC is not the only compiler on Windows. There are many use cases for
 preferring other compilers. Have you read this wiki page for example?
 https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows

 In my personal experience, having recently gotten Julia to compile using
 MSVC for the first time, MSVC as a compiler is highly deficient for many
 needs especially in the scientific software community:
 - C99 (getting better recently, but still not done)
 - ATT syntax assembly
 - C++11 features (also mostly okay now, but not if you're using an older
   MSVC version with Python 2.7, which many people still have to do)
 - 128-bit integer intrinsics
 - cannot cross-compile from anything that isn't Windows
 - build systems foreign relative to shell/makefile systems used by most
   open-source projects, few projects have time to maintain 2 separate build
   systems (cmake helps but takes a lot of initial effort to convert to)
 - no free-as-in-beer Fortran compiler available

 I have none of these problems when I use MinGW-w64. Hence the desire to
 be able to curate an all-MinGW software stack. It's not a matter of open-
 source ideology for me, it's brass tacks can I do the work I need to do.
 With MSVC I can't, with MinGW-w64 I can. Not being able to include CPython
 in an all-MinGW stack hurts, a lot.

 Only cross-compilation and the build system in the above list are relevant
 to CPython, but I hope I have convinced you, Paul Moore, etc. that there are
 real reasons for some groups of users and developers to prefer MinGW-w64
 over MSVC.

 I'd rather see this effort thrown behind compiling extensions,
 including cross compilation.

 There are patches awaiting review that improve this as well. Efforts to
 improve CPython's build system and the handling of extensions are not
 completely independent, in many cases the patches are written by the same
 set of MinGW users. One of these sets of patches is not inherently evil,
 you understandably have less interest in them but it's still disappointing
 to see so little movement on either.

 Having different builds of CPython out there will only fragment the
 community and hurt extension authors far more than it may seem to help.

 The community of people developing and using open-source projects, either
 CPython or otherwise, is already highly fragmented. Ignoring it makes it
 worse. python.org does not have to distribute or endorse MinGW-compiled
 builds of CPython. If the build option never gets incorporated, then it
 will continue to be reverse-engineered.

 Guido van Rossum:
 Here's the crux of the matter. We want compiled extension modules
 distributed via PyPI to work with the binaries distributed from
 python.org.

 Absolutely. I don't think additional options in the build system would
 change this.

 R. David Murray:
 And, at this point, we would NEED A BUILDBOT.  That is, a machine that
 has whatever tools are required installed such that tests added to the
 test suite to test MinGW support in distutils would run, so we can be
 sure we don't break anything when making other changes.

 That's not too hard. I've done this for other projects. AppVeyor works if
 your build is short enough, and I've done cross-compilation from Travis
 CI for other projects. Or Jenkins, or a Vagrant VM. I don't know PSF's
 infrastructure, but I can offer guidance if it would help.

 Steve Dower:
 I'm afraid of users having numpy crash because they're using an MSVC
 CPython instead of a mingw CPython. I'm afraid of users not being able
 to use library A and library B at the same time because A requires MSVC
 CPython and B requires mingw CPython. (I can produce more examples if you
 like, but the general concern is having a fragmented community, as I said
 in my previous post.)

 A valid fear. Mixing C runtimes can cause problems, I've seen this myself.
 Correct me if I'm wrong, but this is nearly as much of an issue if someone
 wants to use a different version of MSVC to compile

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Tony Kelman


If this includes (or would likely include) a significant portion of the
Scientific Computing community, I would think that would be a compelling
use case.


I can't speak for any of the scientific computing community besides myself,
but my thoughts: much of the development, as you know, happens on Linux
with GCC (or OSX with clang). But it's important for users across all
platforms to be able to install binaries with a minimum of fuss.
Limitations of MSVC have already led the numpy/scipy community to
investigate building with MinGW-w64. (See several related threads from
April on the numpy-discussion list.) Ensuring compatibility with CPython's
chosen msvcrt has made that work even more difficult for them.

And Julia is not yet a significant portion of anything, but our community
is growing rapidly. See https://github.com/JuliaLang/IJulia.jl/pull/211 -
with respect to IJulia, Python is just an implementation detail. Even
such a silly thing as automating the execution of the Python installer, to
set up a private only-used-by-IJulia copy, is needlessly difficult to do.
The work on Jupyter will hopefully help this specific situation sooner or
later, but there are other cases where CPython needs to serve as part of
the infrastructure, and the status quo makes that harder to automate.


We'd need to be educated more about the reasons why this approach works
better than remaining compatible with MSVC CPython so we could evaluate
the risks and reward intelligently.


Ideally, we can pursue an approach that would be able to remain compatible
with MSVC CPython. Even if this needs involvement from MinGW-w64 to make
happen, I don't think it's intractable. But I know less about the inner
details of CPython than you do so I could be wrong.


But as has been discussed, it seems better to focus first on fixing the
issues on which we are all in agreement (building extensions with MinGW).


Yes. We'll look into how much of the work has already been done on this.


there *are* people on the core-mentorship list who have expressed
interest in helping out with our automated testing infrastructure,
including (if I understand correctly) adding some level of integration
to other CI systems (which might just be messages to the IRC
channel)[*].  So that could be a fruitful avenue to explore.


If we pursue a fork (which not everyone will like but might happen anyway)
then we likely would do this type of CI integration along the way as Ray
suggested. So even if it turns out to fail as an endeavor, some good may
come of it.

Sincerely,
Tony

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 26 October 2014 13:12, Tony Kelman kel...@berkeley.edu wrote:
 Only cross-compilation and the build system in the above list are relevant
 to CPython, but I hope I have convinced you, Paul Moore, etc. that there are
 real reasons for some groups of users and developers to prefer MinGW-w64
 over MSVC.

Not really, to be honest. I still don't understand why anyone not
directly involved in CPython development would need to build their own
Python executable on Windows. Can you explain a single specific
situation where installing and using the python.org executable is not
possible (on the assumption that the mingw build is functionally
identical and ABI compatible with the CPython build, the claim being
made here)? Note that not possible is different from I don't want
to or it doesn't fit my views about free software or similar. Also
note that building extensions is different - you have to assume that
building extensions using mingw with a mingw-built CPython is just as
hard as building them with a MSVC-built CPython (otherwise you've made
changes to extension building and you should contribute them
independently so that everyone can benefit, not just those who build
their own Python with mingw!)

 Paul Moore:
 If it were possible to cross-compile compatible extensions on Linux,
 projects developed on Linux could supply Windows binaries much more
 easily, which would be a huge benefit to the whole Windows Python
 community.

 I want to do exactly this in an automated repeatable way, preferably on
 a build service. This seems harder to do when CPython cannot itself be
 built and handled as a dependency by that same automated, repeatable
 build service.

I cannot see why you would need to build Python in order to build
extensions. After all, if your build service is on Linux, it couldn't
run a mingw-built Python anyway. If your build service is a Windows
machine, just install the python.org binaries (which is a simple
download and install, that can be fully automated, but which is a
one-off process anyway).

 Unless it becomes possible to cross-compile extensions
 using the build machine's own version of Python, which might be the right
 approach.

This may be where we are getting confused. I see this as the only
practical way of cross-compiling Windows extensions on Linux, by using
the Linux Python. So being able to cross-compile Python is not
relevant.

On a tangential note, any work on supporting mingw builds and
cross-compilation should probably be done using setuptools, so that it
is external to the CPython code. That way (a) it isn't constrained by
the CPython release schedules and backward compatibility constraints,
and (b) it can be used in older versions of Python (which is pretty
much essential if it's to be useful, TBH).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 26 October 2014 17:59, Tony Kelman kel...@berkeley.edu wrote:
 Ensuring compatibility with CPython's
 chosen msvcrt has made that work even more difficult for them.

Ensuring compatibility with CPython's msvcrt is mandatory unless you
want to create a split in the community over which extensions work
with which builds. That's precisely the scenario Steve Dower and
myself (among others) fear, and want to avoid at all cost.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 26 October 2014 14:28, Ray Donnelly mingw.andr...@gmail.com wrote:
 I like this idea. To reduce the workload, we should probably pick
 Python3 (at least initially)?

Aren't the existing patches on the tracker already for Python 3.5+?
They should be, as that's the only version that's likely to be a
possible target (unless you get someone to agree to allow a change
like this as in scope for Pythhon 2.7, which I've seen no indication
of). Maybe I'm confused here.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Ray Donnelly

On Sun, Oct 26, 2014 at 10:41 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 26 October 2014 13:12, Tony Kelman kel...@berkeley.edu wrote:
 Only cross-compilation and the build system in the above list are relevant
 to CPython, but I hope I have convinced you, Paul Moore, etc. that there are
 real reasons for some groups of users and developers to prefer MinGW-w64
 over MSVC.

 Not really, to be honest. I still don't understand why anyone not
 directly involved in CPython development would need to build their own
 Python executable on Windows. Can you explain a single specific
 situation where installing and using the python.org executable is not
 possible (on the assumption that the mingw build is functionally
 identical and ABI compatible with the CPython build, the claim being
 made here)?

I don't know where this ABI compatible thing came into being; I
think Steve Dower eluded to it by stating that we should focus on
enabling MinGW-w64 as an extension-building compiler, using a core
interpreter built with MSVC, and that by limiting the interfaces to
the Windows C calling conventions everything would be OK.
Unfortunately this is not possible. MinGW-w64-built extensions need to
link to msvcrt.dll to do anything useful and you cannot mix two
different msvcr??.dlls in one application. Please see
http://msdn.microsoft.com/en-us/library/abx4dbyh%28v=VS.100%29.aspx
and http://msdn.microsoft.com/en-us/library/ms235460%28v=VS.100%29.aspx
for the details. MinGW-w64 assumes the very old msvcrt.dll files from
Windows XP SP3 and XP64 specifically to avoid this mess. The rest of
your reply assumes that this ABI compatibility is a given so I'll stop
at this point.

 Note that not possible is different from I don't want
 to or it doesn't fit my views about free software or similar. Also
 note that building extensions is different - you have to assume that
 building extensions using mingw with a mingw-built CPython is just as
 hard as building them with a MSVC-built CPython (otherwise you've made
 changes to extension building and you should contribute them
 independently so that everyone can benefit, not just those who build
 their own Python with mingw!)

 Paul Moore:
 If it were possible to cross-compile compatible extensions on Linux,
 projects developed on Linux could supply Windows binaries much more
 easily, which would be a huge benefit to the whole Windows Python
 community.

 I want to do exactly this in an automated repeatable way, preferably on
 a build service. This seems harder to do when CPython cannot itself be
 built and handled as a dependency by that same automated, repeatable
 build service.

 I cannot see why you would need to build Python in order to build
 extensions. After all, if your build service is on Linux, it couldn't
 run a mingw-built Python anyway. If your build service is a Windows
 machine, just install the python.org binaries (which is a simple
 download and install, that can be fully automated, but which is a
 one-off process anyway).

 Unless it becomes possible to cross-compile extensions
 using the build machine's own version of Python, which might be the right
 approach.

 This may be where we are getting confused. I see this as the only
 practical way of cross-compiling Windows extensions on Linux, by using
 the Linux Python. So being able to cross-compile Python is not
 relevant.

 On a tangential note, any work on supporting mingw builds and
 cross-compilation should probably be done using setuptools, so that it
 is external to the CPython code. That way (a) it isn't constrained by
 the CPython release schedules and backward compatibility constraints,
 and (b) it can be used in older versions of Python (which is pretty
 much essential if it's to be useful, TBH).

 Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Tony Kelman


Not really, to be honest. I still don't understand why anyone not
directly involved in CPython development would need to build their own
Python executable on Windows. Can you explain a single specific
situation where installing and using the python.org executable is not
possible


I want, and in many places *need*, an all-MinGW stack. For a great deal
of software that is not Python, I can do this today. I can use build
services, package management, and dependency resolution tools that work
very well together with this all-MinGW software stack. These are problems
that Python has notoriously struggled with on Windows for a long time.
It's not my views on free software, it's the reality of MSVC being a
near-useless compiler for scientific software. (And I don't see that
changing much.) Do my requirements conflict with many non-scientific
Python users on Windows? Probably. So you're welcome to ignore my
requirements and I'll do my own thing, but I don't think I'm alone.
There's likely no desire from the scientific Python community to branch
off and separate in quite the way I'm willing to do from non-scientific
Python, but it would solve some of their problems (while introducing many
others). I suspect a MinGW-w64-oriented equivalent to Conda would be
attractive to many. That's approximately what I'm aiming for.

There are some ways in which I can use the python.org MSVC executable and
installer. But it is nearly impossible for me to integrate it into the rest
of the tools and stack that I am using; it sticks out like a sore thumb.
Likewise MinGW-compiled CPython may stick out like a sore thumb relative
to the existing way things work with Python on Windows. I'm okay with that,
you probably aren't.


changes to extension building and you should contribute them
independently so that everyone can benefit


Noted.


I cannot see why you would need to build Python in order to build
extensions.


No, of course they are separate. CPython is one of my dependencies.
Compiled extensions are other dependencies. Software completely unrelated
to Python is yet another set of dependencies. It's not a very coherent
stack if I can't handle all of these dependencies in a uniform way.


On a tangential note, any work on supporting mingw builds and
cross-compilation should probably be done using setuptools, so that it
is external to the CPython code.


Noted.

Sincerely,
Tony

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 26 October 2014 23:24, Tony Kelman kel...@berkeley.edu wrote:
 I want, and in many places *need*, an all-MinGW stack.

OK, I'm willing to accept that statement. But I don't understand it,
and I don't think you've explained why you *need* your CPython
interpreter to be compiled with mingw (as opposed to a number of other
things you might need around building extensions). You may well need
a mingw-compiled CPython because no-one has yet fixed the issues
around using mingw to build extensions for the python.org python
build. But that's my point - I'd rather they fixed that issue,
rather than perpetuating your need for a non-standard compiler that
uses extensions no-one else can use.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On 26 October 2014 23:11, Ray Donnelly mingw.andr...@gmail.com wrote:
 I don't know where this ABI compatible thing came into being;

Simple. If a mingw-built CPython doesn't work with the same extensions
as a MSVC-built CPython, then the community gets fragmented (because
you can only use the extensions built for your stack). Assuming numpy
needs mingw and ultimately only gets built for a mingw-compiled Python
(because the issues building for MSVC-built Python are too hard) and
assuming that nobody wants to make the effort to build pywin32 under
mingw, then what does someone who needs both numpy and pywin32 do?

Avoiding that issue is what I mean by ABI-compatible. (And that's all
I mean by it, nothing more subtle or controversial).

I view it as critical (because availability of binaries is *already*
enough of a problem in the Windows world, without making it worse)
that we avoid this sort of fragmentation. I'm not seeing an
acknowledgement from the mingw side that they agree. That's my
concern. If we both agree, there's nothing to argue about.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread martin



Zitat von Tony Kelman kel...@berkeley.edu:


A maintainer has volunteered. Others will help. Can any core developers
please begin reviewing some of his patches?


Unfortunately, every attempt to review these patches has failed for me,
every time. In the last iteration of an attempt to add mingw64 support,
I had asked contributors to also provide instructions on how to use these
patches, and haven't received any instructions that actually worked.

I'm hesitant to add code that I cannot verify as actually working.

I guess anybody else reviewing these patches ran into similar problems
(I know some other core developers have tried reviewing them as well,
others have stated here that they are unable to review the patches).

Regards,
Martin


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Tony Kelman


I'm several weeks late to this discussion, but I'm glad to see that it
happened. I'm not a Python developer, and barely a user, but I have several
years of daily experience compiling complicated scientific software cross-
platform, particularly with MinGW-w64 for Windows. The Python community,
both core language and scientific package developers and users, needs to
act here. The problem is bad and getting worse. Luckily much of the work
to start solving it has already been done in bits and pieces, it needs
coordination and participation to come to a conclusion.


Cross compilation is a valid issue, but I hope that build services like
Appveyor also help out here. There is regular talk about the PSF/PyPI
providing something similar


AppVeyor is better than nothing (I've been using it since beta), but it's
a far cry from build services and package management as the Linux world
knows them. Obtaining and setting up build dependencies quickly and
repeatably, and finishing the build of a complicated piece of software
such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
lies), etc. on a small single-core VM with limited memory and a restrictive
time limit is often not possible. These problems are solved within Linux
infrastructure like Koji, Open Build Service, buildd, etc.

MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
compiling a wide variety of libraries from Linux to Windows, in addition to
building conventionally on Windows for Windows. The MSYS2 collection of
MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
been mentioned. Linux distributions including
- Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
- openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
- Arch https://aur.archlinux.org/packages/?K=mingw
and others have projects for providing many hundreds of open-source
packages compiled for Windows. Debian has the cross-compilers available but
not many packages yet (https://packages.debian.org/search?keywords=mingw).

As a developer of a (compiled) open-source library or application, wouldn't
you love to be able to build binaries on Linux for Windows? With some work
and build system patches, you can. For many projects it's a simple matter of
./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
Arch. This is possible with a very, very long set of patches, many of which
have been submitted by Roumen Petrov to the Python bug tracker - see
http://bugs.python.org/issue17605 and other issues linked therein. Roumen
has done a huge amount of work, and he and others who consider the MinGW-w64
compiler important will continue to do so. (Thanks a million Roumen!)


I could step in as maintainer for Cygwin and builds based on GCC using
mingw* APIs.

Regards,
Roumen Petrov


A maintainer has volunteered. Others will help. Can any core developers
please begin reviewing some of his patches? Even if just to say they need
to be rebased. The lack of responses on the bug tracker is disheartening
from an outside perspective. The pile of patches accumulating in external
MinGW packaging projects is tantamount to a fork of CPython. It won't go
away since there are dedicated packagers working to keep their MinGW-w64
builds functional, even in the ad-hoc current state. The patches continue
piling up, making it more difficult for everyone - except for the core
Python developers if they continue to ignore the problem. Bring the people
working on these patches into the fold as contributors. Review the patches.
It would make Python as a language and a community even more diverse and
welcoming.


Deprecate/remove support for compiling CPython itself with compilers
other than MSVC on Windows


I'm not alone in thinking that this would be a bad idea. MSVC can continue
to be the default compiler used for Python on Windows, none of Roumen's
patches change that. They would merely open up the choice for packagers and
users to build CPython (and extension modules, thanks to separate patches)
with alternate compilers, in cross-compilation or otherwise.

Sincerely,
Tony

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Steve Dower

(Apologies for the short reply, posting from my phone.)

MSVC can continue
to be the default compiler used for Python on Windows, none of Roumen's
patches change that. They would merely open up the choice for packagers and
users to build CPython (and extension modules, thanks to separate patches)
with alternate compilers, in cross-compilation or otherwise.

Building CPython for Windows is not something that needs solving. The culture 
on Windows is to redistribute binaries, not source, and both the core team and 
a number of redistributors have this figured out (and it will only become 
easier with VC14 and Python 3.5).

I'd rather see this effort thrown behind compiling extensions, including cross 
compilation. The ABI is well defined enough that any compiler should be usable, 
especially once the new CRT is in use. However, there is work needed to update 
the various tool chains to link to VC14's CRT and we need to figure out the 
inconsistencies between tools so we can document and work through them.

Having different builds of CPython out there will only fragment the community 
and hurt extension authors far more than it may seem to help.

Cheers,
Steve

Top-posted from my Windows Phone

From: Tony Kelmanmailto:kel...@berkeley.edu
Sent: ‎10/‎25/‎2014 9:06
To: python-dev@python.orgmailto:python-dev@python.org
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

I'm several weeks late to this discussion, but I'm glad to see that it
happened. I'm not a Python developer, and barely a user, but I have several
years of daily experience compiling complicated scientific software cross-
platform, particularly with MinGW-w64 for Windows. The Python community,
both core language and scientific package developers and users, needs to
act here. The problem is bad and getting worse. Luckily much of the work
to start solving it has already been done in bits and pieces, it needs
coordination and participation to come to a conclusion.

 Cross compilation is a valid issue, but I hope that build services like
 Appveyor also help out here. There is regular talk about the PSF/PyPI
 providing something similar

AppVeyor is better than nothing (I've been using it since beta), but it's
a far cry from build services and package management as the Linux world
knows them. Obtaining and setting up build dependencies quickly and
repeatably, and finishing the build of a complicated piece of software
such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
lies), etc. on a small single-core VM with limited memory and a restrictive
time limit is often not possible. These problems are solved within Linux
infrastructure like Koji, Open Build Service, buildd, etc.

MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
compiling a wide variety of libraries from Linux to Windows, in addition to
building conventionally on Windows for Windows. The MSYS2 collection of
MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
been mentioned. Linux distributions including
- Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
- openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
- Arch https://aur.archlinux.org/packages/?K=mingw
and others have projects for providing many hundreds of open-source
packages compiled for Windows. Debian has the cross-compilers available but
not many packages yet (https://packages.debian.org/search?keywords=mingw).

As a developer of a (compiled) open-source library or application, wouldn't
you love to be able to build binaries on Linux for Windows? With some work
and build system patches, you can. For many projects it's a simple matter of
./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
Arch. This is possible with a very, very long set of patches, many of which
have been submitted by Roumen Petrov to the Python bug tracker - see
http://bugs.python.org/issue17605 and other issues linked therein. Roumen
has done a huge amount of work, and he and others who consider the MinGW-w64
compiler important will continue to do so. (Thanks a million Roumen!)

 I could step in as maintainer for Cygwin and builds based on GCC using
 mingw* APIs.

 Regards,
 Roumen Petrov

A maintainer has volunteered. Others will help. Can any core developers
please begin reviewing some of his patches? Even if just to say they need
to be rebased. The lack of responses on the bug tracker is disheartening
from an outside perspective. The pile of patches accumulating in external
MinGW packaging projects is tantamount to a fork of CPython. It won't go
away since there are dedicated packagers working to keep their MinGW-w64
builds functional, even in the ad-hoc current state. The patches continue
piling up, making it more difficult for everyone - except for the core
Python developers if they continue to ignore

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread R. David Murray

On Sat, 25 Oct 2014 05:45:24 -0700, Tony Kelman kel...@berkeley.edu wrote:
 As a developer of a (compiled) open-source library or application, wouldn't
 you love to be able to build binaries on Linux for Windows? With some work
 and build system patches, you can. For many projects it's a simple matter of
 ./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
 only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
 Arch. This is possible with a very, very long set of patches, many of which
 have been submitted by Roumen Petrov to the Python bug tracker - see
 http://bugs.python.org/issue17605 and other issues linked therein. Roumen
 has done a huge amount of work, and he and others who consider the MinGW-w64
 compiler important will continue to do so. (Thanks a million Roumen!)

Yes, I for one appreciate Roumen's work, even though I'm not currently
in a position to review the patches.

  I could step in as maintainer for Cygwin and builds based on GCC using
  mingw* APIs.
 
  Regards,
  Roumen Petrov
 
 A maintainer has volunteered. Others will help. Can any core developers
 please begin reviewing some of his patches? Even if just to say they need
 to be rebased. The lack of responses on the bug tracker is disheartening
 from an outside perspective. The pile of patches accumulating in external
 MinGW packaging projects is tantamount to a fork of CPython. It won't go
 away since there are dedicated packagers working to keep their MinGW-w64
 builds functional, even in the ad-hoc current state. The patches continue
 piling up, making it more difficult for everyone - except for the core
 Python developers if they continue to ignore the problem. Bring the people
 working on these patches into the fold as contributors. Review the patches.
 It would make Python as a language and a community even more diverse and
 welcoming.

IIUC, our general policy for bringing new platforms into core support,
as opposed to continuing to be a separately maintained set of patches,
is that there be a big enough community of interest (I don't remember
the definition of big enough) and that there be both committed
maintainers *and* at least one buildbot.

Being able to build windows packages on linux is a compelling argument,
but that only applies to building extensions, not the interpreter.

I would recommend starting with any patches that help MinGW that are not
MinGW specific but instead generally improve the build system and cross
compilation.  There have been a number of such issues opened and
improvements made beyond the MinGW related ones, some coming from
Debian, some coming from the Android community.

So  target those first.  My suggestion would be to pick a patch that is
believed to be commit ready, and come to #python-dev on freenode and
gently bug us until it gets committed.  Then pick the next one, and
repeat.  Working from simplest to more complex would also probably be a
good strategy :)

From there I'd move on to patches that support using MinGW for building
extensions.  There will probably be useful to also get engaged with the
packaging folks.  And, at this point, we would NEED A BUILDBOT.  That
is, a machine that has whatever tools are required installed such that
tests added to the test suite to test MinGW support in distutils would
run, so we can be sure we don't break anything when making other
changes.

For compiling CPython itself with MinGW, we'd first need to develop a
consensus that we want to support it in core.  I'd say get building
extensions working first, and then make that argument.

By the time a bunch of patches get committed, we should be ready (read:
eager :) to promote someone to do it themselves.  Even if we never
decide to support compiling CPython itself with MinGW, I would hope that
getting it to work for extensions would greatly reduce the number of
patches needed to be maintained outside the tree in order to do so.  And
once at least one MinGW advocate is part of the core team, advocacy
becomes easier :)

--David

PS: one meta comment about the MinGW issues: I scanned a few from the
linked bug, and while the issues split the patches out, which is a great
start, there is no discussion on the issues for the individual patches
to give background and motivation and an explanation of what the patch
is trying to accomplish.  But you probably don't want to waste time on
improving ones that apply *only* to compiling CPython itself unless we
get consensus that we want to support that.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Steve Dower

Ray Donnelly wrote:
 What is it that you
 are afraid of if CPython can be compiled out of the box using
 mingw/MinGW-w64? Why are you fighting so hard against having option.

I'm afraid of users having numpy crash because they're using an MSVC CPython 
instead of a mingw CPython. I'm afraid of users not being able to use library A 
and library B at the same time because A requires MSVC CPython and B requires 
mingw CPython. (I can produce more examples if you like, but the general 
concern is having a fragmented community, as I said in my previous post.)

I'm fighting against having options because it will suck up the precious 
volunteer time we have and direct it away from where it would be more useful, 
which is making it easier to build extensions with other compilers.

I would love to see extensions for Windows built on all platforms. I see no 
value in building Python itself for Windows from different platforms.

If other core developers agree with you that a more pure build of Python is 
worthwhile, then they can go ahead and merge the patches (though I suspect most 
would like to see some traction achieved on a fork first). I think it's 
important that I (as Windows build manager) make my own position clear, that's 
all.

(The rest of your email is purely unsubstantiated opinion, which is okay to 
have, but it doesn't demand any reply so I've omitted it.)

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower steve.do...@microsoft.com wrote:
 (Apologies for the short reply, posting from my phone.)

 MSVC can continue
 to be the default compiler used for Python on Windows, none of Roumen's
 patches change that. They would merely open up the choice for packagers and
 users to build CPython (and extension modules, thanks to separate patches)
 with alternate compilers, in cross-compilation or otherwise.

 Building CPython for Windows is not something that needs solving. The
 culture on Windows is to redistribute binaries, not source, and both the
 core team and a number of redistributors have this figured out (and it will
 only become easier with VC14 and Python 3.5).

This is the second time you've used the vacuous culture on Windows
argument, now with an added appeal to (vague) authority. That may be
your opinion and that of some others, but there's a large number of
people who don't care for using non-Free tools. IMHO building CPython
on Windows using Open Source toolchains is very much something that
needs merging upstream and supporting by default. What is it that you
are afraid of if CPython can be compiled out of the box using
mingw/MinGW-w64? Why are you fighting so hard against having option.
If CPython wants to truly call itself an Open Source project then I
consider being able to compile and cross-compile it with capable Open
Source toolchains on all major platforms a requirement.


 I'd rather see this effort thrown behind compiling extensions, including
 cross compilation. The ABI is well defined enough that any compiler should
 be usable, especially once the new CRT is in use. However, there is work
 needed to update the various tool chains to link to VC14's CRT and we need
 to figure out the inconsistencies between tools so we can document and work
 through them.

 Having different builds of CPython out there will only fragment the
 community and hurt extension authors far more than it may seem to help.

 Cheers,
 Steve

 Top-posted from my Windows Phone
 
 From: Tony Kelman
 Sent: ‎10/‎25/‎2014 9:06
 To: python-dev@python.org
 Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

 I'm several weeks late to this discussion, but I'm glad to see that it
 happened. I'm not a Python developer, and barely a user, but I have several
 years of daily experience compiling complicated scientific software cross-
 platform, particularly with MinGW-w64 for Windows. The Python community,
 both core language and scientific package developers and users, needs to
 act here. The problem is bad and getting worse. Luckily much of the work
 to start solving it has already been done in bits and pieces, it needs
 coordination and participation to come to a conclusion.

 Cross compilation is a valid issue, but I hope that build services like
 Appveyor also help out here. There is regular talk about the PSF/PyPI
 providing something similar

 AppVeyor is better than nothing (I've been using it since beta), but it's
 a far cry from build services and package management as the Linux world
 knows them. Obtaining and setting up build dependencies quickly and
 repeatably, and finishing the build of a complicated piece of software
 such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
 lies), etc. on a small single-core VM with limited memory and a restrictive
 time limit is often not possible. These problems are solved within Linux
 infrastructure like Koji, Open Build Service, buildd, etc.

 MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
 compiling a wide variety of libraries from Linux to Windows, in addition to
 building conventionally on Windows for Windows. The MSYS2 collection of
 MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
 been mentioned. Linux distributions including
 - Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
 - openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
 - Arch https://aur.archlinux.org/packages/?K=mingw
 and others have projects for providing many hundreds of open-source
 packages compiled for Windows. Debian has the cross-compilers available but
 not many packages yet (https://packages.debian.org/search?keywords=mingw).

 As a developer of a (compiled) open-source library or application, wouldn't
 you love to be able to build binaries on Linux for Windows? With some work
 and build system patches, you can. For many projects it's a simple matter of
 ./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
 only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
 Arch. This is possible with a very, very long set of patches, many of which
 have been submitted by Roumen Petrov to the Python bug tracker - see
 http://bugs.python.org/issue17605 and other issues linked therein. Roumen
 has done a huge amount of work, and he and others who consider the MinGW-w64

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, Oct 26, 2014 at 7:50 AM, Steve Dower steve.do...@microsoft.com wrote:
 Ray Donnelly wrote:
 What is it that you
 are afraid of if CPython can be compiled out of the box using
 mingw/MinGW-w64? Why are you fighting so hard against having option.

 I'm afraid of users having numpy crash because they're using an MSVC CPython 
 instead of a mingw CPython. I'm afraid of users not being able to use library 
 A and library B at the same time because A requires MSVC CPython and B 
 requires mingw CPython. (I can produce more examples if you like, but the 
 general concern is having a fragmented community, as I said in my previous 
 post.)


It might fragment the community to have multiple different binary
distributions. But it ought to be possible for any person/organization
to say We're going to make our own build of Python, with these
extension modules, built with this compiler, targeting this platform,
and do everything from source. That might mean they can no longer take
the short-cut of download someone's MSVC-built extension and use it
as-is, but they should be able to take anyone's extension and build
it on their chosen compiler. Having MinGW as a formally supported
platform would make life a lot easier for people who want to test
CPython patches, for instance - my building and testing of PEP
463-enhanced Python was Linux-only, because I didn't want to try to
set up an entire new buildchain just to try to get a Windows binary
going. There's absolutely no need for that to be binary-compatible
with anything else; as long as it'll run the standard library, it'll
do.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, 26 Oct 2014 08:11:39 +1100
Chris Angelico ros...@gmail.com wrote:
 
 It might fragment the community to have multiple different binary
 distributions. But it ought to be possible for any person/organization
 to say We're going to make our own build of Python, with these
 extension modules, built with this compiler, targeting this platform,
 and do everything from source. That might mean they can no longer take
 the short-cut of download someone's MSVC-built extension and use it
 as-is, but they should be able to take anyone's extension and build
 it on their chosen compiler. Having MinGW as a formally supported
 platform would make life a lot easier for people who want to test
 CPython patches, for instance - my building and testing of PEP
 463-enhanced Python was Linux-only,

And how do you know that it would have worked with MSVC if you only use
MinGW?
If you want to ensure compatibility with MSVC, you must build with MSVC.
There's no working around that.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, Oct 26, 2014 at 8:47 AM, Antoine Pitrou solip...@pitrou.net wrote:
 And how do you know that it would have worked with MSVC if you only use
 MinGW?
 If you want to ensure compatibility with MSVC, you must build with MSVC.
 There's no working around that.

Precisely. If you build with MinGW, you can't ensure compatibility
with MSVC. Reread my post: I gave two examples of situations where
that isn't a problem.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sat, 25 Oct 2014 21:10:23 +0100
Ray Donnelly mingw.andr...@gmail.com wrote:
 
 This is the second time you've used the vacuous culture on Windows
 argument, now with an added appeal to (vague) authority.
[...]
 Why are you fighting so hard against having option.
 If CPython wants to truly call itself an Open Source project then I
 consider being able to compile and cross-compile it with capable Open
 Source toolchains on all major platforms a requirement.

Now *that* sounds vacuous.

Regarding open source, there's a clear and official definition of it,
which Python satisfies:
http://opensource.org/osd-annotated

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, 26 Oct 2014 08:53:29 +1100
Chris Angelico ros...@gmail.com wrote:
 On Sun, Oct 26, 2014 at 8:47 AM, Antoine Pitrou solip...@pitrou.net wrote:
  And how do you know that it would have worked with MSVC if you only use
  MinGW?
  If you want to ensure compatibility with MSVC, you must build with MSVC.
  There's no working around that.
 
 Precisely. If you build with MinGW, you can't ensure compatibility
 with MSVC. Reread my post: I gave two examples of situations where
 that isn't a problem.

How do you know this isn't a problem, since you haven't *tested* with
MSVC?
Why on Earth would you want to test your PEP work with an unsupported
Windows compiler and runtime, rather than with the officially supported
compiler and runtime?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sat, Oct 25, 2014 at 10:52 PM, Antoine Pitrou solip...@pitrou.net wrote:
 On Sat, 25 Oct 2014 21:10:23 +0100
 Ray Donnelly mingw.andr...@gmail.com wrote:

 This is the second time you've used the vacuous culture on Windows
 argument, now with an added appeal to (vague) authority.
 [...]
 Why are you fighting so hard against having option.
 If CPython wants to truly call itself an Open Source project then I
 consider being able to compile and cross-compile it with capable Open
 Source toolchains on all major platforms a requirement.

 Now *that* sounds vacuous.


Maybe you missed where I said I consider.

 Regarding open source, there's a clear and official definition of it,
 which Python satisfies:
 http://opensource.org/osd-annotated

 Regards

 Antoine.


 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou solip...@pitrou.net wrote:
 How do you know this isn't a problem, since you haven't *tested* with
 MSVC?
 Why on Earth would you want to test your PEP work with an unsupported
 Windows compiler and runtime, rather than with the officially supported
 compiler and runtime?

This discussion revolved around supporting MinGW in addition to MSVC.
If it had been supported when I was doing that, I could have spun
myself up a Windows build and tested it. Since it was (and so far
still is) not, the hassle of hunting down a valid MSVC that could
build for Win XP (as that's what my test box runs) was simply not
worthwhile. My point is that there is no community fragmentation
happening here; the only fragmentation is of binary distribution of
extension modules, and there are several ways in which this needn't be
a problem.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, 26 Oct 2014 09:06:36 +1100
Chris Angelico ros...@gmail.com wrote:
 On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou solip...@pitrou.net wrote:
  How do you know this isn't a problem, since you haven't *tested* with
  MSVC?
  Why on Earth would you want to test your PEP work with an unsupported
  Windows compiler and runtime, rather than with the officially supported
  compiler and runtime?
 
 This discussion revolved around supporting MinGW in addition to MSVC.
 If it had been supported when I was doing that, I could have spun
 myself up a Windows build and tested it.

My point is that your Windows build would not have the same behaviour
as a MSVC-produced Windows build, and so testing it with it would not
certify that your code would actually be compatible with genuine
MSVC builds of CPython, which we will not stop supporting.

Therefore, what you and the OP are proposing would not make it
*easier* to ensure cross-platform compatibility but rather *harder*, by
adding another incompatible build configuration to the mix of supported
configurations.

The only remaining question is whether it is worthwhile adding support
for such an additional platform, and given that MinGW is extremely
marginal amongst Windows developers, the answer is IMHO no.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou solip...@pitrou.net wrote:
 My point is that your Windows build would not have the same behaviour
 as a MSVC-produced Windows build, and so testing it with it would not
 certify that your code would actually be compatible with genuine
 MSVC builds of CPython, which we will not stop supporting.


So you're saying it's impossible to support two compilers?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread R. David Murray

On Sat, 25 Oct 2014 21:10:23 +0100, Ray Donnelly mingw.andr...@gmail.com 
wrote:
 On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower steve.do...@microsoft.com 
 wrote:
  (Apologies for the short reply, posting from my phone.)
 
  MSVC can continue
  to be the default compiler used for Python on Windows, none of Roumen's
  patches change that. They would merely open up the choice for packagers and
  users to build CPython (and extension modules, thanks to separate patches)
  with alternate compilers, in cross-compilation or otherwise.
 
  Building CPython for Windows is not something that needs solving. The
  culture on Windows is to redistribute binaries, not source, and both the
  core team and a number of redistributors have this figured out (and it will
  only become easier with VC14 and Python 3.5).
 
 This is the second time you've used the vacuous culture on Windows
 argument, now with an added appeal to (vague) authority. That may be
 your opinion and that of some others, but there's a large number of
 people who don't care for using non-Free tools. IMHO building CPython
 on Windows using Open Source toolchains is very much something that
 needs merging upstream and supporting by default. What is it that you
 are afraid of if CPython can be compiled out of the box using
 mingw/MinGW-w64? Why are you fighting so hard against having option.
 If CPython wants to truly call itself an Open Source project then I
 consider being able to compile and cross-compile it with capable Open
 Source toolchains on all major platforms a requirement.

You are doing yourself a disservice by this last statement.  There
really can't be any question that Python is an open source project,
so insinuating that the CPython community is doing something wrong
is not going to win you friends and helpers.

A better approach would be to acknowledge that what we are currently
doing works well for supporting Windows (especially since we actually
have some engagement from Microsoft that is *getting some problems
fixed* in ways that help make things more open).

And then say, wouldn't it be *really cool* if we could also build
CPython using an open source toolchain on Windows out of the box?.  You
might not get instant agreement on that (well, clearly you won't), but
it'd be much more likely you'd start garnering support.

Assume that people are well intentioned, and convince them your
suggestions will make things *even better* using positive arguments.
You might not succeed, but you'll have a much better chance.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Terry Reedy


On 10/25/2014 5:11 PM, Chris Angelico wrote:


It might fragment the community to have multiple different binary
distributions. But it ought to be possible for any person/organization
to say We're going to make our own build of Python, with these
extension modules, built with this compiler, targeting this platform,
and do everything from source. That might mean they can no longer take
the short-cut of download someone's MSVC-built extension and use it
as-is, but they should be able to take anyone's extension and build
it on their chosen compiler. Having MinGW as a formally supported
platform would make life a lot easier for people who want to test
CPython patches, for instance - my building and testing of PEP
463-enhanced Python was Linux-only, because I didn't want to try to
set up an entire new buildchain just to try to get a Windows binary
going. There's absolutely no need for that to be binary-compatible
with anything else; as long as it'll run the standard library, it'll
do.


David Murray's unanswered post laid out the path to move in the 
direction you want.  Either take it yourself or try to persuade other 
MinGW fans to do so.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Paul Moore

On 25 October 2014 21:50, Steve Dower steve.do...@microsoft.com wrote:
 Ray Donnelly wrote:
 What is it that you
 are afraid of if CPython can be compiled out of the box using
 mingw/MinGW-w64? Why are you fighting so hard against having option.

 I'm afraid of users having numpy crash because they're using an MSVC
 CPython instead of a mingw CPython. I'm afraid of users not being able
 to use library A and library B at the same time because A requires MSVC
 CPython and B requires mingw CPython. (I can produce more examples
 if you like, but the general concern is having a fragmented community,
 as I said in my previous post.)

Precisely. Either developers test with *all* supported compilers, or
there is a risk of incompatibilities. Advocates of supporting mingw as
a build system for Python generally do *not* suggest that they are
willing to test for, and deal with, cross-version compatibility
issues. Typically mingw is seen as another platform in some sense,
by such advocates, having its own set of supporters and maintainers.

The possibility of extensions built with a mingw-compiled Python
failing when used under a MSVC-built Python is real. It's difficult to
say how big that risk is, but it's certainly there. And I see no-one
offering to be responsible for such compatibility issues (the mingw
supporters generally don't want to set up a MSVC build chain, so it's
difficult to see how they could credibly offer).

 I'm fighting against having options because it will suck up the precious
 volunteer time we have and direct it away from where it would be more
 useful, which is making it easier to build extensions with other compilers.

And claiming that it doesn't suck up developer time ignores the point
I made above - *someone* has to deal with any compatibility issues
that come up. As a starter, does the wheel format need to include tags
to distinguish whether the target Python is MSVC-built and
mingw-built? Who will check that? If there is a need, who will work on
the code needed to incorporate that change into wheel, pip, and the
relevant PEPs?

As Steve says, the Python community has a genuine, strong need for
people with mingw expertise working on making it easier to build
*extensions* using mingw, that work with a MSVC-built CPython. If it
were possible to cross-compile compatible extensions on Linux,
projects developed on Linux could supply Windows binaries much more
easily, which would be a huge benefit to the whole Windows Python
community. But the mingw experts don't want to work on that,
preferring to develop patches allowing CPython to be built with mingw.
No objection from me, it's your free time, use it as you wish, but as
a Windows user of Python I can confirm that it's not what I'd like you
to be doing as your contribution to Python.

 I would love to see extensions for Windows built on all platforms. I see no
 value in building Python itself for Windows from different platforms.

Exactly.

 If other core developers agree with you that a more pure build of Python is
 worthwhile, then they can go ahead and merge the patches (though I suspect
 most would like to see some traction achieved on a fork first). I think it's
 important that I (as Windows build manager) make my own position clear,
 that's all.

Personally, I'm not a core developer, just a long-time member of this
list and occasional contributor to discussions. But I am also a core
pip developer and a Windows user, and from that perspective I am
acutely aware of the common pain points for Python users on Windows.
And they are all about binary extensions, and none at all about
building Python itself. So in my view, being able to build CPython
using mingw is somewhat interesting from a theoretical perspective,
but of little or no practical value[1] and disruptive in a number of
ways, as mentioned above, to improving the overall experience of
Python users on Windows.

Paul

[1] I note, without trying to make a judgement, that many of the
benefits cited for building with mingw are around being able to use
free tools or similar essentially ideological issues.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, 26 Oct 2014 09:22:18 +1100
Chris Angelico ros...@gmail.com wrote:
 On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou solip...@pitrou.net wrote:
  My point is that your Windows build would not have the same behaviour
  as a MSVC-produced Windows build, and so testing it with it would not
  certify that your code would actually be compatible with genuine
  MSVC builds of CPython, which we will not stop supporting.
 
 
 So you're saying it's impossible to support two compilers?

???

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Paul Moore

On 25 October 2014 23:22, Chris Angelico ros...@gmail.com wrote:
 On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou solip...@pitrou.net wrote:
 My point is that your Windows build would not have the same behaviour
 as a MSVC-produced Windows build, and so testing it with it would not
 certify that your code would actually be compatible with genuine
 MSVC builds of CPython, which we will not stop supporting.


 So you're saying it's impossible to support two compilers?

No, rather that Windows currently only has a single supported compiler
(excluding cygwin, which is essentially a different OS). Adding a
second compiler doesn't just involve adding support for it - which is
all that the people offering mingw patches are doing - but also
involves going through the whole CPython ecosystem locating the places
where there is an implicit assumption that all Windows builds use the
same compiler and fixing them. I've already pointed out where this is
a question for pip and wheel. Whoever wants to add support for a
second compiler needs to be willing to do this part of the job as
well.

Handwaving arguments that it's binary compatible aren't enough. Prove it.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread R. David Murray

On Sun, 26 Oct 2014 00:19:44 +0200, Antoine Pitrou solip...@pitrou.net wrote:
 On Sun, 26 Oct 2014 09:06:36 +1100
 Chris Angelico ros...@gmail.com wrote:
  On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou solip...@pitrou.net wrote:
   How do you know this isn't a problem, since you haven't *tested* with
   MSVC?
   Why on Earth would you want to test your PEP work with an unsupported
   Windows compiler and runtime, rather than with the officially supported
   compiler and runtime?
  
  This discussion revolved around supporting MinGW in addition to MSVC.
  If it had been supported when I was doing that, I could have spun
  myself up a Windows build and tested it.
 
 My point is that your Windows build would not have the same behaviour
 as a MSVC-produced Windows build, and so testing it with it would not
 certify that your code would actually be compatible with genuine
 MSVC builds of CPython, which we will not stop supporting.

While true, I don't think that matters for Chris' point.  Given only the
ability to build with the MSVC toolchain, his code (which might even be
pure python for the purposes of this discussion) would not get tested on
Windows until committed and run by the buildbots.  If he could build
CPython using MinGW, he would, and would test his code on Windows.  Even
if there are C components and MSVC/MinGW compatibility issues are
revealed when the buildbots eventually run the code, still the number of
bugs present would probably be lower if he had tested it on Windows
than if he hadn't.

I know I for one do not generally test patches on Windows because I
haven't taken the time to learn how to build CPython on it.  Sure, I
could test pure python changes by applying patches to an installed
Python, but that's an ongoing pain and I'd rather learn to build CPython
on Windows and get to use the normal hg tools.

If I could use a more linux-like toolchain to build CPython on windows,
I would doubtless do much more testing on windows for stuff where I
think windows might behave differently (and I might look at more Windows
bugs...though frankly there are plenty of bugs for me to look at without
looking at Windows bugs).

This is not necessarily a compelling argument for MinGW support.
However, it *is* a valid argument, IMO.

Note: it can be made even less compelling by making it a lot easier to
build CPython on Windows without having an MSVC license (which I think
means not using the GUI, for which I say *yay* :).  I think Zach Ware
has been working on improving the Windows build process, and I keep
meaning to give it a try...

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sat, 25 Oct 2014 19:24:38 -0400
R. David Murray rdmur...@bitdance.com wrote:
 
 I know I for one do not generally test patches on Windows because I
 haven't taken the time to learn how to build CPython on it.  Sure, I
 could test pure python changes by applying patches to an installed
 Python, but that's an ongoing pain and I'd rather learn to build CPython
 on Windows and get to use the normal hg tools.
 
 If I could use a more linux-like toolchain to build CPython on windows,

Well, I don't know how linux-like you want your toolchain, but FTR
you should be able to build from the command line by running
Tools\buildbot\build.bat.

I doubt the MinGW case can be much simpler :-)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Guido van Rossum

On Sat, Oct 25, 2014 at 1:10 PM, Ray Donnelly mingw.andr...@gmail.com
wrote:

 On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower steve.do...@microsoft.com
 wrote:
  Building CPython for Windows is not something that needs solving. The
  culture on Windows is to redistribute binaries, not source, and both the
  core team and a number of redistributors have this figured out (and it
 will
  only become easier with VC14 and Python 3.5).

 This is the second time you've used the vacuous culture on Windows
 argument, now with an added appeal to (vague) authority. That may be
 your opinion and that of some others, but there's a large number of
 people who don't care for using non-Free tools. IMHO building CPython
 on Windows using Open Source toolchains is very much something that
 needs merging upstream and supporting by default. What is it that you
 are afraid of if CPython can be compiled out of the box using
 mingw/MinGW-w64? Why are you fighting so hard against having option.
 If CPython wants to truly call itself an Open Source project then I
 consider being able to compile and cross-compile it with capable Open
 Source toolchains on all major platforms a requirement.


Please stop this ridiculous argument. There's no definition of truly open
source project that has such a requirement, and if you took it to the
extreme you should not be using Windows at all. I appreciate your concern
that building Python for your favorite platform using your favorite
toolchain doesn't work, and if you have patches (or even bug reports) those
are appreciated. But please take your rhetoric about open source elsewhere.


  I'd rather see this effort thrown behind compiling extensions, including
  cross compilation. The ABI is well defined enough that any compiler
 should
  be usable, especially once the new CRT is in use. However, there is work
  needed to update the various tool chains to link to VC14's CRT and we
 need
  to figure out the inconsistencies between tools so we can document and
 work
  through them.
 
  Having different builds of CPython out there will only fragment the
  community and hurt extension authors far more than it may seem to help.


Here's the crux of the matter. We want compiled extension modules
distributed via PyPI to work with the binaries distributed from python.org.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Mark Lawrence


On 26/10/2014 00:24, R. David Murray wrote:

On Sun, 26 Oct 2014 00:19:44 +0200, Antoine Pitrou solip...@pitrou.net wrote:

On Sun, 26 Oct 2014 09:06:36 +1100
Chris Angelico ros...@gmail.com wrote:

On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou solip...@pitrou.net wrote:

How do you know this isn't a problem, since you haven't *tested* with
MSVC?
Why on Earth would you want to test your PEP work with an unsupported
Windows compiler and runtime, rather than with the officially supported
compiler and runtime?


This discussion revolved around supporting MinGW in addition to MSVC.
If it had been supported when I was doing that, I could have spun
myself up a Windows build and tested it.


My point is that your Windows build would not have the same behaviour
as a MSVC-produced Windows build, and so testing it with it would not
certify that your code would actually be compatible with genuine
MSVC builds of CPython, which we will not stop supporting.


While true, I don't think that matters for Chris' point.  Given only the
ability to build with the MSVC toolchain, his code (which might even be
pure python for the purposes of this discussion) would not get tested on
Windows until committed and run by the buildbots.  If he could build
CPython using MinGW, he would, and would test his code on Windows.  Even
if there are C components and MSVC/MinGW compatibility issues are
revealed when the buildbots eventually run the code, still the number of
bugs present would probably be lower if he had tested it on Windows
than if he hadn't.

I know I for one do not generally test patches on Windows because I
haven't taken the time to learn how to build CPython on it.  Sure, I
could test pure python changes by applying patches to an installed
Python, but that's an ongoing pain and I'd rather learn to build CPython
on Windows and get to use the normal hg tools.

If I could use a more linux-like toolchain to build CPython on windows,
I would doubtless do much more testing on windows for stuff where I
think windows might behave differently (and I might look at more Windows
bugs...though frankly there are plenty of bugs for me to look at without
looking at Windows bugs).

This is not necessarily a compelling argument for MinGW support.
However, it *is* a valid argument, IMO.

Note: it can be made even less compelling by making it a lot easier to
build CPython on Windows without having an MSVC license (which I think
means not using the GUI, for which I say *yay* :).  I think Zach Ware
has been working on improving the Windows build process, and I keep
meaning to give it a try...

--David



MSVC Express Edition 2010 works perfectly for building 3.5 so no license 
needed.  Links to older versions have been pointed out on other threads, 
either here or python-ideas, maybe both?  Or use the command line as 
Antoine pointed out elsewhere.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sun, Oct 26, 2014 at 12:30 AM, Antoine Pitrou solip...@pitrou.net wrote:
 On Sat, 25 Oct 2014 19:24:38 -0400
 R. David Murray rdmur...@bitdance.com wrote:

 I know I for one do not generally test patches on Windows because I
 haven't taken the time to learn how to build CPython on it.  Sure, I
 could test pure python changes by applying patches to an installed
 Python, but that's an ongoing pain and I'd rather learn to build CPython
 on Windows and get to use the normal hg tools.

 If I could use a more linux-like toolchain to build CPython on windows,

 Well, I don't know how linux-like you want your toolchain, but FTR
 you should be able to build from the command line by running
 Tools\buildbot\build.bat.

 I doubt the MinGW case can be much simpler :-)


I beg to differ:

Tools\buildbot\build.bat contains:
@rem Used by the buildbot compile step.
cmd /c Tools\buildbot\external.bat
call %VS100COMNTOOLS%vsvars32.bat
cmd /c Tools\buildbot\clean.bat
msbuild PCbuild\pcbuild.sln /p:Configuration=Debug /p:Platform=Win32

^ this involves purchasing and installing MS Visual Studio (I'm not
sure if the Express Edition can be used).

Without explanations for what each step is doing (I posted those last
week), on MSYS2:
Download and run:
http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download
From the MSYS2 shell:
pacman -S git base-devel mingw-w64-x86_64-toolchain mingw-w64-i686-toolchain
git clone https://github.com/Alexpux/MINGW-packages
cd MINGW-packages/mingw-w64-python3
makepkg-mingw -sL --nocheck
(remove --nocheck to run the testsuite. Also you could put this into a
batch file if you were so inclined.)

To install the newly built packages, from the MSYS2 shell again:
pacman -U mingw-w64-*.xz

To run them, you should add /mingw64/bin or /mingw32/bin to your PATH
(or launch a new shell via mingw32_shell.bat or mingw64_shell.bat)
Of course, if you don't want to build it from source you can simply issue:
pacman -S mingw-w64-python3

.. all of the above applies equally to mingw-w64-python2.

 Regards

 Antoine.


 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 25 October 2014 23:22, Chris Angelico ros...@gmail.com wrote:
 On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou solip...@pitrou.net wrote:
 My point is that your Windows build would not have the same behaviour
 as a MSVC-produced Windows build, and so testing it with it would not
 certify that your code would actually be compatible with genuine
 MSVC builds of CPython, which we will not stop supporting.


 So you're saying it's impossible to support two compilers?

 No, rather that Windows currently only has a single supported compiler
 (excluding cygwin, which is essentially a different OS). Adding a
 second compiler doesn't just involve adding support for it - which is
 all that the people offering mingw patches are doing - but also
 involves going through the whole CPython ecosystem locating the places
 where there is an implicit assumption that all Windows builds use the
 same compiler and fixing them. I've already pointed out where this is
 a question for pip and wheel. Whoever wants to add support for a
 second compiler needs to be willing to do this part of the job as
 well.

 Handwaving arguments that it's binary compatible aren't enough. Prove it.

The msvcrt.dlls that MinGW-w64 depends on are those dating back to
Windows XP SP3 / XP64. Ironically, the official Windows CPython
doesn't come with any such crt guarantees and you must ensure that the
same msvcr??.dll is used for *all* extensions. This puts considerable
strain on extension developers to use the correct (or any) version of
Visual Studio to build their extensions for CPython on Windows. Also,
where are the publicly accessible specifications and other technical
descriptions that MinGW-w64 would need to implement strong binary
compatibility with MSVC? As a random example, those for C++ name
mangling and the PDB file format would be very helpful.


 Paul
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows