[issue3871] cross and native build of python for mingw* hosts

[Roumen Petrov]

Roumen Petrov added the comment:

For protocol - as requested patches attached to this issue are split ans posted 
as separate issues.

Please follow those listed below:
issue18654 - modernize mingwcygwin compiler classes (new)
issue17605 - mingw-meta: build interpreter core
issue18653 - mingw-meta: build core modules
issue19245 - mingw-meta: install (new)

Test cases are not covered. Patches will be posted later, if above are resolved.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Matthias Klose]

Matthias Klose added the comment:

 Ray Donnelly added the comment:
 Next time there's a release of Python 3, I'll rebase my patches against that.

sorry, this is the wrong attitude, if you want mingw support to go upstream. 
fetch tip/trunk, re-apply your patches, and submit them.

--
components: +Cross-Build -Build

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Matthias Klose]

Matthias Klose added the comment:

now closing/rejecting this issue. See
http://mail.python.org/pipermail/python-dev/2013-January/123774.html
for the discussion.

--
resolution:  - rejected
stage: patch review - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Roumen Petrov]

Roumen Petrov added the comment:

Please do not post to this thread.
Follow python bugs list for result of patch-split. I will try to post parts of 
this patch to already opened issues or to open new one if necessary.

As basis will be used py3k-20121004-MINGW.patch with following main changes:
- prefer posix installation scheme
- use of system libffi

Lets keep issue open as reference until is reached usable level.

--
components: +Build -Cross-Build

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Ray Donnelly]

Ray Donnelly added the comment:

Great Roumen, please split into new issues then make a new overall tracking 
issue, then this one can be left for historians and archaeologists.

For Posix installation scheme I've got:

https://github.com/niXman/mingw-builds/blob/master/patches/Python-3.3.0/0050-mingw-sysconfig-like-posix.patch

It needs some changes to allow for normal NT scheme when not using MinGW though.

For system libffi there's:

https://github.com/niXman/mingw-builds/blob/master/patches/Python-3.3.0/0035-mingw-system-libffi.patch

When you make your new patches I will base a new set of patches on them.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Ray Donnelly]

Ray Donnelly added the comment:

When I say “our patches” I mean mine and Alexey Pavlov’s jointly maintained 
patch-set.

I hope you don’t mind that I find you saying:

I tried some of these patches, but they aren't very organinzed. I really need 
some docemntaiton to better understand what each patch does and which pairs go 
together and which don't.

...and then continuing to present a single uber-patch to be mildly 
funny/ironic? You've already accepted that you should split them up and remove 
the needless formatting changes to make them more reviewable so I won't labour 
on this point.

ctypes
 bz2
 multiprocessing
 sqlite3
 ssl
 pyexpat
 zlib

Likewise for ours, but also curses, curses_panel, readline, TKinter, IDLE (and 
probably more besides) for both Win32, Win64, Mac OS X 32bit and 64bit.

The main idea behind our patches is (mostly in common with all patches) that 
they:
1. Are as atomic as possible in their operation (which as few interdependencies 
as necessary).
2. Are applied in an obvious ordering.
3. Are named sensibly according to what they do.
4. Build upon - and can reuse updates to - Roumen's uber-patch (and also clean 
it up a bit; let’s face it Roumen did a lot of the hard work and everyone else 
is trying to finish it off so we can build releases from it).
5. Enable comprehensive cross-compilation.

Given the name of each patch and the fact that they do one thing (the novel 
ones at least) and that by-and-large they fit on a few screens, I had hoped 
that they were usefully descriptive.

To explain which ones go together:
The numbering skips 5 each time for each independent bug-fix or feature - to 
allow space for later, related modifications. Therefore, where the numbering 
changes monotonically, the patches are related. This only happens twice, once 
to address the problem caused by Roumen providing an uber-patch and once due to 
a block of fixes all related to curses (ncurses and PDCurses are both 
supported). I assume my addressing of Roumen's uber-patch is what led you to 
write:

However, I'm noticing a ton of descrepancies with these patches. Some are 
removing pthreads, others are adding them back.

There is logic to what you describe as a ton of discrepancies:
Roumen's patch is included in it's entirety as 0030-py3k-20121004-MINGW.patch, 
then I remove one feature of that patch that needed fixing (threading changes) 
with 0031-py3k-20121004-MINGW-removal-of-pthread-patch.patch, then I add a 
patch that provides my newer fix, 0032-py3k-20121004-MINGW-ntthreads.patch. The 
exact same thing happens with the libffi changes. The pthreads stuff needs 
fixing due to libwinpthread in MinGW-w64 and libffi is broken when targetting a 
64bit Windows host.

My reason for doing this is so that I can directly drop updated versions from 
Roumen in over 0030 and it will (in theory at least) still work. He can 
continue to use a uber-patch if he wishes (though I don't think he does this 
for new patches). Also by taking this approach, my patch for the threading 
issues has no dependencies at all on Roumen’s patch, since a previous patch 
“handled” the dependency by removing that part of it.

Everything you said about build environment and the test-suite I agree with. My 
crucifixion-freedom project includes a .vbs to setup the most basic MSYS and 
MinGW-w64 environment: 
https://raw.github.com/mingwandroid/crucifixion-freedom/master/scripts/windows/BootstrapMinGW64.vbs
...this is essential to prevent spending time chasing down issues due to 
discrepancies between build environments.

However, we must go further and add that the patches *cannot* break any other 
native or cross-compilation, which - as I think Matthias is alluding to - is 
probably not the case with your patch. This issue is called cross and native 
build of python for mingw* hosts, so that your patch possibly breaks builds on 
build-machine is unfortunate (especially so as a build-machine Python is 
required to run setup.py as part of the cross-build procedure). Even if it 
doesn't break building for the build-machine it breaks building the 
cross-MinGW-w64. Each occurrence of /mingw in your patch is probably a part 
of this problem (you can’t expect cross build systems to be be able to be 
rooted to /mingw, often you’ll find ~/mingw or /tmp/mingw)

You said, It seems some are also compiling and using the mingw version outside 
of the msys shell, which I'm not certain of the advantages there when u can 
just use the mscv version, I’m not entirely sure what you mean here; if 
compiling MinGW Python on a Windows system with my patches, MSYS is 100% a 
requirement (cmd.exe will never run the configure script). I can only guess 
this relates to the mingw_posix2win_path stuff in your patch. Forgive me if I’m 
wrong, but it seems that you are conflating MSYS and MinGW when they are not 
related (except by frequent proximity). Without wanting to point out the 
obvious, MinGW-w64 is a sysroot and compiler toolchain for making 

[issue3871] cross and native build of python for mingw* hosts

[Matthias Klose]

Matthias Klose added the comment:

 IMHO, updating these patches to track the latest Python is a pointless goal.

sorry, no. it's the *only* way to get these patches upstream. The mingw patches 
will never see the light of the 3.3 branch. So the best thing to do is to 
actively submit the patches for trunk.

If you do want to maintain your patches, then split your set of patches into 
those which are upstream (and then pack-ported), and which are not yet upstream.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Matthias Klose]

Matthias Klose added the comment:

 However, we must go further and add that the patches *cannot* break any
 other native or cross-compilation, which - as I think Matthias is
 alluding to - is probably not the case with your patch. This issue is
 called cross and native build of python for mingw* hosts,

no, I just didn't check. And I probably won't check. However it's more likely 
that your patches are accepted if e.g. it doesn't break the normal Windows 
build and an unrelated build, e.g. for a Linux target.

Maybe you could even separate the patches into a patch supporting a native 
mingw build and a cmingw ross build?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Ray Donnelly]

Ray Donnelly added the comment:

Re: basing the patches against the latest master branch or targeting released 
versions, I wasn't clear enough about my thinking.

For sure, when trying to get any patches merged, the submitted patch must be 
re-based (forward ported) and tested against the master branch. 

But from the perspective of having a project that people can reliably expect to 
build a working MinGW-w64 Python from (until that elusive fully-merged-day 
comes), having patches based on the latest released versions seems to me to be 
the best thing to do (for that goal of course)

I'll keep forward porting these patches to each new release, and at those times 
I'll also submit some of them as issues to bugs.python.org as that's the best 
way for me to manage my limited spare time.

Do you know what the situation is with packaging/distutils2? I think our next 
good merge opportunity for the bulk of these patches will be when we can target 
that instead of distutils.

 Maybe you could even separate the patches into a patch supporting a native 
 mingw build and a cmingw ross build?

I'm already doing that:
https://github.com/mingwandroid/crucifixion-freedom/tree/master/patches/python/3.3.0

 and 0005 are now merged with upstream.  you merged today, 0005 was 
merged also (mostly via Roumen's patches which 2/3 of my patch duplicated). At 
this stage, the framework for cross compiling Python is mostly merged; other 
than 0065-cross-dont-add-multiarch-paths-if-cross-compiling.patch

The stumbling block for me is that there's no working example of 
cross-compilation in CPython so it will surely rot. This was what I tried to 
fix with my next patch, 0010-cross-darwin-feature.patch. I went for darwin 
instead of MinGW as the amount of changes needed is tiny but it was met with 
some resistance and hasn't seen any activity for 3 months:

http://bugs.python.org/issue16291

I should also clarify, by cross compilation I'm specifically excluding 
GNU-Linux/GNU-Linux combinations such as x86-GNU-Linux/arm-GNU-Linux as they're 
not as exhaustive a test-case as cross-compiling between different OSes.

 Maybe you could even separate the patches into a patch supporting a native 
 mingw build and a cmingw ross build?

Not sure if that was directed at me as my set of patches are logically split up 
I think. Ones with 'mingw' in the name relate to mingw (either cross or 
native), ones with 'msys' in the name relate to building mingw natively and 
ones with 'cross' in the name relate to general cross compilation matters; ones 
with 'hack' in the name generally need improving, rewriting or removing.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Roumen Petrov]

Roumen Petrov added the comment:

Matthias Klose wrote:

 Matthias Klose added the comment:

 some random comments about py3k-20121004-MINGW.patch:

 - Modules/_ctypes/libffi_msvc/win32.S
Please can you get rid of libffi_msvc and use libffi?
afaics, libffi has support for mingw32.

No python still use custom hack for MSVC.  win32.S include this hack and 
could be compiled with gcc.

 - there seem to be chunks which are unrelated to mingw, like:

 @@ -830,15 +926,18 @@ class PyBuildExt(build_ext):
   if have_usable_openssl:
   # The _hashlib module wraps optimized implementations
   # of hash functions from the OpenSSL library.
 +# NOTE: _hashlib require only OpenSSL crypto library !
   exts.append( Extension('_hashlib', ['_hashopenssl.c'],
  depends = ['hashlib.h'],
  include_dirs = ssl_incs,
  library_dirs = ssl_libs,
 -   libraries = ['ssl', 'crypto']) )
 +   libraries = ['crypto']) )

 please file separate issues.


 - why setup_info.in. looks like something which could be done with 
 get_config_var.

Yes and no.  get_config_var will use makefile variables (indirectly from 
sysconfigdata).
If you add a make macro(variable) then it will be installed.
Setup info is designed to communicate from configure to setup.py without 
to expose data to installation.

 - why re-reading files in setup.py, and grepping these for config options?

For historical reasons. Before other commits to start to look info some 
generated by autotool files.


 I think the patch would benefit from splitting it up into several 
 self-contained chunks.

May be.
I don't know how to group split. If is based on functionality patch 
order will be important as more then one patch will update near lines -
for instance pyport.h is difficult to split :(.
cygwinccompiler.py could be rewritten as separate issue, already opened.

 did you try to do builds for windows and linux after this patch was applied?

Title is correct. Hosts include native linux, cygwin, mingw and cross 
linux-mingw, linux-android ( not published ). All from one and the 
same source tree.

Roumen

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

This discussion got very long and it's not clear (to me at least) what the 
status of the patch is and whether it has been accepted for inclusion.
Maybe it makes sense to bring this up to python-dev mailing list instead of 
keep stressing this thread?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Ray Donnelly]

Ray Donnelly added the comment:

Roumen, I think it would be really great if you could split 
py3k-20121004-MINGW.patch up into separate bits.

The pthread stuff and libffi stuff being two obvious candidates for atomic 
parts.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Roumen Petrov]

Roumen Petrov added the comment:

As patch 0005-CROSS-revert-issue13150-i.e.-python-solution-with-_s.patch 
(CROSS-revert issue13150, i.e. python solution with _sysconfigdata.py instead 
Makefile) from issue3754 is now obsolete by issue 13547,  14774,  16342, 15298 
it is save to switch  sysconfigdata with proposed changes in PC/getpathp.c .

--
Added file: 
http://bugs.python.org/file28857/0003-MINGW-implement-exec-prefix-i.e.-use-content-of-pybu.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Luke Kenneth Casson Leighton]

Luke Kenneth Casson Leighton added the comment:

the last time this was brought up on python-dev the opinions of the primary 
python developers was made very very clear: anything that is not written by 
them is treated with extreme hostile and predudicial contempt.

what i mean by that is that the view of the primary python developers is that 
when they make decisions to decrease their workload, other people are *NOT* 
welcome to question those decisions.  [i questioned their decisions; all 
bugreports were terminated with excuses without good technical justification, 
and when i questioned those, access to the bugtracker was terminated].

roumen's work should have been encouraged and welcomed when it was first 
initiated - FIVE years ago.

the best bet therefore is to go over the heads of the primary python developers 
and to approach the Python Software Foundation's Board of Directors.  they are 
required to enact the PSF Charter in the best interests of the Python 
Programming Language.  what the existing primary python developers can or 
cannot cope with is of secondary consequence for the Board.

the main concern of the primary python developers was that they would be landed 
with yet another platform to support, and, as they are struggling to cope with 
the existing ones, they made - and make - absolutely any excuse that they can, 
regardless of actual merit, to ensure that no other platforms are added.

roumen, my opinion is that you have demonstrated over the past five years that 
you are committed to ensuring that python gets mingw support, and that you 
yourself are the best person to become the maintainer of python-mingw.  you 
are, already, de-facto, its maintainer.

answering some other points:

* python-mingw actually requires an entire new platform (built in to sys and os 
at a fundamental level).  neither the platform rules for cygwin, nor the 
platform rules for gcc, *nor* the platform rules for win32 correctly apply.  
the MSVC platform rules should also be obviously understood to be useless.  
mingw uses gcc, but it is gcc for win32: it therefore falls through *all* the 
existing platforms and requires its own separate class in distutils.

* the autoconf-generated pyconfig.h is so insanely slow to generate under mingw 
(any configure script running under MSYS is insanely slow) that it is virtually 
impossible to do any development.  also, the hard-coded config header file for 
win32 is also useless: firstly mingw doesn't have all the features of win32 
(the headers are reverse-engineered and so are incomplete).  secondly, the 
hard-coded config header file for win32 has MSVC-specific details in it that 
make it useless.

* distutils was frozen because several immature developers made improperly 
tested commits that caused massive disruption.  rather than put in proper 
review procedures, the primary python developers decided to terminate all 
possibility of ever adding in a new platform - ever - and began distutils2.  
this situation has to be reversed.  mingw *requires* that distutils have 
support for the mingw compiler.  a way to achieve this is to add separate files 
into distutils (which cannot be questioned as to their effect on existing files 
in distutils, because they are separate and therefore have zero effect), and 
then, once those files have been added and accepted, create a *ONE* line patch 
to distutils which pulls in the new mingw distutils functionality.  a one-line 
patch cannot really be argued with, esp. if it is only an include.

roumen and others: i recommend that you approach the Python Software Foundation 
and ask that this project be sponsored for 3 months and/or until it is 
completed and the patches are in.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Matthias Klose]

Matthias Klose added the comment:

some random comments about py3k-20121004-MINGW.patch:

- Modules/_ctypes/libffi_msvc/win32.S
  Please can you get rid of libffi_msvc and use libffi?
  afaics, libffi has support for mingw32.

- there seem to be chunks which are unrelated to mingw, like:

@@ -830,15 +926,18 @@ class PyBuildExt(build_ext):
 if have_usable_openssl:
 # The _hashlib module wraps optimized implementations
 # of hash functions from the OpenSSL library.
+# NOTE: _hashlib require only OpenSSL crypto library !
 exts.append( Extension('_hashlib', ['_hashopenssl.c'],
depends = ['hashlib.h'],
include_dirs = ssl_incs,
library_dirs = ssl_libs,
-   libraries = ['ssl', 'crypto']) )
+   libraries = ['crypto']) )

please file separate issues.


- why setup_info.in. looks like something which could be done with 
get_config_var.

- why re-reading files in setup.py, and grepping these for config options?


I think the patch would benefit from splitting it up into several 
self-contained chunks.

did you try to do builds for windows and linux after this patch was applied?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Jason Huntley]

Jason Huntley added the comment:

Finally! I was able to get all building on mingw64, gcc 4.7.2, and python 3.3. 
I had to take everybody's work here a little further, and I have produced a new 
patch which will build python33 using the latest minw64 compiler in an msys 
environment.

I cloned the 3.3 branch and applied the latest patch available here, 
py3k-20121004-MINGW.patch. I had to make quite a few more updates to get all 
the modules compiling with 4.7.2 and python find all the modules in the correct 
paths in an msys/mingw64 environment. I've compiled another patch, which 
addresses all the latest issues.

For all interested, run my setup script from the following location:

https://github.com/onepremise/MinGW-AD64S/

Just open dos in admin mode and run setup.bat.

This will build all the required dependencies and finally python 3.3.

--
Added file: http://bugs.python.org/file28699/py3k-20121227-MINGW.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Jason Huntley]

Jason Huntley added the comment:

Hey Guys,

I've furthered the progress of the patch I've been working on. THis patch 
builds Python3.3 in mingw64  msys using gcc 4.7.2 compiler. This patch can 
also enable and build the following modules:

ctypes
bz2
multiprocessing
sqlite3
ssl
pyexpat
zlib

Of course, there are many others, but these were some of the problem modules I 
encounted previously.

Partial success running the python tests. Many of the basic tests pass, I still 
have many to resolve. I have one thread test failing I need to resolve as it's 
preventing from all the tests running. Regardless, seems like some progress.

I think it would be great if everyone could collaborate better with this 
stuff;

I agree; I would really like for us to make this a joint effort. However, I'm 
noticing a ton of descrepancies with these patches. Some are removing pthreads, 
others are adding them back. Some patches are ignoring the PC/pyconfig while 
others are trying to generate a custom mingw version. It seems some are also 
compiling and using the mingw version outside of the msys shell, which I'm not 
certain of the advantages there when u can just use the mscv version. I think 
we need to take a minute to agree on some specific goals and criteria this 
mingw port should be used for. Just to kick things off, here are some of the 
requirements which I think we should aim for.

* The build must be reproducible in the mingw msys shell, in the default 
environment described on the mingw64 site, also included some other good doc 
references:
  * https://sourceforge.net/apps/trac/mingw-w64/wiki/MSYS
  * 
https://sourceforge.net/apps/trac/mingw-w64/wiki/Cross%20Win32%20and%20Win64%20compiler
  * http://www.gaia-gis.it/gaia-sins/mingw64_how_to.html#env
  * These documented steps have also been automated by the sripts here 
https://github.com/onepremise/MinGW-AD64S
  * Currently, I have the mingw build using the PC platform specified config 
headers and source.
* too many issues reside around enabling posix threads as the pthread_t 
identifier in mingw doesn't match existing py thread framework returning longs.
* As a result, the patch I provided builds with the ntthread implementation.
* In addition, PC/pyconfig.h and PC/getpath.c are being used with slight 
modifications for __MINGW32__
  * Initial python mingw64 deployment layout, version identifiers should follow 
existing scheme, 3.3 not 33:
* should deploy libraries to /mingw/lib, libpython3.3.a, Format: 
libpython3.3.dll, libpython3.3.dll.a.
* executable binaries should exist in /mingw/bin. Format: python.exe, 
python3.3.exe.
* The python script library deploys to /mingw/lib/Python3.3.
  * /mingw/lib/Python3.3/lib-dynload
  * /mingw/lib/Python3.3/site-packates
* Headers located /mingw/include/Python3.3
  * Having Python build isn't satisfactory enough to promote a patch, patches 
should not cause new test failures or regressions. Preferably they should 
resolve core compilation issues and module compilation issues, while not 
creating new test failures and preferrably address/fix failing tests. (python 
-m test)
  * covered on the dev-guilde, http://docs.python.org/devguide/
  
Please feel free to contriube, update, remove, discuss etc...
  
https://github.com/mingwandroid/crucifixion-freedom/tree/master/patches/python/3.3.0
 [Python cross and MinGW compilation]
https://github.com/niXman/mingw-builds/tree/master/patches/Python-3.3.0 
[MinGW toolchains including Python pretty-printing GDB]

I tried some of these patches, but they aren't very organinzed. I really need 
some docemntaiton to better understand what each patch does and which pairs go 
together and which don't. Wiki over there is blank.

in fact, just after Roumen's huge patch we apply some patches to remove large 
parts 
of it so we can re-introduce those independent fixes via separate patches).

I had to do the same as well. There were regressions in some of these patches 
causing the python tests to fail and prevent the python setup.py script from 
running.

Our patching strategies are more sane (many patches as opposed to a 
monolithic patch - 
Given how intrusive these patches are, getting them merged into CPython is a 
bit of a 
pipe dream (I'd love to be wrong about that), so making it as easy as 
possible 
for the interested parties to maintain (forward port), build, test and 
package 
Python  releases is probably the best we can hope for. If we can make the 
patchessmall and independently reviewable then maybe some parts could be 
merged.

I prefer this as well. It will make code reviewing easier. Otherwise, some kind 
of staging location would be useful. I'm fine using rietveld, the tracker. We 
just need to name the patches better, including prefix/dates or provide 
descriptions in the comments below. Right now it's hard to tell which patches 
to apply just from reading through the comments

I would like to disagree with 'hard codded' compiler flags into 

[issue3871] cross and native build of python for mingw* hosts

[Jason Huntley]

Changes by Jason Huntley oneprem...@gmail.com:


Removed file: http://bugs.python.org/file28699/py3k-20121227-MINGW.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Jason Huntley]

Jason Huntley added the comment:

Hey Guys,

I've furthered the progress of the patch I've been working on. THis patch 
builds Python3.3 in mingw64  msys using gcc 4.7.2 compiler. This patch can 
also enable and build the following modules:

ctypes
bz2
multiprocessing
sqlite3
ssl
pyexpat
zlib

Of course, there are many others, but these were some of the problem modules I 
encounted previously.

Partial success running the python tests. Many of the basic tests pass, I still 
have many to resolve. I have one thread test failing I need to resolve as it's 
preventing from all the tests running. Regardless, seems like some progress.

I think it would be great if everyone could collaborate better with this 
stuff;

I agree; I would really like for us to make this a joint effort. However, I'm 
noticing a ton of descrepancies with these patches. Some are removing pthreads, 
others are adding them back. Some patches are ignoring the PC/pyconfig while 
others are trying to generate a custom mingw version. It seems some are also 
compiling and using the mingw version outside of the msys shell, which I'm not 
certain of the advantages there when u can just use the mscv version. I think 
we need to take a minute to agree on some specific goals and criteria this 
mingw port should be used for. Just to kick things off, here are some of the 
requirements which I think we should aim for.

* The build must be reproducible in the mingw msys shell, in the default 
environment described on the mingw64 site, also included some other good doc 
references:
  * https://sourceforge.net/apps/trac/mingw-w64/wiki/MSYS
  * 
https://sourceforge.net/apps/trac/mingw-w64/wiki/Cross%20Win32%20and%20Win64%20compiler
  * http://www.gaia-gis.it/gaia-sins/mingw64_how_to.html#env
  * These documented steps have also been automated by the sripts here 
https://github.com/onepremise/MinGW-AD64S
  * Currently, I have the mingw build using the PC platform specified config 
headers and source.
* too many issues reside around enabling posix threads as the pthread_t 
identifier in mingw doesn't match existing py thread framework returning longs.
* As a result, the patch I provided builds with the ntthread implementation.
* In addition, PC/pyconfig.h and PC/getpath.c are being used with slight 
modifications for __MINGW32__
  * Initial python mingw64 deployment layout, version identifiers should follow 
existing scheme, 3.3 not 33:
* should deploy libraries to /mingw/lib, libpython3.3.a, Format: 
libpython3.3.dll, libpython3.3.dll.a.
* executable binaries should exist in /mingw/bin. Format: python.exe, 
python3.3.exe.
* The python script library deploys to /mingw/lib/Python3.3.
  * /mingw/lib/Python3.3/lib-dynload
  * /mingw/lib/Python3.3/site-packates
* Headers located /mingw/include/Python3.3
  * Having Python build isn't satisfactory enough to promote a patch, patches 
should not cause new test failures or regressions. Preferably they should 
resolve core compilation issues and module compilation issues, while not 
creating new test failures and preferrably address/fix failing tests. (python 
-m test)
  * covered on the dev-guilde, http://docs.python.org/devguide/
  
Please feel free to contriube, update, remove, discuss etc...
  
https://github.com/mingwandroid/crucifixion-freedom/tree/master/patches/python/3.3.0
 [Python cross and MinGW compilation]
https://github.com/niXman/mingw-builds/tree/master/patches/Python-3.3.0 
[MinGW toolchains including Python pretty-printing GDB]

I tried some of these patches, but they aren't very organinzed. I really need 
some docemntaiton to better understand what each patch does and which pairs go 
together and which don't. Wiki over there is blank.

in fact, just after Roumen's huge patch we apply some patches to remove large 
parts 
of it so we can re-introduce those independent fixes via separate patches).

I had to do the same as well. There were regressions in some of these patches 
causing the python tests to fail and prevent the python setup.py script from 
running.

Our patching strategies are more sane (many patches as opposed to a 
monolithic patch - 
Given how intrusive these patches are, getting them merged into CPython is a 
bit of a 
pipe dream (I'd love to be wrong about that), so making it as easy as 
possible 
for the interested parties to maintain (forward port), build, test and 
package 
Python  releases is probably the best we can hope for. If we can make the 
patchessmall and independently reviewable then maybe some parts could be 
merged.

I prefer this as well. It will make code reviewing easier. Otherwise, some kind 
of staging location would be useful. I'm fine using rietveld, the tracker. We 
just need to name the patches better, including prefix/dates or provide 
descriptions in the comments below. Right now it's hard to tell which patches 
to apply just from reading through the comments

I would like to disagree with 'hard codded' compiler flags into 

[issue3871] cross and native build of python for mingw* hosts

[Roumen Petrov]

Roumen Petrov added the comment:

Jason Huntley wrote:
 Jason Huntley added the comment:

 Finally! I was able to get all building on mingw64, gcc 4.7.2, and python 
 3.3. I had to take everybody's work here a little further, and I have 
 produced a new patch which will build python33 using the latest minw64 
 compiler in an msys environment.

 I cloned the 3.3 branch and applied the latest patch available here, 
 py3k-20121004-MINGW.patch. I had to make quite a few more updates to get all 
 the modules compiling with 4.7.2 and python find all the modules in the 
 correct paths in an msys/mingw64 environment. I've compiled another patch, 
 which addresses all the latest issues.
Until now I did not change Mingw compiler class, just because is 
reported in a number of separate issues. I would like to disagree with 
'hard codded' compiler flags into python code. For instance 
-mms-bitfields is default for GCC 4.7+ . Also -pthread has to be 
detected by configure if is required for host platform.
Another point is mno-cygwin - reported in issue 12641. Unfortunately one 
of PSF project directors misread error message and issue remain unresolved.

Please provide more details for change in Lib/ntpath.py .

What is reasons to add more build-in modules (time, math, ...) ?

Why is updated PC/pyconfig.h ? This is so called by python community 
posix-build and pyconfig.h is generated by configure script. Did I 
miss something ?

File configure.ac contain so many unrelated changes - replace tabs with 
spaces. Please post as separate diff you changes.

Why is updated Lib/plat-generic/regen ?

Currently my private repository reintroduce support _sysconfigdata.py 
but this require some changes in PC/getpathp.c . I left current 
installation from makefile as is and after this with an extra script i 
move file for location same as primary build. This is reason do not 
include more search paths in this file. Issue for unification of python 
installation across platforms may be will be resolved with 14302 and 9654.

Roumen

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Jason Huntley]

Jason Huntley added the comment:

Finally! I was able to get all building on mingw64, gcc 4.7.2, and python 3.3. 
I had to take everybody's work here a little further, and I have produced a new 
patch which will build python33 using the latest minw64 compiler in an msys 
environment.

I cloned the 3.3 branch and applied the latest patch available here, 
py3k-20121004-MINGW.patch. I had to make quite a few more updates to get all 
the modules compiling with 4.7.2 and python find all the modules in the correct 
paths in an msys/mingw64 environment. I've compiled another patch, which 
addresses all the latest issues.

For all interested, run my setup script from the following location:

https://github.com/onepremise/MinGW-AD64S/

Just open dos in admin mode and run setup.bat.

This will build all the required dependencies and finally python 3.3.

--
Added file: http://bugs.python.org/file28471/py3k-20121227-MINGW.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Ray Donnelly]

Ray Donnelly added the comment:

Good work Jason,

I think it would be great if everyone could collaborate better with this stuff; 
I'm as guilty as the next person on that score though. I might have been able 
to save you some effort by pointing you at either:

https://github.com/mingwandroid/crucifixion-freedom/tree/master/patches/python/3.3.0
 [Python cross and MinGW compilation]

or:

https://github.com/niXman/mingw-builds/tree/master/patches/Python-3.3.0 [MinGW 
toolchains including Python pretty-printing GDB]

..there might be some patches in there that are useful for you. Both these 
projects use basically the same patches, the 'source' of them is 
crucifixion-freedom.

Our patching strategies are more sane (many patches as opposed to a monolithic 
patch - in fact, just after Roumen's huge patch we apply some patches to remove 
large parts of it so we can re-introduce those independent fixes via separate 
patches).

Both crucifixion-freedom and mingw-builds also provide patches for 2.7.3 and 
support building tkinter and IDLE. The unique feature of crucifixion-freedom is 
fairly comprehensive cross compilation capability 
({Linux,Windows,Mac}*{32bit,64bit}).

Given how intrusive these patches are, getting them merged into CPython is a 
bit of a pipe dream (I'd love to be wrong about that), so making it as easy as 
possible for the interested parties to maintain (forward port), build, test and 
package Python releases is probably the best we can hope for. If we can make 
the patches small and independently reviewable then maybe some parts could be 
merged.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Roumen Petrov]

Roumen Petrov added the comment:

Jason Huntley wrote:

 patch -t -p1  python-py3k-20120729-MINGW.patch
Please run autoheader (optional) and autoconf (required .
 ./configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 
 --prefix=/mingw
...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Jason Huntley]

Jason Huntley added the comment:

I'm attempting to build Python-3.3.0 with mingw64. I get a minute or two into 
the build and fail with this error:

./Include/pythonrun.h:178:1: warning: function declaration isn't a prototype [-W
strict-prototypes]
./Modules/getpath.c: In function 'isfile':
./Modules/getpath.c:153:5: warning: implicit declaration of function '_Py_wstat'
 [-Wimplicit-function-declaration]
./Modules/getpath.c: In function 'find_env_config_value':
./Modules/getpath.c:298:17: error: too many arguments to function 'wcstok'
In file included from c:\projects\tools\osm\dev\mapnik\custom\environment\instal
l\mingw64\bin\../lib/gcc/x86_64-w64-mingw32/4.7.2/../../../../x86_64-w64-mingw32
/include/io.h:10:0,
 from ./PC/pyconfig.h:68,
 from ./Include/Python.h:8,
 from ./Modules/getpath.c:3:

It seems some people here were using this patch against 3.3.0 at one point. Did 
you guys ever succeed? Once I've applied the patch, i can get a little further, 
but the build eventually fails again. Also, conftest continually fails to run 
during configure.

I'm using the following settings to get the build going:

export CFLAGS=-I/mingw/include
export LDFLAGS=-L/mingw/lib
export CPPFLAGS=-I/mingw/include 
export CFLAGS=$CFLAGS -I./PC -DMS_WIN64 -D__MINGW32__


#http://bugs.python.org/issue3871
#wget http://bugs.python.org/file27474/py3k-20121004-MINGW.patch
wget http://bugs.python.org/file26572/python-py3k-20120729-MINGW.patch

#patch -t -p1  py3k-20121004-MINGW.patch
patch -t -p1  python-py3k-20120729-MINGW.patch

./configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --prefix=/mingw

make

If I patch before configure, I get the following exception when running 
./configure:

bad word @INITSYS@ in @INITSYS@ posixmodule.c
mv: cannot stat `config.c': No such file or directory

After configure, is where it gets further, until it finally fails on:

c:\projects\tools\osm\dev\mapnik\custom\environment\install\mingw64\bin\../lib/g
cc/x86_64-w64-mingw32/4.7.2/../../../../x86_64-w64-mingw32/include/io.h:311:15:
note: previous declaration of 'chmod' was here
./Modules/posixmodule.c:262:32: error: unknown type name 'uid_t'
./Modules/posixmodule.c:262:39: error: unknown type name 'gid_t'

You can replicate my environment exactly by running the following deployment 
scripts from my project here:

https://github.com/onepremise/MinGW-AD64S/

Just open dos in admin mode and run setup.bat.

Any help is greatly appreciated. Thanks!

--
nosy: +jhuntley

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Tshepang Lekhonkhobe]

Changes by Tshepang Lekhonkhobe tshep...@gmail.com:


--
nosy: +tshepang

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3871] cross and native build of python for mingw* hosts

[Roumen Petrov]

Roumen Petrov added the comment:

Extract of 2012-10-04 is after patch set py3k-20121004-CROSS.tgz from issue 
#3754 .

--
title: cross and native build of python for mingw32 with packaging - cross and 
native build of python for mingw* hosts
Added file: http://bugs.python.org/file27474/py3k-20121004-MINGW.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3871
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com