This seems like a lot of work to support Windows users who might want to use pre-compiled python modules.
I think we should distribute a VS2015 based version of Python 2.7 binaries and call it a day. We can worry about 2020 problems in 2020. =) Reasonable people may disagree. On Tue, Feb 24, 2015 at 11:56 PM, Zachary Turner <ztur...@google.com> wrote: > A little background: The single biggest painpoint for working with LLDB on > Windows currently is Python. There is a long > <https://mail.python.org/pipermail/distutils-sig/2013-February/020006.html> > documented <https://docs.python.org/2/extending/windows.html> history of > the problems with python extension modules on Windows. Part > <https://msdn.microsoft.com/en-us/library/ms235460.aspx> of it is > Microsoft's <https://msdn.microsoft.com/en-us/library/bb531344.aspx> > fault, part of it is Python's fault (PEP 384 > <https://www.python.org/dev/peps/pep-0384/> attempts to solve this, but > it appears stalled), but the end result is that it's really terrible and > there's nothing anyone can do about it. > > The implications of this for LLDB on Windows are the following: > 1) Anyone building LLDB on Windows needs to compile Python from source > 2) Even after doing so, configuring the LLDB build is difficult and > requires a lot of manual steps > 3) You can't use any other extension modules in your custom built version > of python, including any of the over 50,000 from PyPI, unless you also > build them from source, which isn't even always possible. > > If you want to be compatible with the binary release of Python and > interoperate with the version of Python that probably 99% of Windows people > have on their machine, you *must* compile your extension module with a > very old version of the compiler. So old, in fact, that it's almost > end-of-lifed. If it weren't for Python actually, it would have been > end-of-lifed already, and Microsoft ships a free version > <http://www.microsoft.com/en-us/download/details.aspx?id=44266> of this > toolchain for the express purpose of compiling python extension modules. > > I've been thinking about this for many months, and I believe I finally > have a solution (2 solutions actually, although I think we should do both. > I'll get to that later). Both will probably be painful, but in the end for > the better on all platforms. > > *Solution 1:* *Decouple embedded python code from LLDB* > *Rationale:* Currently calls to embedded python code live in a few > different places. Primarily in source/Interpreter (e.g. > ScriptInterpreterPython) and the generated SWIG code, but there's a few > utility headers scattered around. > > I'm proposing moving all of this code to a single shared library with > nothing else and creating some heavy restrictions on what kind of code can > go into this library, primarily to satisfy the requirement that it be > compilable with the old version of the compiler in question. These > restrictions would be: > 1) Can't use fancy C++. It's hard to say exactly what subset of C++ we > could use here, but a good rough approximation is to say C++98. > 2) Can't depend on any LLVM headers or even other LLDB libraries. Other > LLDB projects could depend on it, but it has to stand alone to guarantee > that it doesn't pick up C++ that won't compile under the old MSVC compiler. > > I understand that the first restriction is very annoying, but it would > only be imposed upon a very small amount of source code. About 2 or 3 > source files. > > The second restriction, and the decoupling as a whole will probably cause > some pain and breakage for out-of-tree code, but I believe it's a positive > in the long run. In particular, it opens the door to replacing the > embedded interpreter with that of a different language. By separating the > code in this fashion, all one has to do is write a new module for their own > language and initialize it instead of ScriptInterpreterPython. I've > already seen people asking on the list about generating bindings for other > languages and replacing the interpreter, and i believe a separation of this > kind is a pre-requisite anyway. > > *Solution 2:* *Upgrade to Python 3.5* > *Rationale:* Hopefully I didn't send you running for the hills. This is > going to have to happen someday anyway. Python 2.7 end of life is set to > 2020 <https://hg.python.org/peps/rev/76d43e52d978>. Seems like a long > time, but it'll be here before we know it, and if we aren't prepared, it's > going to be a world of hurt. > > Why does this help us on Windows? Visual Studio 2015, which releases > sometime this year, is finally set to have a stable ABI > <http://blogs.msdn.com/b/vcblog/archive/2014/06/10/the-great-crt-refactoring.aspx> > for it's CRT. This means that any application compiled against VS2015 or > later will be forward compatible with future versions of the CRT forever > <https://mail.python.org/pipermail/python-dev/2014-June/134888.html>. > Python 3.5 is not yet released, but the current proposal is for Python 3.5 > <https://mail.python.org/pipermail/python-dev/2014-June/134866.html> to > ship its binary release compiled with VC++ 2015, effectively killing this > problem. > > I understand that there is a lot of out-of-tree code that is written > against Python 2.7. I would encourage the people who own this code to > begin thinking about migrating to Python 3 soon. In the meantime, I > believe we can begin to address this in-tree in 3 main phases. > > 1) Port the test suite to Python 3.5, using a subset of Python 3.5 that is > also compatible with 2.7. This ensures no out of tree code is broken. > 2) Upgrading ScriptInterpreterPython with preprocessor flags to select > whether you want to link against Python 2.7 and 3.5, and expose these > options from CMake and Xcode so you can choose what you link against. > 3) Making multiple buildbots with different configurations and different > versions of Python to get better code coverage to ensure that tests work > under both language versions. > > I realize that the impact of both of these changes is high, but we have a > very strong desire to solve this problem on Windows and so we need to push > some kind of solution forward. I think both solutions actually contribute > to the long term benefit of LLDB as a whole, and so I think it's worth > seriously considering both of these and trying to come up with a path > forward. > > _______________________________________________ > lldb-dev mailing list > lldb-dev@cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev > > -- Vince Harron | Technical Lead Manager | vhar...@google.com | 858-442-0868
_______________________________________________ lldb-dev mailing list lldb-dev@cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
