On 27 January 2010 08:37, David Malcolm <dmalc...@redhat.com> wrote:
> On Thu, 2010-01-21 at 14:46 -0800, Jeffrey Yasskin wrote:
> > On Thu, Jan 21, 2010 at 10:09 AM, Hanno Schlichting <ha...@hannosch.eu>
> wrote:
> > > I'm a relative outsider to core development (I'm just a Plone release
> > > manager), but'll allow myself a couple of questions. Feel free to
> > > ignore them, if you think they are not relevant at this point :-) I'd
> > > note that I'm generally enthusiastic and supportive of the proposal :)
> > > As a data point, I can add that all tests of Zope 2.12 / Plone 4.0 and
> > > their dependency set run fine under Unladen Swallow.
> >
> > Hi, thanks for checking that!
> >
> > > On Wed, Jan 20, 2010 at 11:27 PM, Collin Winter <
> collinwin...@google.com> wrote:
> > >> We have chosen to reuse a set of existing compiler libraries called
> LLVM
> > >> [#llvm]_ for code generation and code optimization.
> > >
> > > Would it be prudent to ask for more information about the llvm
> > > project? Especially in terms of its non-code related aspects. I can
> > > try to hunt down this information myself, but as a complete outsider
> > > to the llvm project this takes much longer, compared to someone who
> > > has interacted with the project as closely as you have.
> > >
> > > Questions like:
> > >
>
> [snip]
>
> > >> Managing LLVM Releases, C++ API Changes
> > >> ---------------------------------------
> > >>
> > >> LLVM is released regularly every six months. This means that LLVM may
> be
> > >> released two or three times during the course of development of a
> CPython 3.x
> > >> release. Each LLVM release brings newer and more powerful
> optimizations,
> > >> improved platform support and more sophisticated code generation.
> > >
> > > How does the support and maintenance policy of llvm releases look
> > > like? If a Python version is pegged to a specific llvm release, it
> > > needs to be able to rely on critical bug fixes and security fixes to
> > > be made for that release for a rather prolonged time. How does this
> > > match the llvm policies given their frequent time based releases?
> >
> > LLVM doesn't currently do "dot" releases. So, once 2.7 is released,
> > it's very unlikely there would be a 2.6.1. They do make release
> > branches, and they've said they're open to dot releases if someone
> > else does them, so if we need a patch release for some issue we could
> > make it ourselves. I recognize that's not ideal, but I also expect
> > that we'll be able to work around LLVM bugs with changes in Python,
> > rather than needing to change LLVM.
>
> [snip]
>
> (I don't think the following has specifically been asked yet, though
> this thread has become large)
>
> As a downstream distributor of Python, a major pain point for me is when
> Python embeds a copy of a library's source code, rather than linking
> against a system library (zlib, libffi and expat spring to mind): if
> bugs (e.g. security issues) arise in a library, I have to go chasing
> down all of the embedded copies of the library, rather than having
> dynamic linking deal with it for me.
>
> So I have some concerns about having a copy of LLVM embedded in Python's
> source tree, which I believe other distributors of Python would echo;
> our rough preference ordering is:
>
> dynamic linking > static linking > source code copy
>
> I would like CPython to be faster, and if it means dynamically linking
> against the system LLVM, that's probably OK (though I have some C++
> concerns already discussed elsewhere in this thread). If it means
> statically linking, or worse, having a separate copy of the LLVM source
> as an implementation detail of CPython, that would be painful.
>
> I see that the u-s developers have been run into issues in LLVM itself,
> and fixed them (bravo!), and seem to have done a good job of sending
> those fixes back to LLVM for inclusion. [1]
>
Hi David, I'm unladen-swallow's resident "llvm consultant", so I'll answer
the questions I can from an llvm upstream point of view.
> Some questions for the U-S devs:
> - will it be possible to dynamically link against the system LLVM?
> (the PEP currently seems to speak of statically linking against it)
>
Sadly, LLVM still only offers static libraries (there are two exceptions,
but they don't fit unladen-swallow). This is a something we want but
nobody's stepped up to make it happen yet.
> - does the PEP anticipate that the Python source tree will start
> embedding a copy of the LLVM source tree?
> - if so, what can be done to mitigate the risk of drift from upstream?
> (this is the motivation for some of the following questions)
>
Jeff and I are working to make sure that everything unladen-swallow needs is
in LLVM 2.7 when it releases.
- to what extent do you anticipate further changes needed in LLVM for
> U-S? (given the work you've already put in, I expect the answer is
> "probably a lot, but we can't know what those will be yet")
>
Unladen-swallow works with upstream LLVM now. The changes we're making
include refactoring interfaces and adding new optimization features which
python would make use of in the future.
- do you anticipate all of these changes being accepted by the
> upstream LLVM maintainers?
>
Yup. The patches generally aren't controvertial :-)
> - to what extent would these changes be likely to break API and/or ABI
> compat with other users of LLVM (i.e. would a downstream distributor of
> CPython be able to simply apply the necessary patches to the system LLVM
> in order to track? if they did so, would it require a recompilation of
> all of the other users of the system LLVM?)
>
That's complicated. LLVM's API and ABI are not stable in the sense that we
make any effort to preserve API/ABI compatibility between versions. To make
matters worse, because LLVM is a library 'kit of parts' almost every API is
exposed publicly. But what happens in reality is that people use
higher-level APIs, or only such a small part of the API that we can change
things without requiring a change in users source, or only a mechanical one.
I can get into concrete examples using LLVM patches coming out of
unladen-swallow if you want solid evidence.
The thumb rule is that any change that actually impacts LLVM's core internal
language will almost certainly be API breaking, while things like bug fixes
to the JIT and new minor optimizations or entirely new optimization passes
don't change the API at all. ABI breakage is a whole different beast, as you
can break the C++ ABI just by changing the order of methods declared in a
class. If you want to do point releases of LLVM, you would have to think
about it very carefully, as you would for any C++ ABI problem.
- if Python needed to make a dot-release of LLVM, would LLVM allow
> Python to increment the SONAME version identifying the ABI within the
> DSO (".so") files, and guarantee not to reuse that SONAME version? (so
> that automated ABI dependency tracking in e.g. RPM can identify the ABI
> incompatibilities without being stomped on by a future upstream LLVM
> release)
>
I don't see why not. Just make sure you let us know on the llvm-dev mailing
list when you use an SONAME.
I should mention that the reason LLVM doesn't have dot-releases now is that
we don't have users with a vested interest in producing them. If the python
developers thought it were important, it shouldn't be too much work for you
to own the release branch after a release. The hard part is finding the
platforms (and time) with which to re-run the nightly tests.
- is your aim to minimize the difference between upstream LLVM and the
> U-S copy of LLVM? ("yes", probably)
>
Yes, the aim is to use a bog-standard LLVM 2.7 release.
Nick
- will you publish/track the differences between upstream LLVM and the
> U-S copy of LLVM somewhere? I see that you currently do this here: [2],
> though it strikes me that the canonical representation of the LLVM
> you're using is in the embedded copy in Utils/llvm, rather than e.g. a
> recipe like "grab upstream release foo and apply this short list of
> patches" (a distributed SCM might help here, or a tool like jhbuild [3])
> - would be it be possible to continuously build and benchmark
> --with-system-llvm rather than against the embedded LLVM copy?
> (etc)
>
> Thanks!
> Dave
>
> [1] e.g. http://llvm.org/docs/DebuggingJITedCode.html
> [2] http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches
> [3] http://live.gnome.org/Jhbuild
>
>
>
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
