Re: [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.
On Tue, Nov 17, 2015 at 8:03 PM Todd Fialawrote: > Nothing concrete at the moment; however, it could be interesting to look > at the clang community and see what could be done for llvm-based language > implementations. The angle that I think would be interesting would be if > we can generate bindings more effectively based on the in-depth > understanding of the language that is afforded by languages built on top of > LLVM. This is probably less interesting for Python (particularly since we > have a functioning solution) and more interesting for languages built on > LLVM or clang. > > Honestly, though, I haven't spent much time on that. > > For the time being, I am going to not change the path for everyone on > swig, and only use a static binding if swig cannot be found. This will be > minimal impact for everyone and doesn't interfere with anyone using a > specific version of swig. We can revisit larger questions about > who/what/when on static bindings after we gain some experience with > enabling them for those who don't have swig. We can review and adjust > based on our collective experience. The two files this seems like it will > be are the LLDBWrapPython.cpp and the lldb.py file that comes out of > python. I hope to have this working in the next day or so. > To try this another way, I really would like to voice my opinion very very strongly for moving away from having different ways of doing things for different platforms / build configurations / etc unless it is *required* to support a hard requirement of someone's environment. Here's our current configuration matrix. Platforms: Windows, Linux, FreeBSD, Darwin, NetBSD, Other(?) Build Systems: CMake, Xcode SWIG version: 1.3x, latest Python version: None, 2.7, 3.x (in progress) SWIG Binding Generation: on-the-fly, static (proposed) In all of these cases (except the proposed), the matrix choices are justifiable because they are there to support a hard requirement of someone's environment, and I do not think we should grow for anything that is not also a hard requirement of someone's environment. We definitely should not grow it out of convenience, and *especially* not if it's only a minor convenience. So I still am looking for a clear answer regarding what problem this is solving. Is not having a swig binary on every machine a hard requirement? You said > We can revisit larger questions about who/what/when on static bindings after we gain some experience with enabling them *for those who don't have swig*" And my question is, who doesn't have swig? Maybe there is a legitimate use case, I just want to understand what that is before we add more different ways of building. I mentioned earlier that one way there would be a definite tangible benefit is if we could use these static bindings in conjunction with a swig bot that would automatically generate swig on a remote server and send you back the result. This way we could remove one item from the configuration matrix, which I think we all agree is a good thing based on the fact that the original idea was to get everyone on 1.3x (which isn't possible since it doesn't work well with Python 3.5). Compare: Platforms: Windows, Linux, FreeBSD, OSX, NetBSD, Other(?) Build Systems: CMake, Xcode SWIG version: 1.3x, latest Python version: None, 2.7, 3.x (in progress) SWIG Binding Generation: on-the-fly, static with Platforms: Windows, Linux, FreeBSD, OSX, NetBSD, Other(?) Build Systems: CMake, Xcode Python version: None, 2.7, 3.x (in progress) The latter is much better, right? Can we discuss whether this swig server is a viable solution for getting to a single system that works for everyone? Again, I'm willing to do the brunt of the work getting it up and running if it is (I can probably reuse the portion of work you've already done related to running swig on the input files) If the goal is just some kind of cleanup where it would be nice to not have to install swig, I'm specifically arguing against that, because I don't think that's a strong enough case to add one item to the configuration matrix. > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.
Usage of static bindings for the prepare_bindings.py script went in here: r253448. Only Xcode provides the flag to use it. The commit description indicates how it works in this incarnation. Essentially it only uses the static Python binding if and only if a swig isn't specified or cannot be found. I'll be testing this on Ubuntu 14.04 and 15.10 in the morning. Once I have that working, I'll provide a cmake flag to vector over to it, defaulting to not do so. -Todd On Tue, Nov 17, 2015 at 8:12 PM, Todd Fialawrote: > > that comes out of python > > that comes out of swig, rather (i.e. the binding generation output). > > On Tue, Nov 17, 2015 at 8:03 PM, Todd Fiala wrote: > >> Nothing concrete at the moment; however, it could be interesting to look >> at the clang community and see what could be done for llvm-based language >> implementations. The angle that I think would be interesting would be if >> we can generate bindings more effectively based on the in-depth >> understanding of the language that is afforded by languages built on top of >> LLVM. This is probably less interesting for Python (particularly since we >> have a functioning solution) and more interesting for languages built on >> LLVM or clang. >> >> Honestly, though, I haven't spent much time on that. >> >> For the time being, I am going to not change the path for everyone on >> swig, and only use a static binding if swig cannot be found. This will be >> minimal impact for everyone and doesn't interfere with anyone using a >> specific version of swig. We can revisit larger questions about >> who/what/when on static bindings after we gain some experience with >> enabling them for those who don't have swig. We can review and adjust >> based on our collective experience. The two files this seems like it will >> be are the LLDBWrapPython.cpp and the lldb.py file that comes out of >> python. I hope to have this working in the next day or so. >> >> >> On Tue, Nov 17, 2015 at 7:26 PM, Bruce Mitchener < >> bruce.mitche...@gmail.com> wrote: >> >>> Stepping one step back further in the thread ... >>> >>> On Wed, Nov 18, 2015 at 8:35 AM, Zachary Turner via lldb-commits < >>> lldb-commits@lists.llvm.org> wrote: >>> Moving this back over to the list since I'm sure others have some input here. Also +lldb-dev since it has more visibility than lldb-commits. On Tue, Nov 17, 2015 at 11:25 AM Zachary Turner wrote: > On Tue, Nov 17, 2015 at 8:18 AM Todd Fiala > wrote: > >> Breaking out the binding generation into a separate step will also be >> important for a couple reasons: >> >> * (from before) I want to eliminate the requirement for the vast >> majority of the builds to have a swig on their system, and >> >> * (not stated before) we'd like to move away from swig for binding >> generation at some point. >> > >>> Is there any discussion or thoughts about what the options would be for >>> moving away from swig? >>> >>> - Bruce >>> >>> >> >> >> >> -- >> -Todd >> > > > > -- > -Todd > -- -Todd ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.
Is there any way to have the step that runs the swig executable not be part of this script? Seems like running swig should be a build step, and running these scripts on the output of swig should be a step that follows On Mon, Nov 16, 2015 at 11:20 PM Todd Fiala via lldb-commits < lldb-commits@lists.llvm.org> wrote: > Author: tfiala > Date: Tue Nov 17 01:17:38 2015 > New Revision: 253317 > > URL: http://llvm.org/viewvc/llvm-project?rev=253317=rev > Log: > Add Pythonic language binding wrapper generation script. > > This is only used by Xcode at the moment. It replaces the > buildSwigWrapperClasses.py and related per-script-language > scripts. It also fixes a couple bugs in those w/r/t Xcode > usage: > > * the presence of the GCC_PREPROCESSOR_DEFINITIONS env var > should not be short-circuiting generation of the language > binding; rather, only if LLDB_DISABLE_PYTHON is present > within that environment variable. > > * some logic around what to do when building in "non-Makefile" > mode. I've switched the handling of that to be on a > "--framework" flag - if specified, we build an OS X-style > framework; otherwise, we go with non. > > Putting this up now only attached to the Xcode build so > others can look at it but not be affected by it yet. > After this, I'll tackle the finalizer, along with trying > it locally on Linux. > > Added: > lldb/trunk/scripts/Python/prepare_binding_Python.py > lldb/trunk/scripts/prepare_bindings.py (with props) > Modified: > lldb/trunk/lldb.xcodeproj/project.pbxproj > > Modified: lldb/trunk/lldb.xcodeproj/project.pbxproj > URL: > http://llvm.org/viewvc/llvm-project/lldb/trunk/lldb.xcodeproj/project.pbxproj?rev=253317=253316=253317=diff > > == > --- lldb/trunk/lldb.xcodeproj/project.pbxproj (original) > +++ lldb/trunk/lldb.xcodeproj/project.pbxproj Tue Nov 17 01:17:38 2015 > @@ -5984,7 +5984,7 @@ > isa = PBXNativeTarget; > buildConfigurationList = 2668020B115FD0EE008E1FE4 > /* Build configuration list for PBXNativeTarget "LLDB" */; > buildPhases = ( > - 26DC6A5813380D4300FF7998 /* Build swig > wrapper classes */, > + 26DC6A5813380D4300FF7998 /* Prepare Swig > Bindings */, > 26680202115FD0ED008E1FE4 /* Headers */, > 26680203115FD0ED008E1FE4 /* Resources */, > 26680204115FD0ED008E1FE4 /* Sources */, > @@ -6214,19 +6214,19 @@ > shellPath = /bin/sh; > shellScript = "perl $SRCROOT/scripts/build-llvm.pl > "; > }; > - 26DC6A5813380D4300FF7998 /* Build swig wrapper classes */ > = { > + 26DC6A5813380D4300FF7998 /* Prepare Swig Bindings */ = { > isa = PBXShellScriptBuildPhase; > buildActionMask = 2147483647; > files = ( > ); > inputPaths = ( > ); > - name = "Build swig wrapper classes"; > + name = "Prepare Swig Bindings"; > outputPaths = ( > ); > runOnlyForDeploymentPostprocessing = 0; > - shellPath = /bin/sh; > - shellScript = > "$SRCROOT/scripts/build-swig-wrapper-classes.sh $SRCROOT $TARGET_BUILD_DIR > $CONFIGURATION_BUILD_DIR \"\"\n"; > + shellPath = /bin/bash; > + shellScript = "/usr/bin/python > $SRCROOT/scripts/prepare_bindings.py --framework --src-root $SRCROOT > --target-dir $TARGET_BUILD_DIR --config-build-dir $CONFIGURATION_BUILD_DIR > --swig-executable `which swig`"; > }; > 4959511A1A1ACE9500F6F8FC /* Install Clang compiler headers > */ = { > isa = PBXShellScriptBuildPhase; > > Added: lldb/trunk/scripts/Python/prepare_binding_Python.py > URL: > http://llvm.org/viewvc/llvm-project/lldb/trunk/scripts/Python/prepare_binding_Python.py?rev=253317=auto > > == > --- lldb/trunk/scripts/Python/prepare_binding_Python.py (added) > +++ lldb/trunk/scripts/Python/prepare_binding_Python.py Tue Nov 17 > 01:17:38 2015 > @@ -0,0 +1,435 @@ > +""" > + The LLVM Compiler Infrastructure > + > +This file is distributed under the University of Illinois Open Source > +License. See LICENSE.TXT for details. > + > +Python binding preparation script. > +""" > + > +# Python modules: > +from __future__ import print_function > + > +import logging > +import os > +import re > +import shutil > +import subprocess > +import sys > + > + > +class SwigSettings(object): > +"""Provides a single object to
Re: [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.
Yes, I am planning on separating out the usage of the swig output from the massaging of that into the build. Right now I'm just in phase 1, which is to clean up the existing process without modifying it, and adopting it on Xcode. Humorously, with our recent removal of the searching for swig in the python scripts (which our OS X CI depends on, but since we weren't using the Python ones, I didn't detect), I ended breaking our CI. I'm adding the searching back in. On Tue, Nov 17, 2015 at 8:12 AM, Zachary Turnerwrote: > Is there any way to have the step that runs the swig executable not be > part of this script? Seems like running swig should be a build step, and > running these scripts on the output of swig should be a step that follows > > On Mon, Nov 16, 2015 at 11:20 PM Todd Fiala via lldb-commits < > lldb-commits@lists.llvm.org> wrote: > >> Author: tfiala >> Date: Tue Nov 17 01:17:38 2015 >> New Revision: 253317 >> >> URL: http://llvm.org/viewvc/llvm-project?rev=253317=rev >> Log: >> Add Pythonic language binding wrapper generation script. >> >> This is only used by Xcode at the moment. It replaces the >> buildSwigWrapperClasses.py and related per-script-language >> scripts. It also fixes a couple bugs in those w/r/t Xcode >> usage: >> >> * the presence of the GCC_PREPROCESSOR_DEFINITIONS env var >> should not be short-circuiting generation of the language >> binding; rather, only if LLDB_DISABLE_PYTHON is present >> within that environment variable. >> >> * some logic around what to do when building in "non-Makefile" >> mode. I've switched the handling of that to be on a >> "--framework" flag - if specified, we build an OS X-style >> framework; otherwise, we go with non. >> >> Putting this up now only attached to the Xcode build so >> others can look at it but not be affected by it yet. >> After this, I'll tackle the finalizer, along with trying >> it locally on Linux. >> >> Added: >> lldb/trunk/scripts/Python/prepare_binding_Python.py >> lldb/trunk/scripts/prepare_bindings.py (with props) >> Modified: >> lldb/trunk/lldb.xcodeproj/project.pbxproj >> >> Modified: lldb/trunk/lldb.xcodeproj/project.pbxproj >> URL: >> http://llvm.org/viewvc/llvm-project/lldb/trunk/lldb.xcodeproj/project.pbxproj?rev=253317=253316=253317=diff >> >> == >> --- lldb/trunk/lldb.xcodeproj/project.pbxproj (original) >> +++ lldb/trunk/lldb.xcodeproj/project.pbxproj Tue Nov 17 01:17:38 2015 >> @@ -5984,7 +5984,7 @@ >> isa = PBXNativeTarget; >> buildConfigurationList = 2668020B115FD0EE008E1FE4 >> /* Build configuration list for PBXNativeTarget "LLDB" */; >> buildPhases = ( >> - 26DC6A5813380D4300FF7998 /* Build swig >> wrapper classes */, >> + 26DC6A5813380D4300FF7998 /* Prepare Swig >> Bindings */, >> 26680202115FD0ED008E1FE4 /* Headers */, >> 26680203115FD0ED008E1FE4 /* Resources */, >> 26680204115FD0ED008E1FE4 /* Sources */, >> @@ -6214,19 +6214,19 @@ >> shellPath = /bin/sh; >> shellScript = "perl $SRCROOT/scripts/ >> build-llvm.pl"; >> }; >> - 26DC6A5813380D4300FF7998 /* Build swig wrapper classes */ >> = { >> + 26DC6A5813380D4300FF7998 /* Prepare Swig Bindings */ = { >> isa = PBXShellScriptBuildPhase; >> buildActionMask = 2147483647; >> files = ( >> ); >> inputPaths = ( >> ); >> - name = "Build swig wrapper classes"; >> + name = "Prepare Swig Bindings"; >> outputPaths = ( >> ); >> runOnlyForDeploymentPostprocessing = 0; >> - shellPath = /bin/sh; >> - shellScript = >> "$SRCROOT/scripts/build-swig-wrapper-classes.sh $SRCROOT $TARGET_BUILD_DIR >> $CONFIGURATION_BUILD_DIR \"\"\n"; >> + shellPath = /bin/bash; >> + shellScript = "/usr/bin/python >> $SRCROOT/scripts/prepare_bindings.py --framework --src-root $SRCROOT >> --target-dir $TARGET_BUILD_DIR --config-build-dir $CONFIGURATION_BUILD_DIR >> --swig-executable `which swig`"; >> }; >> 4959511A1A1ACE9500F6F8FC /* Install Clang compiler >> headers */ = { >> isa = PBXShellScriptBuildPhase; >> >> Added: lldb/trunk/scripts/Python/prepare_binding_Python.py >> URL: >> http://llvm.org/viewvc/llvm-project/lldb/trunk/scripts/Python/prepare_binding_Python.py?rev=253317=auto >> >>
Re: [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.
On Tue, Nov 17, 2015 at 8:18 AM Todd Fialawrote: > Breaking out the binding generation into a separate step will also be > important for a couple reasons: > > * (from before) I want to eliminate the requirement for the vast majority > of the builds to have a swig on their system, and > > * (not stated before) we'd like to move away from swig for binding > generation at some point. > > -Todd > Unless these plans (i.e. moving away from swig) are imminent, I would prefer to keep the binding generation step automatic so people can use whatever swig version is installed on their system. I know there are pros and cons to each, but at the end of the day, we need various bugfixes from newer versions of SWIG for the Python 3 stuff, and if someone decides they want bindings for Go, or Rust, or some other languages, they too might need a different minimum SWIG version. We could start checking in multiple sets of generated bindings, but then we start having multiple maintainers, and the checked in bindings become out of sync with each other, and it's more trouble than it's worth. We have buildbots building and testing the various configurations, so if someone uses something that is incompatible we shoudl catch it. And just yesterday I added some options to the @expectedFailure and @skipIf decorators that allow you to skip tests based on the SWIG version and/or python version. Letting people using the SWIG on their system is the easiest way to ensure that everyone's needs are still met while still getting early notification when someone does something incompatible, and it can be fixed. ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.
Moving this back over to the list since I'm sure others have some input here. Also +lldb-dev since it has more visibility than lldb-commits. On Tue, Nov 17, 2015 at 11:25 AM Zachary Turnerwrote: > On Tue, Nov 17, 2015 at 8:18 AM Todd Fiala wrote: > >> Breaking out the binding generation into a separate step will also be >> important for a couple reasons: >> >> * (from before) I want to eliminate the requirement for the vast majority >> of the builds to have a swig on their system, and >> >> * (not stated before) we'd like to move away from swig for binding >> generation at some point. >> >> -Todd >> > > Unless these plans (i.e. moving away from swig) are imminent, I would > prefer to keep the binding generation step automatic so people can use > whatever swig version is installed on their system. I know there are pros > and cons to each, but at the end of the day, we need various bugfixes from > newer versions of SWIG for the Python 3 stuff, and if someone decides they > want bindings for Go, or Rust, or some other languages, they too might need > a different minimum SWIG version. We could start checking in multiple sets > of generated bindings, but then we start having multiple maintainers, and > the checked in bindings become out of sync with each other, and it's more > trouble than it's worth. > > We have buildbots building and testing the various configurations, so if > someone uses something that is incompatible we shoudl catch it. And just > yesterday I added some options to the @expectedFailure and @skipIf > decorators that allow you to skip tests based on the SWIG version and/or > python version. > > Letting people using the SWIG on their system is the easiest way to ensure > that everyone's needs are still met while still getting early notification > when someone does something incompatible, and it can be fixed. > > One possibility (which I mentioned to you offline, but I'll put it here for others to see) is that we make a swig bot which is hosted in the cloud much like our public build bots. We provide a Python script that can be run on your machine, which sends requests over to the swig bot to run swig and send back the results. Availability of the service would be governed by the SLA of Google Compute Engine, viewable here: https://cloud.google.com/compute/sla?hl=en If we do something like this, it would allow us to raise the SWIG version in the upstream, and at that point I can see some benefit in checking the bindings in. Short of that, I still dont' see the value proposition in checking bindings in to the repo. So far all I've heard is that it is so that we don't need swig to be on everyone's machines, but I also don't see the value proposition in that either, given that we already require many other tools on peoples' machines. If it means we can get off of SWIG 1.x in the upstream, I will do the work to make remote swig generation service and get it up and running. From the user's perspective, it's already identical to what you're proposing. No swig executable on your machine (or anyone's machine for that matter), and you run a simple script to generate bindings and check them in whenever you make changes. ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.
Nothing concrete at the moment; however, it could be interesting to look at the clang community and see what could be done for llvm-based language implementations. The angle that I think would be interesting would be if we can generate bindings more effectively based on the in-depth understanding of the language that is afforded by languages built on top of LLVM. This is probably less interesting for Python (particularly since we have a functioning solution) and more interesting for languages built on LLVM or clang. Honestly, though, I haven't spent much time on that. For the time being, I am going to not change the path for everyone on swig, and only use a static binding if swig cannot be found. This will be minimal impact for everyone and doesn't interfere with anyone using a specific version of swig. We can revisit larger questions about who/what/when on static bindings after we gain some experience with enabling them for those who don't have swig. We can review and adjust based on our collective experience. The two files this seems like it will be are the LLDBWrapPython.cpp and the lldb.py file that comes out of python. I hope to have this working in the next day or so. On Tue, Nov 17, 2015 at 7:26 PM, Bruce Mitchenerwrote: > Stepping one step back further in the thread ... > > On Wed, Nov 18, 2015 at 8:35 AM, Zachary Turner via lldb-commits < > lldb-commits@lists.llvm.org> wrote: > >> Moving this back over to the list since I'm sure others have some input >> here. Also +lldb-dev since it has more visibility than lldb-commits. >> >> >> On Tue, Nov 17, 2015 at 11:25 AM Zachary Turner >> wrote: >> >>> On Tue, Nov 17, 2015 at 8:18 AM Todd Fiala wrote: >>> Breaking out the binding generation into a separate step will also be important for a couple reasons: * (from before) I want to eliminate the requirement for the vast majority of the builds to have a swig on their system, and * (not stated before) we'd like to move away from swig for binding generation at some point. >>> > Is there any discussion or thoughts about what the options would be for > moving away from swig? > > - Bruce > > -- -Todd ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.
Stepping one step back further in the thread ... On Wed, Nov 18, 2015 at 8:35 AM, Zachary Turner via lldb-commits < lldb-commits@lists.llvm.org> wrote: > Moving this back over to the list since I'm sure others have some input > here. Also +lldb-dev since it has more visibility than lldb-commits. > > > On Tue, Nov 17, 2015 at 11:25 AM Zachary Turner> wrote: > >> On Tue, Nov 17, 2015 at 8:18 AM Todd Fiala wrote: >> >>> Breaking out the binding generation into a separate step will also be >>> important for a couple reasons: >>> >>> * (from before) I want to eliminate the requirement for the vast >>> majority of the builds to have a swig on their system, and >>> >>> * (not stated before) we'd like to move away from swig for binding >>> generation at some point. >>> >> Is there any discussion or thoughts about what the options would be for moving away from swig? - Bruce ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits