Re: [Scons-dev] Progressing D tool support
Hi Bill, On 09.08.2012 20:23, William Deegan wrote: Russel, Perhaps you we need to make a how to setup a buildslave page and add instructions for installing D to that page? I'll be taking a look at the buildbot today to see if I can recuperate it. I would be very interested in a small Howto, showing me how to get a Buildbot slave going (after installing the Twisted and Buildbot packages). I'm currently in the process of setting up two virtual machines (Windows + Linux) that I could throw in the mix, more or less full time. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Failing buildbot runs...
Hi,
On 30.08.2012 15:45, Gary Oberbrunner wrote:
On Wed, Aug 29, 2012 at 6:14 PM, Dirk Bächletshor...@gmx.de wrote:
Hi,
the new buildslaves are doing good work...and throw a lot of fails. ;)
I'd like to open a bug for fixing a few issues, such that we can hopefully
get all tests to pass again.
Just wanted to check and make sure that nobody else has started to work on
this.
If you do, raise your hand please...else I'll start tomorrow.
This would be great! I fixed a few Windows failures last week, but
there are still some out there. I think (?) all of them are test-code
rather than actual SCons bugs. I know at least one is just a
case-sensitive string compare of drive letters (C: vs. c:), and I did
add a case-insensitive QMtest string matcher a few weeks ago which
would probably fix that one.
I started work on this (issue #2872) and will probably need a few days. ;)
It's not that complicated overall, just a lot to do.
Anyway, we still have some real problems:
- The translation issue, some tests expect a system message in
English. This breaks with a German Windows
installed, or any other language. Should I try to get a full
translation scheme going, using gettext?
I can try, but do we really want to open this can of worms?
- Several MSVS tests expect that VisualStudio is installed and call
cl directly. Should I simply check with a
test.where_is('cl') and then skip it if nothing is found (=only
MinGW installed)? Is this safe enough for now?
Or should we provide a better way to detect the installed toolchain
before the actual tests are carried out?
- Fortran support seems to be broken badly under Windows. Didn't have
time to investigate this further so far,
but the suffixes (*.f77, ...) are not properly recognized.
Hints, pointers and solutions are welcome! :)
Finally, a request regarding the Buildbot setup. Bill, can you please
increase the timeout setting for the Fedora17 slave from 1200s to
something like 5400s? Yeah, I know that's a lot...but I timed the test
KeyboardInterrupt.py on the VirtualMachine without any significant load.
It takes a full 18min to finish, so we need to give it more time.
(These are 2*12 test runs with '-j' increasing from 8 up to 64...)
Thanks a lot in advance.
Best regards,
Dirk
___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Tigris
Hi there, On 15.09.2012 08:04, Russel Winder wrote: On Fri, 2012-09-14 at 09:22 -0400, Gary Oberbrunner wrote: […] Nobody really likes the Tigris tracker. There was some work to start migrating away from it a while ago (Anatoly I think?), but I think we decided to get the code over to bitbucket first, and then worry about the tracker. Maybe that time has come? I don't think the Tigris tracker is that bad. I could live with it for another while. ;) So where to go? The obvious place is the BitBucket issue tracker that can be associated with the SCons mainline. But I had the feeling that people thought it even worse than Tigris for various reasons. On the other hand it would integrate with the pull requests and the repository. Whether we switch to something else, and to which tool, depends on what we want our workflows to be. In the past we had rather strict rules regarding the bug triaging and setting of milestones. And in my view, a user that enters a bug still shouldn't care about whom to assign it to. It's the task of the SCons team to negotiate who cares about the issue, or the single developer if he simply grabs it. I haven't looked at the Bitbucket tracker in a while. But back then it wasn't possible to set milestones, and priorities were grouped into something like easy, middle and hard. Could we live with this? I guess the real question is what is the toolchain for getting all the material out of Tigris and intowherever. I would have guessed that there should be an XML or JSON dump of the Tigris database possible. I wrote some Python classes that are able to extract data from the issues database and write them into an XML format. The other problem is now to get this data into the new tracker on the other end. How would you accomplish this with the Bitbucket tracker? Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] RPM helper functions, where should they go?
Hi there, for fixing the current Buildbot failures I still have to fight down several RPM tests. They check the names of the created RPM files, which differ depending on the used hardware/os combination. I'd like to wrap the original RPM functions for canonicalizing machine/system names and all the supporting stuff to a Python module, but I can't make up my mind about where the file should go. It should be possible to import it from the RPM/packaging stuff and the test suite in QMTest as well. My suggestion would be a new file rpmrc.py in the SCons/Tools dir, alongside the rpm.py tool. Does this make sense or do we have a better place? Best regards, Dirk P.S.: For cases like this it would be cool to have a scons-common package with stuff that can get used by SCons itself, the test framework and perhaps even Parts. It could offer basic things like starting processes, is-this-a-list? and similar stuff, which are partly spread over the whole codebase in different variations... ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Buildslave install instructions...
+++ Quick reminder +++ Hi, I still have a text, describing my steps for setting up the latest buildslaves under Windows and Linux. I just need a Wiki page (linked from the frontpage) where I can put it... ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] README in repository
Hi guys, On 09.10.2012 19:39, William Deegan wrote: Russel, On Oct 9, 2012, at 9:22 AM, Russel Winder rus...@winder.org.uk wrote: I think we have to take a vote on whether to switch the read me file at the top of the repository hierarchy from plain text to a) ReStructured Text, README.rst b) Markdown, README.md or README.markdown I have a slight preference fro Markdown due to the way it handles header markup compared to ReStructured Text. On the other hand Markdown is very Ruby, GitHub, whilst ReStructured Text is very Python. Why the change? So that the read me file look reasonable on the BitBucket page. I can do the markup once a decision is made so there is no need for anything other than a decision. I vote for .rst (mainly because I'm already using it on a number of other projects and it's python'y) +1 from me for reST (*.rst). I use it quite a lot recently...at home as well as at work. Dirk -Bill ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Thanks!
Hi Gary, and a very big Thanks! to you in return, for taking the lead over this bunch of crazy SCons guys, which are all doing very good work (Hey Russel, the new SCons intro page looks awesome!). It's a real pleasure to drive this interesting and challenging project further, together with all you nice people out there... :) Regards, Dirk P.S.: Soon to come are the final fixes for the still failing WinXP Buildslave, and after that the new docs toolchain. So fasten your seat-belts everyone... ;) On 13.10.2012 17:06, Gary Oberbrunner wrote: I just wanted to give a bit shout-out to Dirk Baechle, who has worked heroically over the last few weeks to get the SCons buildbots green again (buildbot.scons.org http://buildbot.scons.org). He not only revamped the test system to allow test fixtures (both files and directories) and make it possible to run tests outside the SCons source tree, but also made a lot of test cleanups to get both Linux and Windows passing all the tests. In other news, check out the new look on our bitbucket site, https://bitbucket.org/scons/scons; Russel Winder just converted the top-level README to much nicer looking RST format so it's much more readable. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Tests on Fedora 17
Russel, one of the Buildslaves is a Fedora 17 system, so I wonder why you still get so many fails. I added some notes about which packages I installed, to http://scons.org/wiki/InstallingBuildbotSlaves . Maybe you can have a look and compare what's needed. If things still don't work out, bug #2872 is still open. If you could attach a full error log there (containing all the stdout and stderr outputs) I'd be able to take a crack at it. Regards, Dirk On 21.10.2012 02:53, William Deegan wrote: Russel, Do you have any development tools installed on that system? -Bill On Oct 20, 2012, at 4:11 AM, Russel Winder rus...@winder.org.uk wrote: On a new Fedora 17 install, I got the following test errors: src/engine/SCons/SConfTests.py [...] test/symlink/VariantDir.py -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Issue 2869 - Versioned shared libraries
Hi Rob, just had a quick look at your changes...thanks a lot for taking care of this issue. On 25.10.2012 06:09, Managan, Rob wrote: I want to get some input on this issue. I created a fork for this at https://bitbucket.org/managan/scons_soname and put Eric Raymond's code into Environment.py that added the methods VersionedSharedLibrary and VersionedSharedLibraryInstall. ... We could just create the sym links for any library whose name includes a 3 digit version number like libtest.2.5.4.so or libtest.dylib.2.5.4. Is that rare enough that it is OK to just do it or what do people think about how to roll this behaviour into the main methods? Another way to say this is: what should the user interface be?? Your suggestions are fine with me, this should be what most users want...and like this, they have it at their fingertips. +1 from me. ;) For keeping everything ultra-flexible, you might want to take the following into consideration: I'd like to see the code for detecting this is the name of a versioned shared lib and spitting out the basename and major/minor numbers encapsulated in a small function. The default behaviour, as defined by you so far, is definitely good enough to go. Although the RPM docs try to remind people that the x.y.z numbering is not a convention, it can be seen as one in current practice, at least from my angle. But for those weird cases someone might come up with in half a year or so, it would be cool if I could override the versioned lib detection with my own code for a single Environment...(e.g. using a no-op function to suppress any further actions, like adding symbolic links, for a versioned lib). This doesn't have to be user-friendly, it should somehow be possible. Just my 2 cents. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Tool order
Hi Russel, yes the order can be important, depending on which variables and Builders are touched by the Tools. If both are working on disjunct sets of settings there shouldn't be a difference though...but it's better to not rely on this. There is no enforcement rule saying that Tools always have to work in an orthogonal way. Best regards, Dirk On 11.11.2012 10:43, Russel Winder wrote: When specifying a tool chain is the order important? For example, should: Environment(tools=['gdc', 'gnulink']) be the same or different from: Environment(tools=['gnulink', 'gdc']) Thanks. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Early Warning: Ceylon is a pain for SCons
Hi Russel, and what exactly would be a problem for SCons when trying to build Ceylon projects? Can we extract requirements for SCons from it? Regards, Dirk On 14.11.2012 16:52, Russel Winder wrote: Remember the problems of Java for SCons. Scala of course is far, far worse. These however are trivial in comparison to Ceylon. I think I have already given up trying to use SCons for working with Ceylon :-( ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Run a single test
Hi Anatoly, specifying the full relative path, like in python runtest.py test/Delete.py , should work. Dirk On 20.12.2012 15:11, anatoly techtonik wrote: Am I right that there is no way to run a single test_ currently in SCons test suite? -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Extensions to the Tool subsystem...
Hello developers, based on my proposed changes to the current tests in src/test there has been some discussion about how a Tool should work. Especially in connection with the LaTeX Tool, questions like: - Do we want to have one latex Tool for all, or separate ones for miktex, texlive...? - Should a Tool try to find a Miktex installation in the current system, or simply search for the latex executable while relying on a correct setup of the PATH variable? showed up. Before simply repeating my pull request of the changes mentioned above, I'd like to reach a consensus about some basic guidelines and design decisions for the Tool subsystem. I reread the PlatformToolConfig page in the Wiki, describing the design of the long-planned IAPAT extension. Its requirements are partly taken into account, but I'd like to keep the GnuBS-like configure context out of this discussion. My idea is to come up with a plan for our Tools, that hopefully makes it easy to add all the configure stuff later. I also want to make a distinction between: - defining the Tool (and its supporting classes) as basic class in the framework and - how we use this Tool module to organize the build/configure/test workflows in our default implementation of SCons. On the PlatformToolConfig page, the following remark can be found: quote (Comment: As I was writing this page, I found myself flipping back and forth as to whether a Tool module configured a tool (that is, a single command) or a toolchain (that is, a series of commands). The current Tool modules actually implement toolchains (e.g., the gcc.py module provides the environment variables for the compiler, the linker, the static archiver, the shared archiver, and the bundle archiver). This isn't good modularization, which suggests that there should be a higher-level module explicitly for toolchains that can invoke one or more tool modules as building blocks. That isn't in this proposal (should it be?), but it's something that should be kept in mind for the future.) /quote Since we try to provide a framework for build systems, our Tool shouldn't care. Both should be possible ways for a user to extend the build engine. With this being said... What does a Tool do? = A Tool has the task of changing an existing Environment. It can alter construction variables like CC or the ENV['PATH'] and is most often used to add Builders or modify existing ones. What can the Tools subsystem do? = (My christmas wishlist ;) for the SCons Tools) PlatformInfo I'd like to have a common module for detecting, storing and retrieving infos about the current host architecture, (platform, processor type, vendor, kernel, OS,...). It should be used throughout the whole source tree, including the test framework and the tests themselves. Toolchains/Tools ## A Toolchain class should get added, as an abstraction for a series (=list) of tools being initialized by a single keyword. It should be possible to: + Check whether we can load/use a toolchain or single tool in our current environment (as given by os.environ['PATH']). + Check whether we can load/use a toolchain or single tool in a special environment. + Get a list of possible toolchains for a Tooltopic (can't think of another name right now). A Tooltopic would be C++, and possible toolchains include mingw and msvs. This selection would probably depend on the PlatformInfo. + Get a default toolchain for a topic, that is actually installed in the current system. + Dynamically add new toolchains. + Dynamically add new tools to a toolchain. + Dynamically change the preferred order in which toolchains are tried. + Add new Toolchains system-wide, that contain parameterized versions of existing tools, e.g. a cxx-embedded for a cross-compiling gcc that requires special options. The single Tool support as it is now should still be available and work as expected. External Tools # For better support of the external tools we could use: + A way to install a Tool in the local SCons distribution and + to deinstall it again. + SCons should be able to display (--version) that there are external tools installed and, on request (--list-external) which Tools exactly (and in which version!). + For this, single tools should support a version string. Questions! === - How do we want our Tools to be organized for the standard SCons implementation? Especially, when we have a Tool like latex that basically looks the same for all distributions (miktex, texlive, ...) in terms of command calls. Do we want a miktex Tool under the covers, that gets automatically selected by the latex Toolchain? Should there be a LatexCommon.py in cases like this? And how do we go about tests for these toolchains? Do we stop using live tests and always provide a fake LaTeX distribution, like for the C compiler and linking
[Scons-dev] Docbook Tool as core module?
Hi there, for the new SCons doc toolchain that I'm currently working on, I'd like to use my Docbook Builder/Tool from: https://bitbucket.org/dirkbaechle/scons_docbook . In its current state it has a simple manual and tests. It basically works...but only if called from the top build directory (the same chdir problem as for the LaTeX Builder, when using references to local images/files in custom stylesheets). Another problem might be that I include a full copy (~5MB) of the Docbook stylesheets, trying to be as convenient for the user as possible. Despite all this, is there a chance to get it into the SCons core? I'd then add a proper *.xml reference file for the Builder and prepare a pull request. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Buildbots
On 20.12.2012 21:27, Managan, Rob wrote: [...] Not sure what is causing the MSVS failures. Changing line 1658 in src/engine/SCons/Tool/msvs.py to: if float(env['MSVS_VERSION']) = 10.0: makes them pass successfully on my side. Not sure whether this is a proper fix though... Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] 2013+ projects
Hi Gary, thanks for compiling this list...it should keep us busy for the next three years or so. ;) See my further comments below. On 27.01.2013 23:26, Gary Oberbrunner wrote: Here's my ideas about what projects are important this year (and into the future -- there's too much here for a year unless we attract some new developers). I put this out as food for thought, and to start discussion -- I'm sure you will have different opinions, things I haven't thought about. Please chime in! SCons projects for 2013 * Toolchain revamp Yes, this is important and gets my full support. About the goals of decoupling tools from the core and allowing tools to take args I had the following ideas: 1.) We could try to establish a contrib or add-on folder in the SCons installation. In a separate repository (scons_contrib) we collect all the more recent (and stable) SCons Tools/Builders and ConfigChecks. A user can download this whole package and install it, which would place everything alongside the normal SCons install as it exists now. By adding the contrib directory to Python's search path, it would be transparent to the user from where exactly he imports a Tool or a ConfCheck. Sounds like additional work, and it probably is. But most of it can be automated I think...we could add the convention that external SCons Tools have to tag their stable revisions with stable-x.y.z and then pick the latest number as candidate for the contrib package. 2.) The default options of Tools, like name and path of an executable (miktex vs. pdftex) or the preferred detection order, could be stored in a config file (I'd suggest YAML format because it allows comments, in contrast to JSON). The path search order for them would be the same like for the Tool modules themselves, such that users can override system settings by adding another config file in their private site_scons/site_tools directory. Just a half-baked idea so far... * o * Node cleanup I'd rather name this one design cleanup. We should take a close look at our current architecture and try to improve it, this would probably also be helpful for documenting how SCons works internally (see also my latest pull request for pylint compliance). When it comes to optimizing for speed, the subst machine should be a top priority. Anyway, +1 from me. * Taskmaster o not most important in 2013 Right, it's not the Taskmaster's fault... * o * Python 3 Not so important for me personally, but I can understand the people who crave to have it. We should probably just dive in, give 2to3 a spin and see how big the problems really are. Other things that might need attention: - Cleanup after the switch to Mercurial. With this I mean saying Goodbye to our old workflows (e.g. bug party) and documenting this properly on the Wiki. - Bug slashing, especially for all the (very) old documentation issues. - Updating the web front page by replacing the comments of people/groups that aren't actually using SCons anymore. So much for today, best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] GSOC this year?
Hi Bill, On 28.03.2013 19:50, William Deegan wrote: All, We need to turn in the proposal by tomorrow. ... Here's some other thoughts I have: * Change SCons code to be runnable on py2.7 and py3.0 * Get SCons to work better/at all on cygwin * Change code to use more modern constructs (slots, generators/iterators) the code changes for #1 and #3 could possibly be wrapped/implemented in the form of fixer scripts, just as Steven and Greg had done for the switch to Python 2.4? One could add a little testing setup (based on our current QMTest framework, and pylint maybe) for CI, ensuring that no examples or tests are broken by the rewrites. This would make the task easily scalable and open-ended in a way. Just as an idea... Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New SCons doc toolchain...
Gary, On 22.04.2013 02:21, Gary Oberbrunner wrote: On Sun, Apr 21, 2013 at 5:57 PM, Dirk Bächle tshor...@gmx.de mailto:tshor...@gmx.de wrote: Hi Gary, On 21.04.2013 23 tel:21.04.2013%2023:38, Gary Oberbrunner wrote: [...] Hi, Dirk! I just cloned this on my Linux box (Ubuntu 11.10 - also tried 12.04), but running scons bootstrap.py gives errors: scons: *** [design.xml] XMLSyntaxError : Specification mandate value for attribute object, line 1, column 31 scons: building terminated because of errors. I was able to reproduce the error on a SuSE 11 box. The lxml support for the Docbook Tool needed a few fixes, I uploaded 2 commits to the new_doc_toolchain branch. Please update and try again... While doing some further testing, I discovered that jw can render PDF from Docbook files, but it doesn't cope with XML input (only SGML) in all cases. That's why I removed it from the list of PDF renderers for the toolchain, so only fop and xep are left now. Make sure that you have fop and one of the XML Python bindings installed (lxml or libxml2)...the latter is to be preferred because it is much faster, but both should work fine now. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New SCons doc toolchain...
On 23.04.2013 18:12, Russel Winder wrote: On Mon, 2013-04-22 at 23:44 +0200, Dirk Bächle wrote: […] Make sure that you have fop and one of the XML Python bindings installed (lxml or libxml2)...the latter is to be preferred because it is much faster, but both should work fine now. Uurrr… isn't lxml a wrapper over libxml2 to provide the ElementTree API (and other things like a validating parser and XPath). Yes, that appears to be true for libxml2 (the C library, that python-lxml depends on)...but not python-libxml2 (the Python bindings), which is the lib I'm actually talking about. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New SCons doc toolchain...
Hi Russel, thanks a lot for all your comments. I won't go into detail about each one of them, but would like to say a few words in general. There still may be some quirks with fonts or layouts and fop is certainly not state of the art for PDF rendering...whatever. To be honest, I don't care that much...if you do, I'll gladly accept your pull requests. I tried to improve the overall procedure for creating the documents, especially for a user that wants to contribute by writing a paragraph or two for the manual or the UserGuide. And I had some success with that, at least it was the best I could give and I, personally, am happy with the result. So there it is now, and can be used by the SCons project. If you guys are not convinced or have better ideas, that's good. Let's talk about them and if they can support all the current features we need and look even prettier, that would be the way to go then. I'm cool with that... What I don't want to happen is, that we do nothing just because the fonts don't look pretty enough yet, or some hyphenations are still wrong. I'd rather go into a possibly wrong direction first and then correct, instead of not moving at all and being stuck with SGML and troff. Best regards, Dirk On 28.04.2013 08:41, Russel Winder wrote: [...] I think human being should never have to read or write XML, not even DocBook/XML. XML-based toolchains are clearly now the norm in publishing for re-purposing, but should this lead to requiring authors to write DocBook/XML? Yes, in our case I think it should. Because it allows us to validate the documents, such that we can put the main work load on the user, not us. ;) [...] Seriously, I do worry that using XML is a sufficient barrier to entry that we will not be evolving the content of the documentation just the form. Well, troff is a barrier as well. Wouldn't it help to move away from that, as a first step? Dirk deserves a prize for taking this on and doing what he has. If we finally get a little bit of movement about this topic, that'll be enough of a prize to me. :) ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New SCons doc toolchain...
On 28.04.2013 20:20, Gary Oberbrunner wrote: On Sun, Apr 28, 2013 at 10:06 AM, Russel Winder rus...@winder.org.uk mailto:rus...@winder.org.uk wrote: Given the current system is XML based, with xml files and in files required, the new system is an improvement and should be accepted. Glad you agree, I feel the same way. This way all the doc uses the same source language and in the same way, with a much more consistent (and verifiable) pipeline. I want to review some of the non-PDF generated stuff to make sure it's all there (as well as the old system did anyway), but Dirk, why don't you start prepping a pull request. Once it's in we can sweat the details (Russel's list is good to start). Bill, what do you think? I am ready to prepare a pull request any time...if we all agree that the current status of my experimental branch is good enough to go, I'll latch on. Would you rather like the pull request to be one single commit, or should I transplant all my single revisions for having a history that makes single changes/decisions more trackable? Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New SCons doc toolchain...
On 29.04.2013 21:51, William Deegan wrote: All, I see the following when running bootstrap.py [...] Also I had to install the following (on ubuntu 10.04) sudo apt-get install python-libxml2 python-libxslt1 python-epydoc fop python2.6-dev Note that without the proper tools installed the build failed complaining about scons.1 missing. Would it be possible to allow bootstrap.py to complete skipping the parts which won't build due to missing tools? Okay, I pushed a new revision. Creating all the doc outputs is happening in the build directory now. I also updated scons_dev_master.py and added a check for lxml/libxml2 to the SConscript for the doc folder. It skips the documentation step if neither of the required packages can be found. Please update and check, if you find the time. Meanwhile, I'll prepare the actual pull request. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New SCons doc toolchain...
On 01.05.2013 21:30, Bill Deegan wrote: Dirk, Should you also add fop or the other alternative to the scons_dev_master? Thanks, Bill Isn't it in the list on your side? I can see it in my revision...and on the bitbucket commit. The xep renderer is a commercial one, but there is a free trial download available. So I can't add this one to the packages, sorry. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New SCons doc toolchain...
Hi Gour, On 09.05.2013 10:33, Gour wrote: On Tue, 26 Mar 2013 00:50:07 +0100 Dirk Bächle tshor...@gmx.de wrote: [...] I settled to use SCons for my PyQt project and was reading the User Manual yesterday and e.g. found that the email addresses listed are wrong (e.g. us...@scons.tigris.org) which led me to check how are docs handled in order to know how to contribute some 'patches'. So, I was a bit surprised that SCons uses Docbook toolchain considering that some time ago project switched from SVN to (more pythonic) Mercurial, I wonder why not using (more pythonic) reST/Sphinx which, imho, requires less admin work and it provides lower-hanging fruit for potential contributors who are probably more familiar with reST/Sphinx than Docbook. Moreover, the current user manual is ~350p and not so complicated to require the above-mentioned features. I hope that you will find this email only as constructive feedback meant to improve SCons project by simplifying doc toolchain in order to spend valuable time of not-to-big developer's team on more important issues. thanks a lot for trying out the new doc toolchain. It's still very fresh and there's probably a lot of room for improvement...including a rewrite to another input format (like asciidoc, reST, whatever). So, we're very interested in hearing about first impressions from avid users like you. If it were only about creating plain text with some graphics, you're right. This could easily be handled by a lot of other toolchains. However, I'd like you to have a short look over the discussion at http://www.scons.org/wiki/DeveloperGuide/Documentation and http://www.scons.org/wiki/DeveloperGuide/Documentation/Discussion , respectively. There is also a description of the current implementation in the file doc/overview.rst. Together they might shed some light on why we (errm, I) finally chose Docbook. If you have further thoughts or questions, please don't hesitate to ask. Thanks again for your feedback. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New SCons doc toolchain...
Hello Gour, On 09.05.2013 22:53, Gour wrote: On Thu, 09 May 2013 12:14:29 +0200 Dirk Bächle tshor...@gmx.de wrote: [...] If it's good-enough for Python project docs itself, I believe it should be for SCons as well. that's okay...but to make me believe this as well, you (or someone else) has to deliver actual results. ;) As I stated before in this thread, as long as the same functionality is kept regarding the automatic creation of examples (and the generated list of tools and builders, while delivering the same output formats we have now), a toolchain based on a Markdown processor gets my full support. Knowledge of Docbook authoring is not very common in general and certainly not within Python community, so I'm afraid that contributing docs to SCons project would remain niche for a few people only. Moreover, the current output of the manual shows that the complexity of the markup used to write it is not in proportion the quality of output and tweaking/theming is still, imho, much easier to do with Sphinx. That's more because the stylesheets have been neglected in the past (obviously nobody wanted to fiddle with DSSSL) and because parts of the document processing relied on home-brewed SGML parsing without proper support for XML. Like this, not all valid XML/Docbook constructs would have worked, which held back people a little to use the full power of the Docbook stylesheets. This can (and hopefully will) change now... Finally, in regard to the argument of converting Docbook to something else, there is wonderful tool called Pandoc (http://johnmacfarlane.net/pandoc/) which is capable of reading Docbook markup and select it to several other markup formats. Then just try to convert a few SCons documents with it, and send us some selected pages (not the whole document!) of output samples. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Using SCons via bootstrap.py
Hi Russel, On 17.05.2013 19:13, Russel Winder wrote: Is anyone else finding that the bootstrap.py script no longer works? It worked for me before I went away for a short break and now after updating Debian Unstable, it is failing. I get the same behaviour with Python 2.6, 2.7 and PyPy 2.0. Traceback (most recent call last): File /home/users/russel/Repositories/Mercurial/Masters/SCons_D_Tooling/bootstrap.py, line 233, in module main() File /home/users/russel/Repositories/Mercurial/Masters/SCons_D_Tooling/bootstrap.py, line 204, in main for x in parseManifestLines(src_engine, open(MANIFEST_in).readlines())] File /home/users/russel/Repositories/Mercurial/Masters/SCons_D_Tooling/bootstrap.py, line 85, in parseManifestLines os.chdir(basedir) OSError: [Errno 2] No such file or directory: 'src/engine' the bootstrap.py works fine for me, I guess because I have a src/engine folder. Where did yours go? ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] [Fwd: [Scons-users] [patch] SCons fails to run in standalone mode]
On 08.07.2013 18:32, Managan, Rob wrote: [...] For a bash shell then the equal sign is correct but you need to change setenv to export. That is what I was trying to convey in the first email. Ahhh, okay. I got it now. Maybe we should take some action about this. Instead of all this long writing about how a proper shell script should look like, we could simply provide one for Linux and Windows each? I can publish the first samples in a pull request...my Linux version would be based on bash though (let the wars begin ;) ). Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] [Fwd: [Scons-users] [patch] SCons fails to run in standalone mode]
On 09.07.2013 00:53, Russel Winder wrote: On Mon, 2013-07-08 at 20:18 +0200, Dirk Bächle wrote: On 08.07.2013 18:32, Managan, Rob wrote: [...] For a bash shell then the equal sign is correct but you need to change setenv to export. That is what I was trying to convey in the first email. Ahhh, okay. I got it now. Maybe we should take some action about this. Instead of all this long writing about how a proper shell script should look like, we could simply provide one for Linux and Windows each? I can publish the first samples in a pull request...my Linux version would be based on bash though (let the wars begin ;) ). So what is/was the role of bootstrap.py which had been touted as the way of using a SCons checkout live? In my view, bootstrap.py is more for creating the build packages. It can also copy together a local working copy (the bootstrap folder), which is fine for most cases when you simply want to start SCons without fully installing it. But I'm doing some speed measurements and profiling with various revisions at the moment, so I don't want to get the overhead of checking for existing files and copying them, in the way of my results. That's why I prefer a simple script in this case... Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Unicode support in print_tree / render_tree
Hi Remko, On 13.07.2013 09:32, Remko Tronçon wrote: Hi, I filed a bug report for --tree=... crashing if the dependency tree contains unicode characters ( http://scons.tigris.org/issues/show_bug.cgi?id=2910 ) I fixed the bug locally by calling repr() on every element in the tree (which is also done by the code that dumps actions) and thanks for looking a bit into this Unicode issue. We had a discussion about these kind of problems a while ago, and reached the consensus to not attack these directly, if I recall correctly. Instead we try to push the general conversion of the sources to Python3, and get Unicode support in all the places along the way (not for free, that's sure ;) ). Have you tried to actually compile/build stuff that uses Unicode chars in filenames? Or are you only analyzing the dependencies? Can you please attach your patch to the Tigris tracker issue or point us to the repository, if you cloned the SCons sources? Then we can have a closer look together and better decide what to do next. Thanks a lot in advance, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] [Fwd: [Scons-users] [patch] SCons fails to run in standalone mode]
On 13.07.2013 16:53, Russel Winder wrote: On Tue, 2013-07-09 at 09:17 +0200, Dirk Bächle wrote: […] In my view, bootstrap.py is more for creating the build packages. It can also copy together a local working copy (the bootstrap folder), which is fine for most cases when you simply want to start SCons without fully installing it. […] I think we perhaps need to clarify the role of bootstrap.py then as I have always known it as the entry into using the current HEAD as SCons (which is what H S Teoh is doing as well). Using boostrap.py leads to having all the pyc files not in the checked out repository which is a Good Thing™. This seems to work well for you, so stick to it. Trying to use a shell script instead leads to pyc files being in the checked out source tree, not good at all. This appears to work well for me (it has other advantages), so I'll stick to it. ;) The role of bootstrap.py is probably defined quite well, so I don't see a need to discuss it at the moment. All I wanted to offer is the simple scripts that I use to start SCons directly, such that we could replace the misleading instructions in the REAME.rst by a pointer to the files. Fixing the text, as Rob suggested, would be another option of course. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Unicode support in print_tree / render_tree
Remko,
On 14.07.2013 13:46, Remko Tronçon wrote:
Hi Dirk,
On 14 July 2013 13:16, Dirk Bächle tshor...@gmx.de wrote:
So it would help us a lot if you could create a simple testcase for this,
which breaks with the current code but should run successfully in the end.
Here's a failing case:
env = Environment()
env.Tool(textfile)
env.Textfile(Foo, unichr(0xe7).encode('utf-8'))
'scons' runs fine, 'scons --tree=all' throws an exception.
cheers,
Remko
thanks a lot for the example, I'll attach it to issue #2910...
Best regards,
Dirk
___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] [Fwd: [Scons-users] [patch] SCons fails to run in standalone mode]
Hi Gary, On 15.07.2013 02:57, Gary Oberbrunner wrote: I'm still here -- just working 24/7 at my day job -- soon Ill be back to SConsing I think it's important that bootstrap.py be useful to run scons directly out of its repo checkout. I use it like that. Haven't been following the discussion closely but breaking that would not make me happy. maybe you (or Bill?) could find the time and accept/merge only the pending pull request #79, which deals with fixing bootstrap.py. It's a rather simple one This would calm everybody down a bit, I guess. ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] [Fwd: [Scons-users] [patch] SCons fails to run in standalone mode]
Hi there, On 20.07.2013 10:24, Russel Winder wrote: On Fri, 2013-07-19 at 19:56 -0400, Gary Oberbrunner wrote: […] OK, sorry that took so long. Merged. Hope that helps. Thanks for picking up on this. Sadly, I get: | python /home/Checkouts/Mercurial/SCons/bootstrap.py /usr/bin/python /home/Checkouts/Mercurial/SCons/bootstrap/src/script/scons.py SCons import failed. Trying to run from source directory Traceback (most recent call last): File /home/Checkouts/Mercurial/SCons/bootstrap/src/script/scons.py, line 188, in module import SCons.Script File /home/Checkouts/Mercurial/SCons/bootstrap/src/engine/SCons/__init__.py, line 43, in module import SCons.compat ImportError: No module named compat which seems to indicate the sys.path is still not entirely correct :-( I pushed a new pull request (#83) a few minutes ago. Looks like my bootstrap folder wasn't as clean as it should've been, when I tested the previous version. Sorry... Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Pull request to automate web-site deployment
Hi there, On 23.07.2013 21:47, Gary Oberbrunner wrote: [...] Anyone else know Fabric and care to chime in? I haven't used Fabric either, yet. As long as its usage is optional, I don't mind. Just my 2 cents. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Testing non-core tools
On 26.07.2013 19:30, Russel Winder wrote: Problem solved: TestSCons.TestSCons() causes the chdir() so by getting the getcwd() before that I am alright. Is this gotcha documented anywhere? It is mentioned at http://www.scons.org/wiki/DeveloperGuide/TestingMethodology for the Hello World SCons Test Script, but there is no humongous DANGER! sign next to it. ;) Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Testing non-core tools
...and please don't use the scons_test_framework repo for further development (just got the notice that you cloned it). The current code is in the normal SCons repo (runtest.py and QMTest). Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Testing non-core tools
On 26.07.2013 20:00, Russel Winder wrote: On Fri, 2013-07-26 at 19:55 +0200, Dirk Bächle wrote: […] Keeping all test scripts under a top-level folder test, and adding sconstest.skip files where needed, should give you a start pretty quickly though. So no more sconstest-XXX.py files? The runtest.py now detects any *.py file as test script, that's why the sconstest.skip (or even .exclude_tests) exists and escapes this behaviour where needed. You can read a bit about this in the Finding tests section of QMTest/test-framework.rst. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Testing non-core tools
On 26.07.2013 20:27, Russel Winder wrote: On Fri, 2013-07-26 at 20:11 +0200, Dirk Bächle wrote: […] [...] I have a test directory in the non-core tool package. If I run path.to.scons.installtion/runtest.py -a I get: If you call the runtest.py for external tests, you have to specify the -e option as well. Does it work then? Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Testing non-core tools
On 27.07.2013 15:51, Russel Winder wrote: [...] But the need to specify the -e to test non-core tool packages is already a marker that they are being handled specially. Also from the above tool sources are special structures and deserve special support. Given tests must run under the test framework and the framework knows it is doing something special, special support seems eminently allowable. There is a conventional layout already in place so I think if it is at all possible the -e should prepend '.' to the tool path. Okay, I'm convinced now and will try to prepare a pull request. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Wiki hacked? Again?
Hi there, is it possible that our SCons Wiki got hacked again? I just wanted to add a link to the ToolchainRevamp page and noticed that the Roadmap shows some different content: http://www.scons.org/wiki/Roadmap Other examples: http://www.scons.org/wiki/AboutSCons http://www.scons.org/wiki/BasicConcepts Can anyone check this further? Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] [PATCH] scons soname on OpenBSD
Hi Stefan, On 09.09.2013 19:45, Stefan Sperling wrote: On Sat, Sep 07, 2013 at 04:24:41PM -0400, Gary Oberbrunner wrote: [...] The ideal way to contribute to SCons is to fork the mercurial repo at https://bitbucket.org/scons/scons, make your change, then submit a pull request. Patches sent to the mailing list can get lost. If possible I'd like to ask someone on this list to commit the patch for me. I don't have an account on bitbucket. I used the time it would take me to read through atlassian's terms of services to write a unit test instead :) thanks a lot for the patch and the test case. I'll file a bug in the Tigris tracker and then submit a pull request for your fix. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] SCons speedup and profiling results...
Hi there, a few minutes ago I added a new page to our Wiki. It's called http://scons.org/wiki/WhySconsIsNotSlow and shows a few results of the speedup and profiling experiments that I did recently. As mentioned in the Repositories section, you can also download the full set of results and the little test framework that I used from the Bitbucket repos: http://www.bitbucket.org/dirkbaechle/scons_testresults http://www.bitbucket.org/dirkbaechle/scons_testsuite So just dive in if you're interested, and let me hear what you think... By the way, I'm sorry for certain parts of the text being quite terse. The language could also be improved a little, I guess...but right now I'm lacking the time and patience for it (I just wanted this project off my desk ;) ). So if any native speakers out there should feel the itch to correct and extend my writings, just go ahead please. I'd highly appreciate it...and it's a Wiki after all. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] mercurial vs. git
On 29.09.2013 20:07, Gary Oberbrunner wrote: Now that we've all been living with hg for a while, what are people's opinions on hg vs. git for SCons? I'll admit I'm much deeper into git these days and I think overall it's a better system. But I'm interested in what you all think. We could switch pretty easily if there was enough interest, or we could just stay with hg and get on with life. :-) At the moment I see no compelling reason to change our VCS again. Our current workflows are fine and documented (a bit). Since SCons switched to hg, I'm now using it for all my private work as well and am quite happy with it. So, my bias leans more towards hg. -1 from me, but I wouldn't run away from the project if I had to use git in the future. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons speedup and profiling results...
On 26.09.2013 02:08, Gary Oberbrunner wrote: [...] I think this is excellent work! Solid analysis. I know there's been some thought given to caching subst() before; it's trickier than one might think but in many cases it should work, and it definitely speeds things up. I'm also impressed by a 30% memory reduction -- interested to hear how that comes out. I continued my work on reducing the overall memory consumption in SCons. By combining my old branch (where I switched the core to using slots) with some additional patches, I am now able to save up to 50% memory...depending on the project. Please find some results attached (a comparison between the current default tip and my experimental branch), the code can be cloned from: hg clone http://bitbucket.org/dirkbaechle/scons_experimental -r reduced_memory_updated I also achieved up to 20% speed improvements on updates, by a first version of a caching for the env.subst() method. Best regards, Dirk Title: Comparing default to lowmem wonderbuild Times Runrun [s]update [s]update_implicit [s] Previous1172.325.619.4 Current1131.622.015.5 Factor0.970.860.80 Memory Runrun [MByte]update [MByte] Previous451.4424.2 Current251.5197.8 Factor0.560.47 sconsbld Times Runrun [s]update [s]update_implicit [s] Previous440.335.026.1 Current343.828.719.8 Factor0.780.820.76 Memory Runrun [MByte]update [MByte] Previous538.6554.4 Current231.6238.7 Factor0.430.43 questfperf Times Runrun [s]update [s]update_implicit [s] Previous1022.424.216.9 Current984.820.412.9 Factor0.960.840.77 Memory Runrun [MByte]update [MByte] Previous378.9391.1 Current210.9196.3 Factor0.560.50 mapnik Times Runrun [s]update [s]update_implicit [s] Previous867.212.54.7 Current867.59.44.1 Factor1.000.750.87 Memory Runrun [MByte]update [MByte] Previous151.5144.1 Current110.9110.7 Factor0.730.77 ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Documentation in EPUB-Format?
Hi devs, since we're now using DocBook as source format for all our documentation, it would technically be relatively easy to publish things like the MAN page or the UserGuide in EPUB format as well. The latest versions of pandoc ( 1.12.x) offer a DocBook reader and do a, more or less, good job in rendering the single markups. Find a shortened version of the UserGuide attached, as a first example... The question now is: Should we go this route? Then I'd try to integrate an ebook target into the build and bootstrapping process, such that EPUBs get created automatically if pandoc can be found on the current system. Note, how this would add a dependency to pandoc for all the ReleaseManagers...and this currently means installing the full Haskell platform package with approx 400MB. ;) Your comments? Best regards, Dirk scons.epub Description: application/epub ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Working branches
Hi Russel, On 04.10.2013 19:23, Russel Winder wrote: Now we have default and python3-port as working branches, we need a workflow that ensures they are kept in sync. If python3-port is left behind, then all the work to date will have been for nought. hmmm, we probably should discuss (and then decide) how we want development to look like, once we have a working python3 branch. Will we continue to support both major versions 2.x and 3.x? If yes, in which direction (2-3 or 3-2) do we merge preferably, meaning: where do we do most of our development? Or does the 2.x stuff get abandoned and we care about Python 3.x only? I also need advice on how to handle feature clones: I have the SCons_D_Tooling clone and have no idea how to manage it now with the two working branches. I have my own repository scons_experimental for things like this. I pull from the SCons main repo whenever I feel the need to update, but never push back. So, I can merge freely and can also use named branches as much as I like...one for every feature, basically. For features to be published, I create a patch and apply it to my clone of the main repo...then issue a pull request from there. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Documentation in EPUB-Format?
Hi Rob, On 05.10.2013 01:42, Managan, Rob wrote: I like the idea since I like ebooks. However, I will be honest and say that for code development I would not be too likely to use it since on my desktop machines I don't have a great epub viewers. it's more like for users starting with SCons and wanting to read more about it...wherever they go. ;) So I guess I would say that if there was no extra effort or file space required go ahead and add it but if the extra effort is noticeable don't bother. Okay, I'll give it a try. Should the EPUBs be just a side-effect of the doc build for now, or should they get archived in the different distribution packages too? Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Documentation in EPUB-Format?
On 06.10.2013 01:24, Bill Deegan wrote: Dirk, How big are the ePub files? vs pdf.. For the full UserGuide, I currently have: PDF = 2394kB vs. EPUB = 219kB , but you have to take into account that the additional graphics for the style and the titlepage eat up a lot of space for the PDF. I haven't tried yet, but I guess additional images would bloat up both formats in about the same way. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Documentation in EPUB-format
Andrew, On 06.10.2013 21:07, Dirk Bächle wrote: [...] The DocBook Tool is not (yet) part of the core sources. By keeping compatibility to older SCons and Python versions, we don't force people to upgrade if they want to use the Tool. So this still makes sense, I think. sorry I have to correct myself...it *is* a core Tool by now. Anyway, please publish your changes on your scons_docbook fork and send a short note. Then I'll have a look and we can decide how to take it from there. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Documentation in EPUB-format
Andrew, On 13.10.2013 20:50, Andrew Featherstone wrote: [...] Dirk: I've pushed what I've got so far to https://bitbucket.org/ajf58/scons_docbook/branch/epub . I've run a basic Docbook through it and generated an EPUB file that passes the validation test here http://validator.idpf.org/ . In it's current form there's a bug in that the files added to the OEBPS directory aren't re-added when the source files are changed. Is it possible for the action functions passed to a Command builder to modify target and source lists? this is more the task of an Emitter. If you want to do things properly, you'll have to define an additional set of builders for xsltproc, lxml and libxml2. You can then add your specialized Emitter to the constructors of the single Builders and replace the call of __select_builder() with something like: __builder = __select_builder(__lxml_builder_epub, __libxml2_builder_epub, __xsltproc_builder_epub) . Bill: I can raise an issue there if you think that's sensible. A quick glance at http://scons.tigris.org/ shows a large number of open tickets. Is this because issues are tracked elsewhere? https://bitbucket.org/scons/scons itself seems to be active, and indeed shows some work using Ghostscript to create EPUB files. We currently have the situation that our repository is on bitbucket, while the bug database is still managed by Tigris...as long as we don't have decided where we want to migrate to. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Calling PseudoBuilder from an emitter function from a Builder with OverrideEnvironment looses overrides
Hi Andreas, On 21.10.2013 11:54, andreas.a...@de.transport.bombardier.com wrote: Hi all, first of all, thanks for providing such a useful tool as scons. I've found a small glitch in Scons 2.1.0 (as part of Ubuntu 12.04 LTS). I've got an emitter function installed for one of my builders. This builder is called with keyword arguments so that an OverrideEnviroment is used to execute it. When the emitter function is called the OverrideEnvironment is presented to it in the env parameter. But if I call a PseudoBuilder (i.e. a MethodWrapper instance) from there, the PseudoBuilder itsself is called with the original environment and the OverrideEnvironment is no longer available to it. I guess, a possible fix would be to forward the keyword arguments used to call the builder to the emitter function so that these keyword arguments could be used to call the PseudoBuilder. as a first thought on this: I wouldn't expect this to work at all, because it shouldn't be necessary to call any kind of Builder from an Emitter. Can you possibly tell us a bit more about where this setup comes from, or why you need to do it like this? Thanks a lot in advance and best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Schedule for redesign?
Hi, over the last few days I had another look at SCons' speed and memory problems. As posted in an earlier email, I am able to reduce the maximum amount of memory used during runtime (both, clean and update builds) by up to 50% in large C/CPP projects. This is reached by freeing infos in the Node class itself, when they aren't needed anymore after a target is built/visited. Having identified a set of Node attributes like this, somewhat lends itself to refactoring them out into its own class (TargetInfo?). There has always been some rumor about a redesign of the Node classes and the Taskmaster, which is what this is pretty much about...although not in a very broad range. What I'd like to do (and actually already started in a private branch) is to refactor a few classes and imports, like follows: - move the GetOption/SetOption/FakeOptionParser stuff out of SCons.Script.Main into SCons.Options (have this ready) - move the Task classes (Clean/AlwaysBuild/...) out of the SCons.Taskmaster into their own module SCons.Tasks (ready, too) - move the Node attributes mentioned above into their own class SCons.TargetInfo (tbd) In effect, funtionality gets shifted to where it belongs conceptually, and not where it is needed. This might make access to certain parts a little more complicated, in the sense that one more often needs to delegate things or distribute information like option flags to several places. But it also makes the design overall easier to grasp and explain...at least in my opinion. Now my question (see the title): When would this fit into our current development schedule? Should I just go ahead and propose one (or several) pull requests, or wait until the Python 2to3 conversion is over and things have settled. Do you think we need a special deprecation cycle for this, or maybe even should switch the version number to 3.x then? Your comments are welcome... Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Schedule for redesign?
Hi Jason, and thanks for chiming in. On 28.10.2013 17:07, Kenny, Jason L wrote: I hope I would not be a wrench in engine :-) Honestly The issue that this is work and I have to do this at home is the main reason I have not pushed anything at this point. ( And having twin 3 years-old means I don't have the go power at the end of the day...) I would agree with what is being stated. I would go a bit farther however. I understand that you have done some work on redesigning/refactoring parts of SCons, and you certainly don't want to lose it for nothing. So, I'd like to make the following suggestion: I'll factor out some attributes of the Node class into their own module and commit these changes (together with the other rewritings I mentioned) to a named branch in my personal experimental SCons fork. Then you can have a look at it and check whether you'd be able to base your further work on that. If not, you can tell me what should be changed such that you can continue properly with your work. If I get your okay, the changes find their way into the SCons core and you can take it from there whenever you find the time. How does that sound? I would redesign the Node API. But also redo the executioner logic. Ideally we want I have learned is that the task logic needs to be updated. Greg Noel was on the correct path with TNG in that we want to execute builders not nodes. We also need the API ( internal) to decouple the relationships with the tasks and the nodes. I would also look at fixing up the subst engine. I found that this is another area of memory waste at the moment ( and not in the good way). This might be more of an issue with Parts as I currently use the subst engine to share data between different parts/ components. Until I do a new part file format and or a continuous loader I have no way to pass data correctly. During my investigations I tried to speedup subst() by caching some pre-expanded strings, but it didn't work out. My current impression is that you either have to compile it (including optimizations by hand, simply running Cython over it is not enough!) or reduce functionality (like cutting down the support for callables) to get it faster. But don't let me stop you from trying... ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] point release time?
Bill, On 17.12.2013 22:06, Bill Deegan wrote: Dirk, If the memory patch uses __slots, then won't that likely break some user logic? If so then we should push out 2.3.1 without it with a notice in the release notes indicating what such change may break? this patch doesn't use slots at all, it just tries to release as much infos as possible for already built targets...without breaking any existing tests. This means that we don't get the full amount of saved memory that would be possible, but it's a start. Also looks like we have more than a handful of tests broken. So my suggestion would be to fix the broken tests, then push 2.3.1. Then address the other issues? My pull request #96 should fix the Environment bug...which other issues would you like to address? Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
Hi Roberto, On 08.01.2014 22:52, Gary Oberbrunner wrote: On Wed, Jan 8, 2014 at 4:49 PM, roberto de vecchi roberto.devec...@vi-grade.com mailto:roberto.devec...@vi-grade.com wrote: Gary, I think that the problem I'm seeing is generated by the modification in FS.py around line 3052: ... removing the reset of self.cwd things look much better and I can build correctly with interactive mode. Just for your info we are running exclusively with variantdir enabled and no source duplication. If you can confirm the small change makes sense, I'll let some of our developer test more in detail this configuration. It makes sense to me, but Dirk will have to have a say. thanks a lot for testing the new version. Please continue with the reset of self.cwd commented out (or remove the whole line). If you find further issues, just let us know immediately. @Gary: Let's wait another day and see if Roberto finds more errors, then I'd prepare a pull request for removing the offending lines, okay? Trying to reproduce the failing build on our side and then working around it, is probably not a good idea so short before the release. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
On 09.01.2014 14:55, Gary Oberbrunner wrote: I'm also getting some spurious rebuilds with 2.3.1 compared with 2.3.0. Will dig into it. Dirk, I'm not sure if it's your patch or something else that changed. Gary, I created a pull request, switching off the memory savings for the Interactive mode completely. Just to be extra careful... With this, we should get rid of the observed problems. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
On 10.01.2014 18:46, Kenny, Jason L wrote: I have the same issue with the build at my job. I thought it might have been bug in Parts passing data around, a badly define build files that dependson stuff differently if something exists on disk or not ( ie something that is built). However I pretty sure there is a bug in SCons. I leaning towards bugs in way signatures are made.. honestly this is where I go bad to needing to refactor scons. The plus side it seems to happen for me only on one or two cases out of 90k different outputs for the build. Parts tends to get around this when it skips loading the Parts file that defines these nodes as for some reason the corruption seems to happen when the build file happens, the existing stored state seems to be correct. Jason Do you have Builders in your setup that create included files (like C headers) during the run? I'm asking because the current source still has a small problem with tools like SWIG. On the first clean run, a header doesn't exist. So SCons finds the SWIG executable first, and then later (during scanning) the header after it got built. On the second run (update), the header exists already...like this, the MD5 sums of the single implicit dependencies don't change, but their order in the internal list of children. That's what can cause unneeded rebuilds...it should be okay on the third run though. You might want to try and run the build with the environment variable IMPLICIT_COMMAND_DEPENDENCIES set to False env['IMPLICIT_COMMAND_DEPENDENCIES'] = False . Do the errors go away then, or get less frequently? @Gary: Feel free to revert the mem patch anytime...we could also make it an experimental feature, and have it activated by a special command-line option only (for now or the next version). Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
On 13.01.2014 20:18, Gary Oberbrunner wrote:
Dirk, and others: I tracked down my spurious rebuild to the addition
of caching changed-status in File.changed() in Node/FS.py. If I
remove that caching code I don't get the rebuilds:
diff --git a/src/engine/SCons/Node/FS.py b/src/engine/SCons/Node/FS.py
--- a/src/engine/SCons/Node/FS.py
+++ b/src/engine/SCons/Node/FS.py
@@ -3043,13 +3043,15 @@
but we allow the return value to get cached after the reference
to the Executor got released in release_target_info().
-if node is None:
+allow_caching = False
+if node is None and allow_caching: # try this
try:
return self._memo['changed']
except KeyError:
pass
has_changed = SCons.Node.Node.changed(self, node)
+if allow_caching:
self._memo['changed'] = has_changed
return has_changed
I also had to add this code to fix an exception when the file doesn't
have an executor.
diff --git a/src/engine/SCons/Node/__init__.py
b/src/engine/SCons/Node/__init__.py
--- a/src/engine/SCons/Node/__init__.py
+++ b/src/engine/SCons/Node/__init__.py
@@ -1090,7 +1090,10 @@
if t: Trace(': %s changed' % child)
result = True
+if self.get_executor():
contents = self.get_executor().get_contents()
+else:
+contents = None
if self.has_builder():
import SCons.Util
newsig = SCons.Util.MD5signature(contents)
Dirk, what do you think? I'll play with this version for a while.
Okay, these both places are related by the call of
SCons.Node.Node.changed() from SCons.Node.FS.File.changed() (one calls
the other). What's supposed to happen is: in File.release_target_info()
the executor gets released. Before this, the changed() method is called,
such that it caches its value in self._memo['changed'].
If this doesn't work as expected, this would mean the File.changed()
gets called much later sometimes, after the executor got released *and*
the self._memo was reset. Can you try and get a stacktrace for when that
happens?
It's crucial to be able to release the executor early...if we can't do
it, there won't be much of a memory improvement.
Regards,
Dirk
___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Failing tests
On 21.01.2014 19:05, Russel Winder wrote: I just ran all the test on Debian Unstable and got 70 no results, which seems fine, but 9 test fails. 2 of these I understand, the others I have no clue about: test/Docbook/basedir/htmlchunked/htmlchunked_cmd.py test/Docbook/basedir/htmlhelp/htmlhelp_cmd.py test/Docbook/basic/epub/epub.py test/Docbook/basic/epub/epub_cmd.py test/Fortran/F95FLAGS.py test/Fortran/SHF95FLAGS.py test/YACC/live.py Which revision are we talking about, current tip of default? Are these errors related to your question about TestSCons.where_is()? ad Docbook: What kind of support for XSLT processing do you have installed on this machine, xsltproc (command-line) or one of the libxml2/lxml Python bindings? Can you post the error messages (STDOUT/STDERR)? Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Vague recollection of reporting a problem
On 21.01.2014 18:39, Russel Winder wrote: It is clearly the case that TestSCons.TestSCons().where_is(toolSequence) uses the users current PATH to search. However when the tool is actually used, the stripped down PATH is used. This means there appears to be no way of checking whether a test will fail due to a failure to find the executable of a tool. Or am I missing something? The method where_is() accepts a path as second argument, that is used for searching if specified. So you could call TestSCons.where_is() with the stripped down PATH variable as well. This should then give identical results...haven't tried though. ;) Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
On 28.01.2014 23:57, Gary Oberbrunner wrote: More results, no fix yet. The generated file all-defuns.c I mentioned before is definitely part of the problem. I back-ported the tracing code I wrote to just before Dirk's memory optimization. In that version, near the beginning of the build phase something calls node.is_up_to_date() on all-defuns.c and it says it has changed, due to depending on a dir mkl which is in no_state (i.e. hasn't been evaluated yet). In Dirk's version, this result gets cached, which seems sensible. But in the old version, when the taskmaster considers that all-defuns.c node, and calls is_up_to_date(), now it correctly returns True, because by now the mkl dir has been evaluated and it's in up-to-date state. So it's definitely the caching that's causing the trouble, but I don't understand why is_up_to_date should return different values at different times -- especially go from false to true before anything's been built. Perhaps there's another bug in the code which Dirk's patch exposes? I would think is_up_to_date() should never assume that being unprocessed (i.e. no_state) means not up-to-date; I'd think it should have scanned the no_state node before its dependents. Anyone have any clues? When I implemented my changes, I saw that in the old version the changed() (or connected methods) could be actually called after a file was built. And since there was nothing cached, this could lead to creating additional build infos in the same step. So there was a danger of having a build info hash signature in the .sconsign, that would not actually describe how the target was built (more like, how would it be built next time). Maybe your build DAG relied on this fact so far? By the way, are you building with -j or is it failing single-core too? Do you think it's possible to extract an isolated test case from this, now that we know more about what seems to happen? Regarding the is_up_to_date() method, we're pretty much on the same page. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
On 02.02.2014 23:13, Gary Oberbrunner wrote: HA -- got a small repro testcase! [...] Dirk, I guess the ball's in your court! :-) Of course I want to keep helping to solve it but at least you and interested others (hi Roberto!) can give it a try. Thanks a lot for the testcase. I'll have a look at it...no problem. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Fwd: [GSoC Mentors Announce] Now Accepting Applications for Mentoring Organizations for GSoC 2014
On 03.02.2014 20:05, Gary Oberbrunner wrote: Hi folks; if we want to get a GSoC project this year, now's the time to think about it. Top of my priority list for a GSoC student would be someone to convert everything to python3, finishing what we've started already. Other ideas? Looking through the ideas at http://www.scons.org/wiki/GSoC2013Ideas , I'd think that improving the packaging and distributing of SCons would be a worthwhile project too. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Godot game engine released uses SCons to build..
On 11.02.2014 05:57, William Deegan wrote: http://beta.slashdot.org/story/198003 That's cool news...would it make sense to send them a short note, thanking them for choosing SCons, and offering them help on our user mailing list if they need it? Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Fwd: [GSoC Mentors Announce] Now Accepting Applications for Mentoring Organizations for GSoC 2014
The deadline for this is getting closer...do we apply? Dirk On 04.02.2014 01:06, Bill Deegan wrote: I like the packaging idea. For buildbot we use pip to install and run the development version, so easy. No need to set environment variables and such. And if users could pip install the package (which doesn't work right now btw), that would be awesome. -Bill On Mon, Feb 3, 2014 at 12:07 PM, Dirk Bächle tshor...@gmx.de mailto:tshor...@gmx.de wrote: On 03.02.2014 20:05, Gary Oberbrunner wrote: Hi folks; if we want to get a GSoC project this year, now's the time to think about it. Top of my priority list for a GSoC student would be someone to convert everything to python3, finishing what we've started already. Other ideas? Looking through the ideas at http://www.scons.org/wiki/GSoC2013Ideas , I'd think that improving the packaging and distributing of SCons would be a worthwhile project too. Dirk ___ Scons-dev mailing list Scons-dev@scons.org mailto:Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
Hi Gary, On 02.02.2014 23:13, Gary Oberbrunner wrote: HA -- got a small repro testcase! [...] Run that twice as scons all-defuns.obj. The second time _shouldn't_ rebuild anything, but it will re-run the Copy command. SCons 2.3.0 correctly doesn't do anything the second time. looks like I found a solution. The problem is that changed() gets called in different contexts: not only within make_ready_current/release_target_info after building a target, but also during scanning with a call stack like this: changed [FS.py:3052] is_up_to_date [FS.py:3121] current_check [__init__.py:309] __call__ [__init__.py:203] get_found_includes [FS.py:2684] get_implicit_deps [__init__.py:586] scan [Executor.py:474] scan_sources [Executor.py:455] Please find a patch attached and try it on your large build if you find the time. I've added an allowcache argument to the changed() method, that gets only set in the release_target_info path. This let's your simple testcase pass on my side... If you can confirm that this brings your build back to working properly, I'd create a pull request for this fix. Regards, Dirk diff -r d53323337b3a src/engine/SCons/Node/FS.py --- a/src/engine/SCons/Node/FS.py Sun Jan 05 13:27:10 2014 +0100 +++ b/src/engine/SCons/Node/FS.py Wed Feb 12 00:21:29 2014 +0100 @@ -2779,7 +2779,7 @@ if not hasattr(self.attributes, 'keep_targetinfo'): # Cache some required values, before releasing # stuff like env, executor and builder... -self.changed() +self.changed(allowcache=True) self.get_contents_sig() self.get_build_env() # Now purge unneeded stuff to free memory... @@ -3034,7 +3034,7 @@ self.scanner_paths = None -def changed(self, node=None): +def changed(self, node=None, allowcache=False): Returns if the node is up-to-date with respect to the BuildInfo stored last time it was built. @@ -3050,7 +3050,8 @@ pass has_changed = SCons.Node.Node.changed(self, node) -self._memo['changed'] = has_changed +if allowcache: +self._memo['changed'] = has_changed return has_changed def changed_content(self, target, prev_ni): diff -r d53323337b3a src/engine/SCons/Node/__init__.py --- a/src/engine/SCons/Node/__init__.py Sun Jan 05 13:27:10 2014 +0100 +++ b/src/engine/SCons/Node/__init__.py Wed Feb 12 00:21:29 2014 +0100 @@ -1049,7 +1049,7 @@ def Decider(self, function): SCons.Util.AddMethod(self, function, 'changed_since_last_build') -def changed(self, node=None): +def changed(self, node=None, allowcache=False): Returns if the node is up-to-date with respect to the BuildInfo stored last time it was built. The default behavior is to compare ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
On 12.02.2014 00:29, Dirk Bächle wrote: [...] This let's your simple testcase pass on my side... Uppss, please replace with: This lets... :) Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] please try latest default branch
On 13.02.2014 21:43, Gary Oberbrunner wrote: Hey, it works for me too! Okay, I created pull request #109, fixing this issue. Check and merge it when you find the time...and don't stress yourself out on the 2.3.1 release. This is still supposed to feel like fun, not work. ;) Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] [GSoC Mentors Announce] Re: Now Accepting Applications for Mentoring Organizations for GSoC 2014
Hi Russel, On 15.02.2014 08:40, Russel Winder wrote: On Thu, 2014-02-13 at 16:20 -0500, Gary Oberbrunner wrote: Thanks to Manuel Naranjo, our application is now in. Please go to http://www.scons.org/wiki/GSoC2014Ideas (which is just a clone of the 2013 ideas page so far) and add/edit/cleanup. I had a quick look at the 2 → 3 one and started making changes, but cancelled when I realized the final goal seems to be to create a pre 2to3 script so that the combination of the two create a SCons codebase transform. I don't agree with this, I think we should be looking to create a single codebase that runs with Python 2 or Python 3. Once I realized this I backed out of my changes to leave things as they were. please just continue editing the page and set the proper goals for this task. I wrote this text for GSoC 2013, back when our py3 branch didn't exist. So it's not really up-to-date anymore... Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons doesn't bootstrap without libxml2
Hi Anatoly, On 18.02.2014 05:46, anatoly techtonik wrote: Why SCons bootstrap became dependent on external libraries? I find it a major usability regression. Can this be fixed? it didn't suddenly become dependent, it always was. We're now using DocBook, so we need to process and transform XML...that's the single dependency to either libxml2 or lxml. And it's not a dependency for the regular user, but the role of the developer... Before that we required the full list of required tools as given at the bottom of http://www.scons.org/wiki/DeveloperGuide/Documentation . If you'd switch to another tool like e.g. Sphinx, as you suggested under http://www.scons.org/wiki/DeveloperGuide/Documentation/Discussion, then that would be the new dependency. It's just switching names, so I don't see how this is any better... So, - the external dependencies have actually been reduced, and - no, I don't think it can't be fixed. ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons doesn't bootstrap without libxml2
On 18.02.2014 12:12, anatoly techtonik wrote: [...] There is a mistake. The bootstrap process never require documentation tools to be present. Correct, the bootstrap process doesn't require doc tools...but the SConstruct at the top-level does. So unless you call bootstrap.py from the top-level source dir everything should be fine, do you agree? For the other case see my comments below please. hg up 2.3.0 bootstrap.py C:\Python27\python.exe R:\scons\bootstrap\src\script\scons.py scons: Reading SConscript files ... doc: jw not found, skipping building User Guide. doc: WARNING: no groff, skipping text and PostScript versions of man pages ... The bootstrap.py is a minimal build of SCons to bootstrap the full build of all the packages, as specified in our local SConstruct file. Doesn't the full build of all the packages,... include the full documentation? If yes, then it requires one of the libxml2/lxml bindings installed...and I don't see a problem with that. If not, what command do I have to call (assuming the role of a release manager) to get a full release build, creating all packages such that they're ready for shipping with the guarantee that no files are missing? Dirk P.S.: So, - the external dependencies have actually been reduced, and - no, I don't think it can't be fixed. ;) The last item was supposed to read: I don't think it can be fixed. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons doesn't bootstrap without libxml2
On 18.02.2014 19:59, anatoly techtonik wrote: [...] You need to ensure that there are no warnings during the build process and the warning about missing documentation build is among those that you especially should not ignore as a release manager. ;) I could live with both variants for the top-level build (packaging n' stuff): a) The default is to build and package SCons as far as the tools and requirements for the single steps are met. Documentation may get built and archives might get packaged and tested or not, depending on which tools/modules you have installed in your system. The focus is on trying out integration with a minimum of effort, especially regarding running time of the build. b) The default is to guarantee a correct build, which means that all packages get created such that they contain all required files and documents. All these packages pass their tests, especially the ones for self-containment, and are ready to get shipped. If one or more of these goals are not met, the build breaks. So I'd like to let the actual release managers decide. Just chime in guys... Personally, I'd always do the full build anyway. Sorry, but I simply don't believe in getting half-pregnant. ;) Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons doesn't bootstrap without libxml2
On 19.02.2014 00:00, anatoly techtonik wrote: On Tue, Feb 18, 2014 at 11:50 PM, Dirk Bächle tshor...@gmx.de wrote: On 18.02.2014 21:19, anatoly techtonik wrote: [...] Ok. I'll put it the other way. Between automating the job of release manager, which is done once in few months and automating the job of developer, which is more than once in a while, you choose the former. Why? Because my workflow seems to differ from yours. Do you use bootstrap.py to call a development version of SCons, or how do you do it? I use bootstrap.py for a quick test that development version is not completely broken. Okay, and when you have a simple SConstruct in a folder like /tmp/sconstest, change into this folder via cd /tmp/sconstest and then call python /full/path/to/scons/repo/bootstrap.py , does that work in 2.3.0 without having libxml2/lxml installed or do you see an error? ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons doesn't bootstrap without libxml2
On 19.02.2014 00:14, anatoly techtonik wrote: On Wed, Feb 19, 2014 at 2:08 AM, Dirk Bächle tshor...@gmx.de wrote: [...] Okay, and when you have a simple SConstruct in a folder like /tmp/sconstest, change into this folder via cd /tmp/sconstest and then call python /full/path/to/scons/repo/bootstrap.py , does that work in 2.3.0 without having libxml2/lxml installed or do you see an error? There is no error and should not be. Good, so you are able to develop SCons and run a checked-out, or even modified, version of SCons against a build project, right? Because in your earlier mail you said: My opinion is that by adding additional dependencies to run the SCons without errors from a fresh checkout we are significantly increasing contribution barrier and discouraging people from participating. People need to checkout and run to see the power of SCons. Not read, checkout, install, setup, run cycle. Something like this. But this is obviously not the case. When following the first instructions in the top-level README.rst, people are able to call SCons without installing it and without having to resolve any further dependencies. So there is actually no reason to fear that users or first-time developers get a bad first impression of SCons, when they try to use the latest development version. Can you see that too, and agree with me that we don't have a real problem in this very specific use case (cloning the repo, and calling SCons directly)? ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons doesn't bootstrap without libxml2
On 19.02.2014 06:15, Bill Deegan wrote: Anatoly, bootstrap.py is not meant to be run by users, only developers. -Bill I'd even go one step further and say: it's primarily meant to be run by release managers. It's okay if you take on this role for yourself as a developer while you're hacking away with things, but as far as I know there is nowhere documented that this is actually required from you. Nobody forces you now or has forced you in the past, to run this additional step, right? Or is it your understanding that every developer is required to run the full build scenario? I can understand that you are a little confused, and maybe even frustrated, because suddenly things that seemed to work for you show a different behaviour. But that's what happens. Time goes by and things change. And we want some changes for the SCons project, to make it better, right? And that's what we did, we made SCons better such that you don't have to write MAN pages by hand anymore for example. As a consequence of this, you simply don't get away anymore with what you did in the past: running only half of the packaging test without the documentation. But this is also a change to the better side and not meant to be against you personally. It reduces the work load for the actual release managers because errors in the documentation syntax are revealed much earlier in the development process. And you can still get back to your old routine and workflow and help the project even more and better than before, if you decide to take that little step and install the libxml2 or lxml Python bindings. And if you decide to not install it, and simply skip the full packaging build, that'll be fine with everyone too...and you can save even more of your time and invest it in development itself. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons doesn't bootstrap without libxml2
On 19.02.2014 18:07, Bill Deegan wrote: Might I suggest we stop discussing it and just propose pull requests. If you have a specific change in mind, then make it and send a pull request. Yup, I'm all for it. @Anatoly: the commits that introduced the new doc toolchain are 8ca01af:0c9c8af (pull request #77, 2013-05-04). Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Question...(not flamebait)
On 03.03.2014 20:37, Bill Deegan wrote: Anatoly, This is a bikeshed'ing type discussion. So we'll never agree on this. The project has long had the emacs/vim info in each source file, so there's no reason to change. That said, some automated checking in buildbot might not be a bad idea. Though I'd put money that SCons source code (haveing long preceded PEP8 rules) won't conform (at present) to PEP 8. It might be worthwhile to get there though as such automated checking would get easier. Other opinions on PEP-8 formatting compliance? I, personally, don't feel like I would write better code (or more code) with it. And as a user I don't care how the SCons sources look, as long as it's doing what I expect from it. Let's all together try to get Python3-compliant instead... ;) Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Is the release 2.3.1 out?
Hi, has the version 2.3.1 officially been released yet? I see the packages on Sourceforge and the docs on the web page, but I miss the official Announce email. Some of my friends also watch the project via Twitter. Can someone post a short message there? Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Fwd: [Bitbucket] Pull request #119: Major revamp of the D language support (scons/scons)
Hi there, On 19.03.2014 15:08, Gary Oberbrunner wrote: This is worth reading, IMHO. From Martin Geisler. You can see more in the pull request (which has been withdrawn, so you won't see it by browsing open PRs on bitbucket), where he explains why pulling Russel's change causes an explosion of branchiness due to mercurial's changeset ordering. I took the time to read through the comments for this pull request, and now would like to share my first impression with you all. The merge graph may look a bit unusual, but this is simply the first time that a long-lived dev branch gets back onto default. I, personally, wouldn't feel the itch to do anything about it. Let graphs get messy and ugly, but this is what happens when you actually work with a VCS. Polishing, folding and rebasing commits only to make the history look pretty is not really required from my side...but let me stress this again, it's just my personal opinion. I really acknowledge and appreciate all the work that Russel, Gary and Martin have put into this. If you like a clean history better, fine...just continue. Especially if this removes your obvious doubts about whether Mercurial did the right thing (This merge looks mixed up, it can't be right.). But you don't have to do it for me. ;) Best regards, Dirk P.S.: Just as an addendum, trying to provide an explanation for what I wrote above...I see worse things every day, we're using ClearCase at work. ;) ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi there, On 01.04.2014 18:13, Gary Oberbrunner wrote: I've found posix spawn can be much faster than fork/exec with large memory processes, so I'd be in favor of this. Not every system has it though so there would have to be a fallback to fork/exec. -- Gary Oberbrunner (sent from my Android) On Apr 1, 2014 11:52 AM, Kenny, Jason L jason.l.ke...@intel.com mailto:jason.l.ke...@intel.com wrote: Hi guys, I just got a patch to Parts internal at Intel to fix some issues with subprocess. I find myself sort of surprised by this, and honestly felt that this seems to be an issue that should be looked at in Scons as well. [...] Does anyone else have input on this? IF this is a good patch, it seem that we will want to apply it to SCons, for a speed boost. a few minutes ago I updated the page http://www.scons.org/wiki/WhySconsIsNotSlow with my latest results. Check out the bottom of the page, and the repository http://www.bitbucket.org/dirkbaechle/scons_testresults for all the results. I'm also talking to Tzvetan Mikov off-list, he volunteered to write a CPython extension for posix_spawn() to help SCons out. I'll try to get him subscribed to this list, so the three of you can talk some more about this, okay? Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 02.04.2014 23:38, Gary Oberbrunner wrote: On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle tshor...@gmx.de mailto:tshor...@gmx.de wrote: This idea may be feasible, but I'd rather try to get the actual shell spawning to be as fast as possible. We have some valid approaches for this, so let's try them out...maybe one of them is fast enough, such that we don't have to care about the extra work mentioned above anymore. Speeding up the spawn/fork stuff would be more transparent to the user than trying to detect which commands need a full shell and which don't. They're orthogonal. Both useful, but either can be pursued independently of the other. Avoiding shells will be most valuable in builds with lots of tiny commands (could halve the build time). Avoiding fork is most valuable when SCons is using lots of memory (which it often is). My sense, based largely on Dirk's research, is fixing spawn has a bigger payoff right now. Since we now know that the fork problem is something to care about, I'd really like to use the current momentum. I don't want to push anybody, I want us to have a clear vision about how to take steps in the right direction. Is someone working on this right now? If not, who volunteers? Let's just communicate...and be on the same page about knowing *who* does *what*, and *when*. What's a little behind this is, that I think this point has some strategic importance. It's a single issue that keeps a lot of users from switching to SCons. By getting it out of the way, we can present (and kind of sell) our project much better in any media out there. I have submitted a proposal for the poster session at the EuroPython 2014 in Berlin. It's title is How spawning many processes sequentially can kill performance, and if I could not only talk about our problem, but also present a solution that has some benefit for the Python community, that definitely carries some potential in my opinion. So let's really get this crackin'! ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi Jason, On 05.04.2014 00:17, Kenny, Jason L wrote: I think yes, in that it does what should be done by the system under posix_spawn.. ie call vfork and execve. Here is the last version of the monkey patch I have from the people working on it. It has a fallback to the classic fork exec if the API's don't exists. It seems to solve the main speed problem for us at the moment. I believe it still being tested to find possible issues. I tried to let SCons run with this patch, but I'm not able to integrate the patching of subprocess.Popen() successfully. I always get an import error for SCons.Util, and fail to see where this is actually coming from: scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... gcc -o d1_0/f0_sconsbld_d1_0.o -c -Id1_0/lup000_sconsbld_d1_0 -Id1_0/lup001_sconsbld_d1_0 d1_0/f0_sconsbld_d1_0.c Traceback (most recent call last): File /home/dirk/workspace/scons_dirkbaechle/src/engine/SCons/Platform/posix.py, line 42, in module import SCons.Util ImportError: No module named SCons.Util scons: *** [d1_0/f0_sconsbld_d1_0.o] Error 1 scons: building terminated because of errors. Does anybody have a clue and can give me a pointer? I'd like to compare the times for a clean build against the default sources. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi Eugene, thanks a lot for your quick answer and very helpful advice. On 06.04.2014 15:21, Leskinen, Eugene wrote: I have just place the stubprocess.py to /opt/python27/lib/scons-2.1.0/SCons/Platform/ directory and added 'import stubprocess' statement to SCons.Platfrom.posix module just after import subprocess: --- /opt/python27/lib/scons-2.1.0/SCons/Platform/posix.py 2014-04-06 17:17:30.0 +0400 +++ /opt/python27/lib/scons-2.1.0/SCons/Platform/posix.py.new 2014-04-06 17:17:07.0 +0400 @@ -36,6 +36,7 @@ import os import os.path import subprocess +import stubprocess import sys import select This what worked for me. This approach works on my side too. I had tried to copy-paste part of the source code directly to the top of SCons/Platform/posix.py, which gave me the reported errors. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? No, it doesn't work: 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live.py /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py returned 2 STDOUT = scons: Reading SConscript files ... STDERR = KeyError: 'PATH': File /tmp/testcmd.4854.Zu6ZsJ/SConstruct, line 2: foo = Environment() File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 1003: apply_tools(self, tools, toolpath) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 107: env.Tool(tool) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 1787: tool(self) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 183: self.generate(env, *args, **kw) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/default.py, line 40: for t in SCons.Tool.tool_list(env['PLATFORM'], env): File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 819: ], env) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 690: return list(filter (ToolExists, tools)) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 689: return Tool(tool).exists(env) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/wix.py, line 71: for path in os.environ['PATH'].split(os.pathsep): File /usr/lib/python2.7/UserDict.py, line 23: raise KeyError(key) FAILED test of /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py Looks like the wrapping of Subprocess.Popen in stubprocess.py prevents the os.environ settings to get through, so all tests that setup a simple Environment and auto-detect Tools are bound to fail. :( Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 09.04.2014 23:56, William Deegan wrote: Dirk, Is this available in your bitbucket repo? (URL?) No, I just used the subprocess.py from Eugene's mail and put it in my local copy of the SCons source tree (plus the import in posix.py). I didn't bother to add it to any repo yet... Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] tracking command line argument changes in a custom tool ?
Hi Ram, On 11.04.2014 21:06, Ram Bhamidipaty wrote: How do I track build variable changes in a tool ? In my case I have a tool that I invoke like this: env.MyTool(file.o [file.c], V1=1, V2=2, ...) The tool is a bit complicated - I use Builder() objects to construct a file that contains the command line arguments for the compiler and then I use another Builder() to run the compiler. This is modeled on the tools for fools wiki page. The problem I am seeing is that if I change the variables used to invoke MyTool() scons does not rebuild the file. Question: In a tool - how does one track variable changes that would require the output file to be regenerated? you shouldn't have to do anything special, if the dependencies are setup correctly. Did you make sure that your file.o depends on the intermediate file with the compiler command(s)? You are right that your Builder is somewhat complicated, but it probably doesn't have to be. Did you have a look at the command generator feature yet? You can find it in the UserGuide, sect. 18.5 Builders That Create Actions Using a Generator. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi guys, I just found this talk by Christine Spang, given at the PyCon 2014 in Montréal: http://pyvideo.org/video/2640/subprocess-to-ffi-memory-performance-and-why-y It's really worthwhile to watch, I think. However, I don't think CFFI can solve our basic problem...it looks as if there is no support for the redirection of stdout/stderr and only actual library calls can be wrapped. Does anybody know more about this? Given our current results with the stubprocess.py wrapper, I'd like to raise the question again about how to proceed from here in general. I basically have enough material to show at the EuroPython 2014 in Berlin (if my proposal gets accepted), but what's our plan for the further development with this? Your comments are welcome... Best regards, Dirk On 11.04.2014 13:55, Leskinen, Eugene wrote: Here comes the latest version of stubprocess.py. It supports Python up to version 2.7.6 from now. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 23.04.2014 01:42, Bill Deegan wrote: Dirk, So if I understand correctly, the stubprocess patch passes all the regression tests and is signficantly faster than the current implementation? That's correct, but I'm not sure whether things like the redirection of stdout/stderr works in all cases, or might create quirky problems in certain situations. I just can't judge the technical stability level of the stubprocess.py source. Is there any downside to using it? (Does it work on py3?) I haven't tried the wrapper under Python3 yet. But as far as I understood Eugene and Jason, it's not implemented. Another downside is that this doesn't work under Windows obviously, I don't have any infos about OSx, and Alexandre Leblot already asked me in a PM about the support for Solaris. I can't test these things, so I've no idea. ;) If not is there any reason not to send a pull request? Creating a pull request is probably not difficult, on the technical level. But should the wrapper get activated automatically under Linux, transparently for the user? Or would it be an experimental option at first, that the user can activate via a command-line option...since we aren't that sure about all the possible implications? Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New keywords for Tigris bugtracker...
On 27.04.2014 17:53, Gary Oberbrunner wrote: [...] Funny you should mention that. I'm just adding SCons to openhatch (http://openhatch.org/projects/SCons) per your email last week and my conversation with their leader. I added our tigris tracker, or at least tried to... we'll see if it works. One thing they ask for is how to select bite-sized bugs that would be easy for a newbie to help with; their project setup has a field to select keywords for this. I added Easy. I also had exactly OpenHatch in mind, when asking for Windows/Fortran. ;) We could try to find someone for simply testing old issues about whether they still exist, and possibly developing basic strategies for resolving them (which command-line option is needed for this tool to make things work?). Actually fixing stuff would be a step at the next level for them, so they can slowly progress into the project and are not overwhelmed straight away. I just added the two keywords you mentioned as well. Thanks a lot for that, and also for taking action about OpenHatch already. Very cool! Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons and octal constants
On 28.04.2014 07:48, Russel Winder wrote: Since the floor version of SCons is now Python 2.7, we should dispense with the horror that is 1970s C-style octal constants and use the 0o form (*). This applies to the default/default branch just as much to the default/python3-port branch (where it is needed for SCons to run at all on Python 3). +1 from me. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons and octal constants
Hi Bill, On 30.04.2014 00:33, Bill Deegan wrote: All, If someone can point me to a pylint command line which should work for scons repo, I can add it to buildbot. the fix to astroid seems to be working fine. I installed the latest version of pylint: logilab_common 0.61.0 astroid 1.1.1 pylint 1.2.1 (each installed from latest revision of their repositories, so I'm not sure the version numbers are correct). Then, in the top-level of the SCons repository you can call PYTHONPATH=src/engine pylint src/engine/SCons and pylint runs through to the end. We might want to setup a customized config file, for getting rid of all those annoying invalid variable name messages...but just try to get it working on your side for now. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Jar builds fail with SConscript variant directory
Hi William, On 01.05.2014 01:01, William Roberts wrote: I typically set up my builds with a hierachical layout and a variant directory and do some other things to make it all work together. Normally I program in C, and this all works. Recently, I tried using java and broke. I have a very simple example that doesnt work. If I change the SConscript calls to use a variant dir, its broken, if I dont it works... Sconstruct: snip SConscript(script) #SConscript(script, variant_dir=outdir, duplicate=1) snip to snip #SConscript(script) SConscript(script, variant_dir=outdir, duplicate=1) snip it breaks... Can someone explain what's going on here? Thanks. possibly, if you could give us a little more info about your example. Like, what's in the variables script and outdir? What's the exact error message that you get? You'll also reach more people (= faster response times and more thorough discussions) if you post these kind of questions to the user mailing list first. The developers are subscribed to it too, and will chime in if necessary. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Survey: SCons Wiki, access restrictions...
Hi, after the big Wiki attack in 2013 and the restoring of old contents, the SCons pages still show the red banner with a warning message at the top. Maybe it's time to switch this off, to make the Wiki appear a little friendlier? Along the same lines, we might consider removing the blocking of new account creation. With the ApprovalQueue being in place, we are still protected against spam. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] New keywords for Tigris bugtracker...
On 01.05.2014 14:46, Gary Oberbrunner wrote: On Thu, May 1, 2014 at 7:05 AM, Dirk Bächle tshor...@gmx.de mailto:tshor...@gmx.de wrote: ... FYI, I'm currently working on this ( https://openhatch.org/bugs/issue972 ) . The basic implementation of the Tigris bugimporter is completed, now I just need to add proper tests and then get the pull request through. ;) cool! It's now at https://github.com/openhatch/oh-bugimporters/pull/51 , but judging from the last accessed/changed times on the other pull requests, it can take a short while before something happens. Every open source project appears to have the same problems... ;). Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] PyConUK 2014 is coming...
On 03.05.2014 17:54, Russel Winder wrote: On Sat, 2014-05-03 at 14:03 +0200, Dirk Bächle wrote: just saw that PyConUK ( http://pyconuk.org/ ) has started their call for proposals (talks/sprints/whatever), that get accepted by 19th August. I guess this mainly goes out to Russel ;) , but it would be cool if SCons would be on the conference schedule in some way. The question is what is the interesting Python angle? I know, too bad there aren't many build system conferences around. ;) The subprocess issue might be of some interest to people...there is some data and technical discussion available. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Bugtracker and stuff...
Hi there, I'm currently wading through our issue list at tigris.org, trying to clean things up a little (closing resolved bugs and such...). Sorry, if this creates some noise on the corresponding mailing list... I found issue #2739, which is in state STARTED but assigned to issues@scons...so nobody effectively. I think that only NEW or REOPENED issues should be assigned to issues@scons, leaving the question how to resolve this? Can someone take it? Further stirring up old topics ;), I pondered over the bugtracker migration task again. For the issue interface to OpenHatch, I now have a bug importer ready. It's capable of scraping our whole issue database (2947+ issues) from tigris.org to a single JSON file in about 7 minutes. From there it wouldn't be much effort to push all the data via xmlrpc into a roundup ( http://www.roundup-tracker.org ) instance, for example. I'm not sure what our options are for adding new services to our web hosting. But roundup allegedly runs standalone, as well as via mod_python, using WSGI or through CGI. Maybe we could host this ourselves? Even if it's just to get away from Tigris *now*...as a first step. We might have to migrate further, later on. But then we're in a much better position (Python-based tracker, support for xmlrpc) to pull all our data out again, and convert it into a possibly new format. Just as an idea... Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Bugtracker and stuff...
On 04.05.2014 14:29, Gary Oberbrunner wrote: On Sun, May 4, 2014 at 7:38 AM, Dirk Bächle tshor...@gmx.de wrote: [...] This is a great start. Thanks for doing it! My preference would be a hosted bugtracker though, just because it's one less thing to manage (keep patched, ensure uptime etc.). Also, a minor point, but people may have more familiarity with the big hosted ones. On the other hand, roundup is used by python.org and openhatch.org...which is where some new contributors might come from in the future. Now, that said, I don't really have much experience with any of them that I like. So if roundup is nice, hosting it ourselves wouldn't be all that terrible. I'm pretty sure Pair, our current host, can do mod_python. Maybe it would be worth the try to setup a demo instance of roundup. We could run it in some kind of read-only mode, updating its data from the Tigris tracker time to time. Just to get a better feeling for the tool...although I'm pretty sure that it beats Tigris' Issuezilla easily. I mean, we can't list all of our bugs/issues currently...because we have more than 500 entries? Were not in the 80s anymore. ;) Ideally we'd have a stable tracker so we could put bug links into commit messages, and mailing lists, and they'd still work 10 years from now. But that may be an impossible dream. :-) As for #2739, I see I was involved in that. I tried to get the OP to write a test but that apparently was pushing a little too hard for him at the time. I guess I should take it. The batch-mode fixes in there are quite valuable. We have a lot of good patches and enhancements pending in the issue list, and my impression is that they've been rotting there much too long now. With a better overview this wouldn't have happened...but it's probably also the communication from the BugParties that's missing and playing a role here. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Bugtracker and stuff...
On 04.05.2014 15:49, Gary Oberbrunner wrote: [...] Maybe it would be worth the try to setup a demo instance of roundup. We could run it in some kind of read-only mode, updating its data from the Tigris tracker time to time. This should be pretty easy I think, presuming its dependencies are not too onerous. Dirk, do you have access to the web server? If not, I can get you access (send me your ssh pub key off-list) or I can try to set up the demo. (Note that we do NOT have root access on our server, but many tools work with locally-installed dependencies these days.) I don't have access to the web server yet. Before sending you my public key, I'd like to give other devs a chance to jump in. Setting up a bug tracker is certainly an interesting task, but I don't have to be all over the place with SCons. ;) Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Bugtracker and stuff...
On 04.05.2014 20:31, Russel Winder wrote: On Sun, 2014-05-04 at 09:49 -0400, Gary Oberbrunner wrote: […] Yeah, seriously. Along with that I really hate the Tigris bug-tracker interface. It is so needlessly complicated and hard to work with. It's not even easy to _find_ it from the main tigris project page. […] As you may remember, I have been ranting against the Tigris issue system for a while. It violates so many HCI good tenets and practices, it is no wonder to me that every time I think SCons Issue I immediately think Oh f###. And not in a good sense. Russel.told_you_so_karma += 10 :) ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Bugtracker and stuff...
On 05.05.2014 02:01, William Deegan wrote: Do we know the specs of the server pair is providing us with? roundup suggests mysql or postgressql for database. Need be we can put roundup on the same server as buildbot. I’ve installed and customized bugzilla for many many clients, so if you like I can take a whack at this. Cool, just go ahead...I won't mind. In the meantime I'll try to figure out how to push attachments via xmlrpc. Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
