Re: [Toybox] Out of tree build support
On 12/21/2017 11:16 AM, Rob Landley wrote: > On 12/19/2017 06:25 PM, Patrick Oppenlander wrote: >> I'm not receiving emails for this thread. Maybe the mailing list is >> smart enough to know you're CC'ing me? > > Ah, you're using gmail. This is a known perennial bug in gmail: > > https://landley.net/notes-2011.html#05-04-2011 And of course as soon as I send that I notice that the OTHER known perennial bug in gmail has trigered, namely gmail has false positived on an email as spam, and when it refused delivery of that message for every list recipient using gmail, and refused delivery of multiple retries from dreamhost's list server, the list server disabled their list subscriptions because mail was undeliverable. Here's the one it refused: http://lists.landley.net/pipermail/toybox-landley.net/2017-December/009309.html And here's a represenative bounce message: Final-Recipient: rfc822; soapboxcicero+toy...@gmail.com Original-Recipient: rfc822;soapboxcicero+toy...@gmail.com Action: failed Status: 5.7.1 Remote-MTA: dns; gmail-smtp-in.l.google.com Diagnostic-Code: smtp; 550-5.7.1 [64.90.62.195 18] Our system has detected that this message is 550-5.7.1 likely suspicious due to the very low reputation of the sending IP 550-5.7.1 address. To best protect our users from spam, the message has been 550-5.7.1 blocked. Please visit 550 5.7.1 https://support.google.com/mail/answer/188131 for more information. p76si13464492pfa.247 - gsmtp It does this about once a month these days. I remember it did it while it was in tokyo, but it looks like last time I complained about it here was September: http://lists.landley.net/pipermail/toybox-landley.net/2017-September/009149.html So that's another reason why you might not have gotten all the emails. I just finished manually re-enabling each user through the web ui (because dreamhost won't give me command line access to the mailman plumbing), so it should be back... There's a reason I check the web archive so much... Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] Out of tree build support
On 12/19/2017 06:25 PM, Patrick Oppenlander wrote: > On 20/12/17 09:24, Rob Landley wrote: > >> Really what I need to do is come up with a way to build toybox sed and >> toysh standalone from a canned script with no config step, and then >> build the rest with that. > > Makes sense. > >> Speaking of which, did I ever point you at the cp -s trick? >> >> $ cp -rs "$PWD"/toybox walrus >> $ cd walrus >> $ make distclean && make defconfig && make >> >> (I taught toybox to work with relative rather than absolute paths. The >> gnu/dammit version complains about relative paths, and yes that means >> even "cp -rs toybox walrus" won't work unless you're using toybox cp. >> It's because they don't bother to calculate the dynamic ../ stack.) > > That is a neat trick. First time I've seen it, thanks! https://xkcd.com/1053/ (I keep thinking the fact I've repeated something 30 times over the years means people are sick of it, when most of them have never heard of it. One of the reasons I'm so interested in computer history is preserving what used to be common knowledge, because https://web.archive.org/web/20120111055334/http://wrttn.in/04af1a having to come from archive.org sadly _isn't_ ironic, it's harmonic. It _reinforces_ the darn point.) >> One of my todo items is writing a new kconfig to replace the kernel >> plumbing I borrowed way back when. (Which is from 2.6.12.) > > Even before that the issue is that there's no way to build kconfig from > a script, so the *config targets depend on make. There's no way to build the _current_ kconfig plumbing from a script. But if I write new infrastructure... (Alas it's another thing that's beyond the threshold of what I can sit down and do in a couple hours from a standing start, which would be negatively impacted by interruption and forced context switch...) >>> I've pushed the current state of affairs to >>> https://github.com/pattop/toybox if you're interested in taking a look. >> >> H... I see what you're doing, but you're adding complexity to the >> Makefile and I want to strip it out and move it into the scripts. > > I think the changes to the top level Makefile aren't too bad. They > really just boil down to teaching make where to find sources & scripts > by sprinkling in a few $(SRCDIR)s and don't change any of the logic at all. Complexity often accumulates in small amounts. I'd prefer to solve this by _removing_ stuff from Makefile instead of adding it. Looking at the design, I suspect I should rename "scripts" to "make" and have a subdirectory under there for files that aren't a build target (make/include or something). Then the top level Makefile should be a wrapper that _just_ calls scripts under "make". For the moment kconfig/Makefile can stay crazy because that whole mess is its own can of worms. But I need to close tabs before opening a new one just now... > kconfig/Makefile probably needs to be replaced by a script so the > hackery in there is fairly moot. Would you be interested in a script > that replaces this? Somebody wrote some plumbing to (partially) replace it in awk once, but I just couldn't maintain it. :( > Are you happy with the direction of the changes to the scripts in general? Sort of? Make provides an expected command line API. The classic build is "./configure ; make ; make install" and long ago I ranted (in two parts) about how that's annoyingly obsolete at: http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000859.html http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000860.html But we haven't standardized on something new and better yet (the way cvs gave way to git). Instead we've got fragmentation: the gnu guys did their own "automake" syntax which is almost sane but nobody else picked it up because it had gnu all over it and those were the guys who did "info". Android decided to use something called ninja while simultaneously switching to an llvm toolchain that wants to build with cmake... Despite the many downsides of make, it has the advantages that it's posix and still near-ubiquitous. Getting back to configure/make/install: the Linux kernel swapped out configure for "make defconfig" and friends, and as with Linux switching to git lots of people are familiar with that now. It was copied by busybox and uclibc, spread from there to buildroot (which started life as uClibc's test harness), and now openembedded and so on have taken it from there... That said, we shouldn't be tied to a given implementation of either. An implementation is not a standard, so you can't _have_ a standard until there are two compatible implementations. (This was official IETF policy at one point, dunno if it still is.) So I want to write my own mostly-bash-compatible shell, I want to write my own mostly-gmake-compatible make, and I want to write my own kconfig plumbing. Long ago I tried to push some of my kconfig changes from busybox upstream into linux-kernel, this would have been
Re: [Toybox] Out of tree build support
On 12/19/2017 07:01 PM, Patrick Oppenlander wrote: > On 20/12/17 09:24, Rob Landley wrote: >> Really what I need to do is come up with a way to build toybox sed and >> toysh... > > How far are toybox sed & toysh from being usable to build toybox? Sed's fine, I've built Linux From Scratch with it (albeit an older version, 6.3 I think) and even done some patches for things like perl bugs: https://github.com/landley/toybox/commit/32b3587af261 Which I found because I was building perl with it. I haven't properly opened the toysh can of worms yet because toybox isn't my dayjob and that one is going to need multiple consecutive months to do right. Unfortunately most of the big remaining roadmap items are still there because they require large consecutive chunks of focus that the tiny little startup I've been working at since 2014 can't afford to give me without going under. Unfortunately I haven't had as much spare time the past year and a half as I did before my startup ran out of money and went into survival mode. (Which is as much exhausting as time consuming, I _do_ have spare time, but I spend it sleeping. The real problem is I can never focus on anything, I get high priority crisis crisis crisis world on fire interruptions that derail what I'm working on and by the time I get back it would be faster to start over. After about 30 of those I got a little gun-shy about starting anything big. I've sat down and started over on the dd cleanup _four_times_ now, and that's not even all _that_ big.) That said, my 2013 talk on my grand plan for turning android into a development environment is coming up on 5 years ago, and if I'm going to make it happen I have to get unstuck pretty much now. (I'm looking at selling my house. Last ELC Tim Bird asked what it would take to buy six months of my time from my employer to do toysh, and I named him a figure, and he went away and I never heard back. I think Sony had a reorg or something. Sigh, I feel really bad about the rate of development but I don't have as many gaps in my life to fit development into as I used to. Downside of a dayjob at a startup that's _also_ trying to change the world...) But hey, christmas break coming up. I'm hoping I can use it to make next release less embarassingly "I got nothin'" than last release was. Maybe I can get _one_ of my larger mostly finished things done. I called getconf "mostly done" back in March, according to http://landley.net/notes-2017.html#04-03-2017 anyway... Speaking of which, I should catch up on my blog... there we go, another month posted. (I didn't check twitter or my back email for things I forgot to write about at the time, but eh. Edited to at least make the tags match up. Next entry trails off half-finished and requires surgery, of course...) Rob "I am sorry to have wearied you with so long a letter but I did not have time to write you a short one" - Blaise Pascal ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] Out of tree build support
On 20/12/17 09:24, Rob Landley wrote: Really what I need to do is come up with a way to build toybox sed and toysh... How far are toybox sed & toysh from being usable to build toybox? Patrick ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] Out of tree build support
On 20/12/17 09:24, Rob Landley wrote: Really what I need to do is come up with a way to build toybox sed and toysh standalone from a canned script with no config step, and then build the rest with that. Makes sense. Speaking of which, did I ever point you at the cp -s trick? $ cp -rs "$PWD"/toybox walrus $ cd walrus $ make distclean && make defconfig && make (I taught toybox to work with relative rather than absolute paths. The gnu/dammit version complains about relative paths, and yes that means even "cp -rs toybox walrus" won't work unless you're using toybox cp. It's because they don't bother to calculate the dynamic ../ stack.) That is a neat trick. First time I've seen it, thanks! One of my todo items is writing a new kconfig to replace the kernel plumbing I borrowed way back when. (Which is from 2.6.12.) Even before that the issue is that there's no way to build kconfig from a script, so the *config targets depend on make. I've pushed the current state of affairs to https://github.com/pattop/toybox if you're interested in taking a look. H... I see what you're doing, but you're adding complexity to the Makefile and I want to strip it out and move it into the scripts. I think the changes to the top level Makefile aren't too bad. They really just boil down to teaching make where to find sources & scripts by sprinkling in a few $(SRCDIR)s and don't change any of the logic at all. kconfig/Makefile probably needs to be replaced by a script so the hackery in there is fairly moot. Would you be interested in a script that replaces this? Are you happy with the direction of the changes to the scripts in general? It might be worth considering passing $(MAKECMDGOALS) to make.sh (or a higher level driver script) at some point. It could then call single.sh or test.sh and eliminate the requirement for .singlemake. You could then call make.sh from a catch all rule in the top level makefile and really turn the makefile into a trivial wrapper. The symlinks in kconfig are a bit nasty (they really should go in the build directory to avoid polluting the source dir, but as they're always the same...). I got this working with vpaths earlier but wasn't too thrilled with the result. What's the reason for the symlinks in kconfig anyway? I had a quick look through the history and couldn't find anything obvious. CC'ing you as I don't seem to be getting list emails at the moment. I seem to be, but I think I've been cc'd on everything the past couple days anyway? I'm not receiving emails for this thread. Maybe the mailing list is smart enough to know you're CC'ing me? Patrick ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] Out of tree build support
On 12/18/2017 07:49 PM, Patrick Oppenlander wrote: > On 19/12/17 10:41, Rob Landley wrote: > That's a nasty chicken-and-egg problem indeed. I just thought you hated > make :) Oh I do. But I tend to have programming opinions for a reason. :) >> I also have a "configure" file where you can persistently set default >> values for overrideable build options. This is in shell syntax rather >> than Makefile syntax so only the shell scripts import it, if I import it >> from the Makefile it gets confused by the "[ -z "$BLAH" ] && BLAH=" >> syntax. > > I have another project which has the same problem, but it also needs the > variables available for 'ld' to use in link scripts. The only workable > solution I know of is to have a config file which is processed by a > script to generate config.mk, config.ld, config.sh and config.h which > can be included as required. Which is more build dependencies, and the parsing gets nontrivial. Really what I need to do is come up with a way to build toybox sed and toysh standalone from a canned script with no config step, and then build the rest with that. (See also "make install_airlock".) Speaking of which, did I ever point you at the cp -s trick? $ cp -rs "$PWD"/toybox walrus $ cd walrus $ make distclean && make defconfig && make (I taught toybox to work with relative rather than absolute paths. The gnu/dammit version complains about relative paths, and yes that means even "cp -rs toybox walrus" won't work unless you're using toybox cp. It's because they don't bother to calculate the dynamic ../ stack.) > I have done some hacking on the scripts. This currently works: > > mkdir /tmp/toybox_build > make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile defconfig > make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile > > So does this: > > mkdir /tmp/toybox_build > make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile defconfig > cd /tmp/toybox_build && ~/src/ext/toybox/scripts/make.sh > > So do other targets (like tests, etc). bloatcheck looks to be broken on > trunk at the moment (incorrect dep on toybox_old?). Blah, "make baseline" is putting it in generated/unstripped but it looks like "make bloatcheck" is still expecting it at the top level. [fixed] > How do you intend to do the defconfig step using scripts? One of my todo items is writing a new kconfig to replace the kernel plumbing I borrowed way back when. (Which is from 2.6.12.) > I've pushed the current state of affairs to > https://github.com/pattop/toybox if you're interested in taking a look. H... I see what you're doing, but you're adding complexity to the Makefile and I want to strip it out and move it into the scripts. > The symlinks in kconfig are a bit nasty (they really should go in the > build directory to avoid polluting the source dir, but as they're always > the same...). I got this working with vpaths earlier but wasn't too > thrilled with the result. > > CC'ing you as I don't seem to be getting list emails at the moment. I seem to be, but I think I've been cc'd on everything the past couple days anyway? Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] Out of tree build support
On 19/12/17 10:41, Rob Landley wrote: On 12/18/2017 03:57 PM, Patrick Oppenlander wrote: Hi Rob, are you open to accepting patches for building out of tree? I am working on a project where Toybox will be part of an automated build system, and this would make life a lot easier. Patrick It's one of my todo list items, but there's backstory: I've tried to make it so scripts/make.sh can be run separately (not via the Makefile, that should be an essentially cosmetic wrapper around the script-based build infrastructure). (In theory this means you can build it on systems that haven't got a functional "make" yet. I have another todo item to have a scripts/defconfig that populates generated/*.sh with absolutely minimal dependencies, working even with apple's broken sed...) That's a nasty chicken-and-egg problem indeed. I just thought you hated make :) I also have a "configure" file where you can persistently set default values for overrideable build options. This is in shell syntax rather than Makefile syntax so only the shell scripts import it, if I import it from the Makefile it gets confused by the "[ -z "$BLAH" ] && BLAH=" syntax. I have another project which has the same problem, but it also needs the variables available for 'ld' to use in link scripts. The only workable solution I know of is to have a config file which is processed by a script to generate config.mk, config.ld, config.sh and config.h which can be included as required. I may be able to hack something together at some point in the future if you're at all interested in this idea. (The build has three conflicting use cases, which affect the environmental dependencies and expected command line user interface. 1) Conventional "configure/make/install" using gmake. 2) Portable build not depending on anything toybox doesn't itself provide (building on BSD and such). And android's doing ninja, which I should have support for but ninja is a moving target, the one 14.4 installs doesn't work at all for android...) Nasty indeed. I have done some hacking on the scripts. This currently works: mkdir /tmp/toybox_build make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile defconfig make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile So does this: mkdir /tmp/toybox_build make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile defconfig cd /tmp/toybox_build && ~/src/ext/toybox/scripts/make.sh So do other targets (like tests, etc). bloatcheck looks to be broken on trunk at the moment (incorrect dep on toybox_old?). How do you intend to do the defconfig step using scripts? I've pushed the current state of affairs to https://github.com/pattop/toybox if you're interested in taking a look. The symlinks in kconfig are a bit nasty (they really should go in the build directory to avoid polluting the source dir, but as they're always the same...). I got this working with vpaths earlier but wasn't too thrilled with the result. CC'ing you as I don't seem to be getting list emails at the moment. Patrick ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] Out of tree build support
On 12/18/2017 05:41 PM, Rob Landley wrote: > (Probably configure should have VALUE?="blah" and then a shell wrapper > that parses and sets that syntax.) Except when I sat down to try to implement that (again) I hit the problem that configure variable defautls reference other config variables (such as CROSS_COMPILE="$CROSS") and even writing a simple parser to find ?= lines and turn them into = lines, to import that as makefile syntax it has to be CROSS_COMPILE="$(CROSS)" with gratuitous parentheses, and that a giant rathole. Having variables you can set in both shell and makefile contexts turns out to be kind of annoying. What I need to do is remove all the remaining logic from the makefile and put it in the script and have the makefile be just a dumb wrapper around the scripts like it was intended to be... Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] Out of tree build support
On 12/18/2017 03:57 PM, Patrick Oppenlander wrote: > Hi Rob, > > are you open to accepting patches for building out of tree? > > I am working on a project where Toybox will be part of an automated > build system, and this would make life a lot easier. > > Patrick It's one of my todo list items, but there's backstory: I've tried to make it so scripts/make.sh can be run separately (not via the Makefile, that should be an essentially cosmetic wrapper around the script-based build infrastructure). (In theory this means you can build it on systems that haven't got a functional "make" yet. I have another todo item to have a scripts/defconfig that populates generated/*.sh with absolutely minimal dependencies, working even with apple's broken sed...) I also have a "configure" file where you can persistently set default values for overrideable build options. This is in shell syntax rather than Makefile syntax so only the shell scripts import it, if I import it from the Makefile it gets confused by the "[ -z "$BLAH" ] && BLAH=" syntax. The problem is both the Makefile and scripts/make.sh use the generated/ directory, but the configure file isn't in Makefile syntax. I could put the default value in two places but having the same piece of information in two places pretty much guarantees they'll get out of sync eventually... The HOSTCC variable already has this problem. The reason I haven't implemented this yet is every time I sit down to implment it I get distracted by trying to come up with a better design for this... (Probably configure should have VALUE?="blah" and then a shell wrapper that parses and sets that syntax.) Rob (The build has three conflicting use cases, which affect the environmental dependencies and expected command line user interface. 1) Conventional "configure/make/install" using gmake. 2) Portable build not depending on anything toybox doesn't itself provide (building on BSD and such). And android's doing ninja, which I should have support for but ninja is a moving target, the one 14.4 installs doesn't work at all for android...) ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
