On Fri, Sep 22, 2023 at 9:15 AM Rob Landley <r...@landley.net> wrote: > > On 9/20/23 22:37, Oliver Webb wrote: > > ------- Original Message ------- > > On Wednesday, September 20th, 2023 at 3:07 PM, Rob Landley > > <r...@landley.net> wrote: > > > > > >> On 9/20/23 01:41, Oliver Webb via Toybox wrote: > >> > >> > I have made a implementation of the 'dc' command in toybox > >> > > >> > Since dc isn't specified in POSIX or LSB, > >> > >> > >> Which is why it's not in toybox? > >> > >> There are things listed in yocto's root filesystem in the roadmap which I > >> don't > >> intend to add to toybox. > > > > I was unaware of this, The other things that the stuff in the roadmap are > > from have been curated > > to remove unwanted or out of scope commands from the roadmap so I thought > > the yocto project > > would have been evaluated in a similar way before being put in the roadmap. > > My bad, adding yocto to status.html was a bit misleading, but I already had > klibc, sash, sbase, beastiebox, and tizen in there, all of which are data > points > but not necessarily endorsements. > > It's not that I've made a strong decision NOT to add them, it's that they > showed > up in an environment of dubious design judgement, and I haven't seen a > consumer > of them yet. "Under consideration" is probably the best phrasing. > > (I've also personally started to distinguish between "1.0" commands and > "post-1.0" commands like screen and rsync, but don't have a way to annotate > that > in the roadmap yet...) > > Also, this command is mentioned in the roadmap in a context other then the > > yocto > > project. In the evaluation of the LFS packages for which ones we would > > provide > > "complete-ish replacements for", both bc and dc are mentioned. > > Which is from LFS 6 and version 12 just came out. I'm digging towards an > updated > LFS/BLFS to see what's actually still used. I'd also love to build AOSP with > the > command line logging wrapper, but that at least partially sets up its own > $PATH > internally and is a can of worms to fiddle with. > > For any of these build environments that are adding and installing tools, > there's a design question of "you just built your own sed to replace the > existing sed: so I stop logging calls to sed now? Do I let the build stop > using > the toybox one and use the replacement it installed, or keep testing the > toybox > one? It's mostly a "start or end of $PATH" question... > > > both bc and dc are mentioned. > > Back in the 1970s, having a calculator was a big deal. These days, fairly > competent 64-bit $((math)) is built into the shell. (I remember an exchange > with > Denys (busybox maintainer) about that in the > https://lkml.org/lkml/2009/12/8/94 > days fixing up busybox's shell which was the only one _not_ doing 64 bit math > on > 32 bit targets, and yes I'd tested Red Hat 9 from 2003.) Linux developers who > need arbitrary precision or floating point math have generally used perl or > python or similar. > > Posix's "expr" command still gets used a bit for historical reasons even > though > the need to split arguments is silly to modern audiences (expr 2+3 prints > "2+3"). > > The dc command is reverse polish notation, which I wouldn't think has really > been taught to new people since the Apollo program in the early 1970s? Yeah > some > old guys still worship 1970s HP calculators for some reason, but cheap > calculators that could handle infix came along 50 years ago. The TI Cal Tech > advertised it could do algebraic entry in 1967. By the time I got to school > they > were solar powered and disposable. In middle school I had a calculator > wristwatch with tiny buttons, and in high school I had a calculator the size > of > a credit card, made to fit in your wallet along with credit cards; none of > that > did Yoda Math). My Devuan Banana install hasn't got dc in the $PATH and > nothing > I've tried to build has ever complained about its absence, which is why I went > "um, do we really need this?". > > The "bc" command was added to the linux kernel build by Peter Anvin during my > "remove the perl dependencies peter anvin added to the build" patch series in > 2013: > > https://lkml.iu.edu/hypermail/linux/kernel/1302.3/01520.html > > Which I'd been regularly submitting for about 5 years at that point. Peter of > course did NOT participate in the discussion, he instead pushed a patch > directly > to Linus outside linux-kernel, adding a bc dependency. > > https://landley.net/notes-2013.html#28-03-2013 > > Nothing in either Linux From Scratch or Gentoo's base install had used it > before, which I know because both had to add it to build the updated kernel > (and > both were "build the root filesystem from source at install time" distros, so > nothing ELSE used it either). > > I got a bc submission to toybox in a cloud of drama, where the guy submitting > code to me insisted that someone else had stolen his code but he'd gotten away > from the old contributors and had a fresh project now... Which immediately > turned into fresh drama and squabbling they kept cc-ing me on for MONTHS: > > http://lists.landley.net/pipermail/toybox-landley.net/2018-March/017454.html > > http://lists.landley.net/pipermail/toybox-landley.net/2018-August/017677.html > > Busybox did NOT have a bc implementation at the time, the same implementation > was later submitted to busybox. If you check out the busybox source and go: > > find . -name '*.c' | xargs wc | sort -n > > It's the third largest file in the whole of busybox, and almost twice the size > of "awk.c" which is a programming language. If you do the same thing to > toybox, > it is THE largest C file in toybox (over a thousand lines larger than toysh). > > I did some cleanup passes on the bc contributed to toybox, removing a few > hundred lines at a time (some were merged "upstream" by the person drama > inexplicably follows for no reason, and marshalled into the busybox one and > the > standalone one), but... honestly, it would be faster to write a new one from > scratch. It has no business being bigger than awk. But if it wasn't a kernel > dependency nobody would use it today, so the real question is should it just > be > removed? And proving we _don't_ need it is "build LFS and BLFS" territory...
use of bc because $(()) isn't/wasn't available and expr sucks is pretty entrenched: https://cs.android.com/search?q=file:sh$%20%5Cbbc%5Cb%20-file:external%2F(bc%7Ctoybox%7Cmksh)%2F%20case:yes&sq= (most of those are in third-party open source things, not "actual" Android code.) whereas only one jemalloc test script seems to use dc: https://source.corp.google.com/search?q=file:sh$%20%5B%5E-%5D%5Cbdc%5Cb%20-file:external%2Fbc%2F%20case:yes&sq=repo:android%2Fplatform%2Fsuperproject%2Fmain%20b:main%20-file:prebuilts%2Fvndk%2F%20-file:prebuilts%2Fruntime%2F > So we have $((math)), we have expr in pending, we have an overdramatic bc in > pending, and that was the context in which I received your dc submission on > top > of those. > > > Which leads me into another question, what is your opinion about m4, make, > > yacc, and lex being in toybox? > > We need all 4 of them to build a current base Linux OS. I used to punt on lex > and yacc (you could build the gnu versions under an existing chroot without > anything in the minimal native development environment depending on them, so > they were leaf nodes not circular core dependencies), but the linux-kernel > loons > rewrote kconfig to require them in 2018. (Probably Peter Anvin again, he hates > dependency minimization as a concept, still dunno why.) > > The "make" should be gmake and thus big (gotta run kbuild in linux). I'm not > sure how much of the yacc->bison and lex->flex exensions get used because I > never program with either and haven't dug that far into them yet. I _think_ m4 > is mostly posix? (It's basically macro expansion, an alternative to the C > preprocessor that claims to be better in a way I haven't interrogated yet.) M4 > would probably be low hanging fruit if I'd ever used it before. (from a quick glance at m4 in the AOSP context a few years back, the trouble with m4 isn't m4 --- it's that every user of m4 in 2023 is assuming the huge gnu m4 macro library.) > > In every evaluation of the POSIX commands in the talks I have watched and > > documentation I have read, those commands are omitted on reason of being > > exclusively used to build packages, and therefore should be part of the > > compiler tool-chain. > > I felt that way myself once, but that's because I was maintaining a tinycc > fork > which I was teaching to act as a multiplexer (so it could be cc, ld, as, > strip, > etc) and adding "make" and friends to it wouldn't have been a very heavy lift. > Alas, that project's pretty thoroughly parked these days. > > Backstory: many moons ago I maintained a fork of tinycc: > > https://landley.net/code/tinycc > https://landley.net/hg/tinycc > > Which I was trying to A) extend to full C99 (including the vararray nonsense > and > alloca() and so on), because I wanted it to build a vanilla unmodified linux > kernel and busybox and uclibc and itself to create my minimal native > development > environment out of just 4 packages. And then build a bootable LFS under the > result. > > https://landley.net/code/qcc > https://elinux.org/CELF_Project_Proposal/Combine_tcg_with_tcc > https://landley.net/code/tinycc/qcc/todo.txt > > I got started because tinycc had built "tccboot" back in 2004, but that was a > modified kernel (not vanilla, ripped out a list of constructs tinycc didn't > understand yet) and I also vaguely recall that only compiled the _kernel_ on > boot and used an initramfs image with an existing busybox binary? > > https://bellard.org/tcc/tccboot.html > https://bellard.org/tcc/tccboot_readme.html > > Fabrice Bellard created tinycc as a joke in 2002 (an entry in the obfuscated C > code contest, which had something like a 4k size limit on the source code you > could submit: he submitted a C compiler that recompiled itself within the size > limit, and won "best abuse of the rules"). He then de-obfuscated it and > extended > it to _almost_ a full C99 compiler over the next couple years, and even > started > giving it multiple "backends" so it could generate code for multiple > processors. > > And then he started thinking about multiple _frontends_, specifically what if > instead of parsing C it parsed x86 machine code, and translated it to slightly > DIFFERENT machine code, and this turned into QEMU, which led to the concept > of a > "project tumor" that buds off and sucks all your developers away, which I > think > I ranted about here: > > http://landley.net/aboriginal/history.html > > Or maybe here: > > http://lists.busybox.net/pipermail/buildroot/2016-December/180102.html > > Anyway, QEMU ate all fabrice's time and tinycc stagnated for a while and I > proposed picking it up except the source control was in CVS and Fabrice's one > iron rule (at the time) is that the source control STAY in CVS so I did a fork > instead, and Fabrice eventually handed tinycc off to a Windows developer named > Grishka who only cared about using it as a free windows compiler and really > didn't care about Linux, and the REAL annoyance is ever few months we'd > repeat a > cycle where I'd do a bunch of work in my fork, and Grishka would wake up weeks > later and copy about half of it (leaving the half he didn't understand > behind), > and put out a new tinycc release, and Linux Weekly news and friends would > cover > the "real" tinycc and ignore mine, and I'd stop work so grishka's tinycc would > go stagnant again, and when I thought it was dead enough I'd start up work on > my > fork again... rinse repeat. > > (The sad part was seeing the tinycc developers YEARS LATER trying to solve a > problem I'd already solved which they'd left because they only now noticed the > issue existed. That happened multiple times...) > > Anyway, I've since come to the conclusion that writing a C compiler from > scratch > (with basically no optimizer, tinycc wasn't even preserving registers between > operations, it was "load from stack, perform operation, write to stack" even > when two consecutive operations used the same slot, that's why the resulting > executable was 1/3 the speed of other compilers) isn't all that hard, > especially > if I ever found time to read: > > https://norasandler.com/2017/11/29/Write-a-Compiler.html > > Which is a book now: > > https://nostarch.com/writing-c-compiler > > A lot of stuff like strace and readelf were originally in the "belongs in qcc" > bucket. These days, "cc/as/ld" are probably in the toybox post-1.0 command > list > bucket: > > http://lists.landley.net/pipermail/toybox-landley.net/2020-July/011898.html > > > I mostly agree with this, What confuses me is that these commands are some > > of > > the fist items in a list of "Packages toybox plans to provide complete-ish > > replacements for" on the roadmap. > > Commit f4c9a32a1116 from 2020. "Plans" is a strong word there, "could" is > probably better. The bc one was because we had bc, which smelled a lot fresher > in 2018 and it does in 2023, and it seemed easier to clean up a 7000 line pile > of spaghetti than trying to get a small existing working patch into > linux-kernel > through a community that was doing: > > https://github.com/torvalds/linux/commit/ddbd2b7ad99a > > https://github.com/torvalds/linux/commit/8a104f8b5867 > > https://www.cnbc.com/2018/09/17/linux-creator-linus-torvalds-takes-time-off-apologizes-for-behavior.html > > https://en.wikipedia.org/wiki/Identified_patient > > > But when looking at the status page, none of the commands have been > > mentioned as things we actually want, > > I am 99.999% sure these are commands we don't want to be in the project, > > but I thought it would be better to > > ask anyways because those commands were put in that list for a reason. > > Keep in mind what I want and what "the community" wants has an air gap. I > never > wanted selinux support, but here we are. If you come to me with a compelling > case to add dc, I can add dc. If it turns out > https://landley.net/bin/mkroot/0.8.9/linux-patches/0004-Replace-timeconst.bc-with-mktimeconst.c.patch > is not a sufficient strategy for handling bc (I.E. it actually has a > nontrivial > second user), I can clean it up an promote it. (Or write a new one if that's > easier.) > > > >> Do you have an actual user for dc? > > > > When I picked this command to write, I was going off the assumption that, > > because it was the roadmap, Somebody cared enough about it to put it there. > > And therefore would be the "user" who would benefit by it being implemented. > > Alas, no. The logic was "bc got inflicted on linux-kernel in 2013, bc got > submitted to toybox and is in pending albeit promoting it would be nontrivial, > the patch to remove bc from the kernel again is small and simple but political > and removing the FIRST round of this nonsense took 5 years of persistence... > if > we DO replace bc, the package that provides it to LFS and friends also > includes/provides dc, which is posix..." > > 95% likely yocto has dc because it installed the gnu bc package which also > includes dc. > > Sigh. Busybox has a dc, doesn't it? Yup. Sigh. If it was in posix I'd probably > just add it even if it's useless... > > > I could go on about how it has square roots when expr and shell $((math)) > > doesn't, > > or how the RPN stack layout of it _could_ be used to parse the numerical > > output > > of commands easier when combined with external scripts. > > $ echo $(($(seq 1 30 | xargs | tr ' ' +))) > 465 > $ echo $(($(printf '(%s+%s-%s)/%s' $(seq 4 -1 1)))) > 5 > > > But those are justifications, > > not actual use cases. I'd be lying if I said it wasn't mainly a "It's in > > the roadmap and busybox" thing. > > I still don't understand why bash $((math)) doesn't do floating point. You'd > think it would. Did I already ask Chet about this? (He'd almost certainly say > "for historical reasons" and be right.) I also don't understand why expr > doesn't > do floating point, that would give it a reason to exist outside of integer > $((math)), but again... historical. > > "This handles floating point" is a strong positive for dc. "This uses reverse > polish notation" seems like it would eliminates it for anyone except > electrical > engineers of a certain age, or people who have implemented their own infix > calculation engine, but maybe I'm wrong? (Do the younger generations have more > enthusiasm for reverse polish notation than I've been led to believe?) > > Heck, a year and change ago I got an rpn calculator from a hardware engineer, > which I shoved into a copy of hello.c so it's vaguely toybox command shaped. > (Attached.) But I didn't go farther because... reverse polish notation. In > 2023. > He didn't try to make dc, he'd just whipped up a quick command line tool for > his > own use. > > I note that toybox still supports the legacy $[math] bash syntax that got > replaced with $((math)) when posix caught up with the group, and I've been > vaguely tempted to try to make $[math] use floating point. I should ask Chet > how > terrible an idea that is... > > Rob_______________________________________________ > Toybox mailing list > Toybox@lists.landley.net > http://lists.landley.net/listinfo.cgi/toybox-landley.net _______________________________________________ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
