On Tue, Dec 30, 2014 at 11:22 AM, Rob Landley <[email protected]> wrote:
> On 12/30/14 00:24, enh wrote:
>> On Sat, Dec 27, 2014 at 5:02 PM, Rob Landley <[email protected]> wrote:
>>> The problem with BSD-style licenses is that there are a lot of them (2
>>> clause BSD, 3 clause BSD, 4 clause BSD, ISC, MIT, Apache, and so on)
>>> that all try to do the same thing but all of them say "you must copy
>>> this specific wording into your derived work", so if you combine code
>>> from two sources under different BSD variants you wind up concatenating
>>> the licenses, and this can get epically silly (the kindle paperwhite's
>>> about->licenses thing is over 300 pages of concatenated license
>>> boilerplate.)
>>
>> tell me about it.
>> https://android.googlesource.com/platform/bionic/+/master/libc/NOTICE
>
> I gave a talk about this topic at Ohio LinuxFest last year:
>
> https://archive.org/download/OhioLinuxfest2013/24-Rob_Landley-The_Rise_and_Fall_of_Copyleft.mp3
>
> Public domain licenses collapse together. Whether it's unlicense.org or
> http://creativecommons.org/about/cc0 or my "zero clause bsd" or
> https://android.googlesource.com/platform/external/dropbear/+/froyo/libtomcrypt/LICENSE
> the result is the same: you can take code and use it and your result can
> be under _one_ easily understandable license.
>
> None of the BSD variants actually accomplish this, they're so busy
> protecting themselves they develop an autoimmune problem.
>
>>> If, for auditing reasons, you don't want to use toybox's sha1sum but
>>> instead want to use an openssl derived version that shares code with
>>> other sha1sum instances you've already cleared and are using elsewhere,
>>> then that's what makes sense for your deployment. (If you grow to trust
>>> toybox's version later, you can swich to it then after everybody else
>>> has looked at it longer.)
>>
>> yeah, my question really is whether you want me to send patches like
>> that to the list, or just keep them downstream.
>
> I'm all for seeing patches on the list. Even if I don't wind up merging
> them it means there's an archive of them.
>
> Speaking of which, I just got reminded on twitter that there _is_ a
> gmane archive of toybox already, so I added a link to the web page:
>
> http://news.gmane.org/gmane.linux.toybox
>
> Their web view screws up patches (replacing @ with <at>) so I can't cut
> and paste 'em, but I'll live.
>
> (One advantage of two space indents instead of tab characters: applying
> patches with cut and paste gets much easier and I almost never have to
> do "mixed tab and space" policing because you need 4 levels deep
> indenting before you _could_ use a tab with default tabstops...)
>
>>> As for the md5/sha1/sha256/sha3, they're easy to test (their failures
>>> tend to be really obvious), and the two I implemented are inherently
>>> timing invariant and don't have obvious sidechannel attacks. And I _can_
>>> find existing public domain impelmentations of these to start from, such as:
>>>
>>> http://cpansearch.perl.org/src/BJOERN/Compress-Deflate7-1.0/7zip/C/Sha256.c
>>>
>>> So adding the other hashing functions to toybox makes sense to me,
>>> especially since I need them for a traditional /etc/shadow login.c. (I
>>> need to research android's user database and how to access it.)
>>
>> i can save you some time there: there isn't one. bionic's getpwnam and
>> friends will do the right thing, though, so toybox's id works fine.
>> (the patch i sent you fixes bugs that affect id on the desktop too,
>> nothing Android-specific.)
>
> Ok. I note that lib/login.c was an external contribution, and I was
> uncomfortable with it precisely _because_ I think we've got to delegate
> this to libc and punt in the case of android.
at the moment, pw_passwd is always NULL on Android. if it makes your
life easier for it to point to "*" or whatever, let me know, but one
problem we have in places like this -- and one reason i haven't even
bothered with <shadow.h> or <utmpx.h> -- is that in some ways code
that tries to use this stuff is better off if it just doesn't build,
because at least then the author/builder knows they need to sit down
and think about what they're trying to do and what, if anything, that
means on Android.
>>> Um, issue to be aware of: the subdirectories are just a developer
>>> convenience, the command namespace is actually flat. So if you have a
>>> NEWTOY(sha1sum) in toys/lsb and another NEWTOY(sha1sum) in toys/android,
>>> the build will break when it hits the duplicate command name.
>>>
>>> (Actually since you're not using our build infrastructure you can
>>> probably just ignore that, and point your .mk files at the right .c
>>> files for what you're building... :)
>>
>> yes, but i think some of the scripts get confused, plus it's good for
>> me to be able to build and test the desktop version of toybox too if
>> i'm sending you patches. plus removing it locally means git will
>> complain loudly if something changes upstream, so i'll be able to keep
>> track of what's going on.
>
> I have a pending change that I'm trying to figure out how to do in a way
> that won't screw you up. (Or just a clean way in general, it's... awkward.)
>
> The FLAG_x macros are auto-generated from the strings, based on the
> current configuration. The option string that gets parsed has USE_BLAH()
> macros that chop out disabled bits, the bit positions of each macro
> shift when options are configured out. (The bit namespace is always
> efficiently packed.)
>
> The flag parsing infrastructure handles this, comparing the
> "allyesconfig" and current .config versions of the strings at build time
> and generating "#define FLAG_x 0" entries for the disabled flags. this
> means if you do "if (toys.optflags & FLAG_x) blah();" for a disabled
> macro, it becomes & 0 which is always zero due to simple constant
> propogtation (required by c99!) and dead code elimination (which even
> Turbo C for dos got right) kicks in to remove the unused code. (It's
> always syntax checked at compile time so you don't have config-dependent
> build breaks, but it drops out of the binary when not in use.)
>
> Problem: some commands have shared infrastructure outside of lib/*.c,
> and some of them (mv/cp, kill/killall5, and lots of others) have
> overlapping option ranges where _both_ command want to support these
> options, using common code.
>
> What I _used_ to do was make the MV config symbol "depends on CP" and
> then I had the common infrastructure work in CP's flag context. But that
> screwed up the standalone builds ("scripts/single.sh COMMAND" which is
> also used by scripts/test.sh to test individual commands. ("make tests"
> builds the current toybox .config and tests the multiplexer binary, but
> the individual tests build standalone).
>
> The standalone builds do not_ have any other commands enabled. If you
> give them NAME they enable CONFIG_NAME and CONFIG_NAME_* (and a
> selection of CONFIG_TOYBOX_* symbols that should be less handcoded than
> it currently is). So if you "scripts/test.sh mv" it builds a broken mv
> that improperly handles the options shared with cp.
>
> The _fix_ to all this is to redo the option parsing infrastructure to
> leave gaps for the zeroed FLAG macros, and have a #define FORCE_FLAGS I
> can set when doing "#define FOR_COMMAND\n#include <toys.h>" that will
> enable the flags for _all_ options in this command, even the currently
> disabled ones.
>
> The problem is, those option strings I mentiond above with the USE()
> macros in them still drop out the disabled options, so when the
> lib/args.c infrastructure parses them it gets the sparse namespace and
> now the bit positions don't line up.
>
> So I have to have scripts/mkflags.c create a _second_ option string,
> based on the current configuration, with a marker character (I'm using
> ascii 1, I.E. "\001") to show the gaps where it needs to skip a bit. (I
> can't use the original string verbatim because I still want "cp -l" to
> reject the option and give a help message when you build it in posix
> options only mode (without CONFIG_CP_MORE).) This involves tweaking both
> mkflags.c and lib/args.c to know about the new signaling, but eh: not a
> big deal.
>
> Next problem: a couple months back I sped up rebuilds by _not_ have
> every build be a build all. and one of the ways I did that was by
> figuring out which generated/headers.h files wouldn't change based on
> config, and not rebuilding them. Both generated/newtoys.h and
> generated/oldtoys.h are config invariant, and that's where these option
> strings live.
>
> So I need to have the cooked option string data live in a _third_ place
> (the logical place is generated/flags.h which is the file
> scripts/mkflags.c is already generating), out new macros with a new
> prefix, and change main.c to stick that data into toy_list[] instead of
> the strings in the actual generated/newtoys.h it's #including when it
> generates that list, and _this_ is officially awkward.
>
> Did I mention that automating things so they "just work" is REALLY HARD?
> I know projects develop scar tissue as they go along, but I've always
> tried to go back and spend the time to clean UP the design even if it
> required a large intrustive change. Having option strings occur three
> times (even in generated headers) is silly.
>
> But... all three of 'em serve a purpose. The generated/newtoys.h is the
> collection of raw data grepped out of toys/*/*.c and used as the master
> to generate the other two instances. The generated/oldtoys.h has to be
> created by C because I've never managed to get a macro to expand to a
> preprocessor directive that gets invoked, so I can't add #defines at
> compile time. (Well, I could have scripts/make.sh do something horrible
> on the gcc command line spamming -D, but that's not an improvement and
> would render both "make V=1" and generated/build.sh just about
> unintelligible.)
>
> And trying to redefine USE() macros to convert options strings to
> spacers? Even if the preprocessor has some kind of string processing
> facility (which it doesn't seem to), USE_BLAH("a(longopt)") is not
> trivial to parse.
>
> (Well... if I turned _all_ a-zA-Z into \001 the normal lib/args.c parser
> could drop 'em out... except that an option is "anything not parsed as a
> control characgter", I.E. the else case at the end of a big stack. There
> are commands that want "-9" (such as gzip), and 9 is a control character
> in the right context (ala "a#<9")...
>
> So yeah, adding yet MORE crap to generated/flags.h and rejiggling the
> option parsing infrastructure to take the flag lists from there and
> adding a GREAT BIG COMMENT to main.c because "populate toy_list[]"
> doesn't cut it anymore, and explain this in code.html on the website...
>
> So yeah, heads up. You probably care about this one.
>
> (When I do these things right nobody notices I've done _anything_. Which
> is how it should be, but it's still hard.)
while you're there, the one other thing i don't think you can
currently support is the +/- style used by things like lsof. though --
although my usual personal goal is "you shouldn't be able to tell the
difference" -- the world might be a better place if that kind of
nonsense were allowed to die.
>>> So much easier when there's a standards document to blame. (Of course I
>>> vetoed like 1/3 of the posix command list anyway. Nobody needs sccs in
>>> 2014.)
>>
>> (i initially included uuencode/uudecode until one of the other guys on
>> the team asked what they were. turned out only me and the second
>> oldest guy had even heard of them...)
>
> Fun piece of trivia: the guy who submitted them to toybox is the lead
> architect for the Qualcomm Hexagon processor. (As in the main hardware
> guy who designs the actual chip.)
huh. do you remember why? i don't see anything on the list except for
the fact that two people arrived with uuencode at the same time.
(probably the majority of Android devices right now have the Hexagon
DSP, so if this is something that's actually useful to those guys i
might want to put it back!)
>>>> it's a pity the debian popularity contest only has per-package data
>>>> (https://qa.debian.org/popcon.php?package=coreutils). if you ask
>>>> people they always tell you they use everything "all the time". even
>>>> if you broke it two releases ago and removed it one release ago.
>>>
>>> Have you read toybox's roadmap.html page? It may not meet your needs but
>>> at least I have _reason_ for listing the commands I did. :)
>>
>> one thing i was curious about was what busybox configurations tend to
>> get used in the wild. your roadmap implies that you looked at that,
>> but i didn't see where.
>
> After the FSF zealots flamed Tim Bird and Sony to a crisp:
>
> https://lwn.net/Articles/478308/
>
> The various corporations interested in toybox have been VERY quiet about
> it. They email me off-list and read me the Mission Impossible speech
> about the secretary denying all knowledge if we're caught and their
> involvment self destructing in 5 seconds if so.
i've had positive private feedback from an external entity already,
expressed in the form of disappointment that it wasn't in L :-) they
weren't very helpful either when i asked them which toys they
want/need. maybe they're assuming that as long as the system is
expecting toybox, they can just edit the .config themselves?
> I tried to correct the record in the comments, ala
> https://lwn.net/Articles/480382/ and https://lwn.net/Articles/480836/
> but it didn't help. The FSF is a group of political zealots bordering on
> a religion, the old line applies about how you can't use rational
> argument to talk someone out of a position they didn't arrive at
> rationally in the first place. The best I can do is try to prevent the
> next generation from following them down the rathole (hence the Ohio
> Linufest talk, which I really need to redo. I had 3 hours of material
> and a little under an hour to speak, and that was _after_ spending a
> week trying to edit it down.)
>
> So I've gotten a lot of off the record emails with very interesting
> data, and various people going "we really need this, can't say why" and
> I do my best with the information I've got to keep the project on track...
>
>>> Always happy to have another viewpoint to rejuggle the weightings...
>>
>> i'm assuming we (Android) will get some feedback eventually.
>
> Sorry, sorry. The mailing list archive being out of whack really screwed
> up my normal working style, I wash my email through gmail for the spam
> filtering, but that drops out all dupliciate messages so if I'm cc'd on
> a message I get _either_ the inbox copy _or_ the mailing list copy with
> the list-id tag, so neither folder has the complete conversation and if
> I try for threaded view all the threads are broken because the message
> they're replying to is in the other folder. Complaining to the gmail
> guys was a brick wall, so I started using the web archive to keep track
> of stuff.
>
> (Plus I've got a bunch of things like sed debugging and the above
> command line redesign that take a lot of careful study to get right, and
> when I pop my head up I have this giant backlog. (I still haven't dealt
> with Ashwini's giant pile from october.)
>
> Yesterday got eaten by kernel issues (they put PERL back in the 3.18
> kernel build and I had to rip it out again). I have high hopes for today. :)
i actually meant we'd get feedback from OEMs/developers about which
toybox commands they would like to see and/or bugs that cause them
pain. but i have been holding off committing changes in Android until
i get feedback here. i can send out another status mail if that would
be helpful.
>>> When I get my darn server reinstalled and get AOSP on it, I want to run
>>> the AOSP build with the toybox commands. (Aboriginal Linux is using an
>>> old version of linux from scratch as a bootstrapping test, but android's
>>> build needs more commands than that. And may use command line options
>>> that toybox doesn't implement yet. I know _you_ aren't trying to get
>>> android self-hosting anytime soon, but I still am. :)
>>
>> yeah, my builds are more than slow enough even on the fastest desktop
>> hardware :-)
>
> This netbook hasn't even got the free disk space to try.
building for aarch64 or x86-64 pretty much doubles the space
requirements too (since for the time being at least most things need
to be multi-arch) :-(
>>>> i also haven't thought much about "in the binary" versus
>>>> "gets a symlink"; i suspect that the "too much" camp will be further
>>>> subdivided into those who're offended by the binary size and those
>>>> who're offended by the number of symlinks in /system/bin.
>>>
>>> I don't understand the distinction here? (Is your build making
>>> standalone binaries for the toybox commands ala scripts/single.sh? It
>>> didn't look like it was but I have to stare at makefiles a lot to beat
>>> any sense out of 'em...)
>>
>> no, there's one binary that contains n toys, and then m symbolic
>> links, where n != m. think of the ones that don't have links as my
>> level of "pending"ness :-) though it's ill-defined what not having a
>> link means. sometimes it's "i think we'll want this, but i haven't
>> checked it works for us yet", sometimes "i think this probably doesn't
>> make any sense on Android, but it's in the default set and i haven't
>> yet been convinced to kick it out".
>>
>> but my earlier point was that no one is likely to look too closely at
>> what's in the toybox binary, especially as long as it's roughly the
>> same size as the toolbox one used to be. but if they start stumbling
>> across symbolic links for things they think are a waste of space,
>> they're more likely to get on my case about it.
>
> Makes sense.
>
> Rob
_______________________________________________
Toybox mailing list
[email protected]
http://lists.landley.net/listinfo.cgi/toybox-landley.net
