[Toybox] [PATCH] file(1): add ogg, TrueType font, LLVM bitcode, PEM certs, and PE executables.

[enh]

More of the files encountered on the Android system image (plus PE
executables, which we build but which shouldn't actually make it to the
system image!).
---
 toys/posix/file.c | 43 +++
 1 file changed, 43 insertions(+)
From 6fd1cf99b2bfff67f756d95b060074a5af289396 Mon Sep 17 00:00:00 2001
From: Elliott Hughes 
Date: Wed, 1 Nov 2017 11:09:42 -0700
Subject: [PATCH] file(1): add ogg, TrueType font, LLVM bitcode, PEM certs, and
 PE executables.

More of the files encountered on the Android system image (plus PE
executables, which we build but which shouldn't actually make it to the
system image!).
---
 toys/posix/file.c | 43 +++
 1 file changed, 43 insertions(+)

diff --git a/toys/posix/file.c b/toys/posix/file.c
index 2f1e95b..5e5c15b 100644
--- a/toys/posix/file.c
+++ b/toys/posix/file.c
@@ -270,6 +270,49 @@ static void do_regular_file(int fd, char *name, struct stat *sb)
 else what = NULL;
 if (what) xprintf("%s\n", what);
 else xprintf("(bad type %d)\n", s[9]);
+  } else if (len>36 && !memcmp(s, "OggS\x00\x02", 6)) {
+xprintf("Ogg data");
+// https://wiki.xiph.org/MIMETypesCodecs
+if (!memcmp(s+28, "CELT", 8)) xprintf(", celt audio");
+if (!memcmp(s+28, "CMML", 8)) xprintf(", cmml text");
+if (!memcmp(s+28, "BBCD\0", 5)) xprintf(", dirac video");
+if (!memcmp(s+28, "\177FLAC", 5)) xprintf(", flac audio");
+if (!memcmp(s+28, "\x8bJNG\r\n\x1a\n", 8)) xprintf(", jng video");
+if (!memcmp(s+28, "\x80kate\0\0\0", 8)) xprintf(", kate text");
+if (!memcmp(s+28, "OggMIDI\0", 8)) xprintf(", midi text");
+if (!memcmp(s+28, "\x8aMNG\r\n\x1a\n", 8)) xprintf(", mng video");
+if (!memcmp(s+28, "OpusHead", 8)) xprintf(", opus audio");
+if (!memcmp(s+28, "PCM ", 8)) xprintf(", pcm audio");
+if (!memcmp(s+28, "\x89PNG\r\n\x1a\n", 8)) xprintf(", png video");
+if (!memcmp(s+28, "Speex   ", 8)) xprintf(", speex audio");
+if (!memcmp(s+28, "\x80theora", 7)) xprintf(", theora video");
+if (!memcmp(s+28, "\x01vorbis", 7)) xprintf(", vorbis audio");
+if (!memcmp(s+28, "YUV4MPEG", 8)) xprintf(", yuv4mpeg video");
+xputc('\n');
+  } else if (len>12 && !memcmp(s, "\x00\x01\x00\x00", 4)) {
+xputs("TrueType font");
+  } else if (len>12 && !memcmp(s, "ttcf\x00", 5)) {
+xprintf("TrueType font collection, version %d, %d fonts\n",
+(int)peek_be(s+4, 2), (int)peek_be(s+8, 4));
+  } else if (len>4 && !memcmp(s, "BC\xc0\xde", 4)) {
+xputs("LLVM IR bitcode");
+  } else if (strstart(, "-BEGIN CERTIFICATE-")) {
+xputs("PEM certificate");
+
+  // https://msdn.microsoft.com/en-us/library/windows/desktop/ms680547(v=vs.85).aspx
+  } else if (len>0x70 && !memcmp(s, "MZ", 2) &&
+  (magic=peek_le(s+0x3c,4))70) {
+  char *types[] = {0, "native", "GUI", "console", "OS/2", "driver", "CE",
+  "EFI", "EFI boot", "EFI runtime", "EFI ROM", "XBOX", 0, "boot"};
+  int type = peek_le(s+magic+92, 2);
+  char *name = (type>0 && type

Re: [Toybox] Yes, I'm aware dreamhost has deleted the mailing list web archive again.

[enh]

https://github.com/landley/toybox/commit/2d893a4077c12732238f5a9fc9c31fda8bcc3ed9
fixes the cut and chmod tests for me. thanks!

On Tue, Oct 17, 2017 at 11:19 AM, enh  wrote:
> On Wed, Oct 11, 2017 at 1:58 PM, Rob Landley  wrote:
>>
>> On 10/11/2017 12:36 PM, enh wrote:
>>> On Sep 28, 2017 22:47, "Rob Landley" wrote:
>>> >
>>> > It's an annual event with them. They're deeply incompetent.
>>>
>>> Seems like it's not being updated even now... "*Note:*The archive search
>>> index was last rebuilt at Friday, 29 Sep 2017 19:07:57 PDT. Any postings
>>> after that will not be found by a search. Index rebuild is usally done
>>> once every 24 hours for this list."
>>
>> Did I say deeply incompetent?
>>
>>   https://twitter.com/landley/status/917549056544604160
>>
>> I meant deeply, _deeply_ incompetent.
>>
>> Seriously, they've done this at least once a year since something like
>> 2010, usually leaving another hole in the mailing list archive when they
>> (eventually) restore from a stale backup and think that fixes it.
>>
>> I used to collect my interactions with them:
>>
>>   http://landley.net/dreamhost.txt
>>   http://landley.net/dreamhost2.txt
>>
>> But haven't in years because there's no point. They never learn because
>> I'm never talking to the same people. Even in the same thread, each
>> message is replied to by whichever junior intern is on duty right then,
>> with no institutional memory at all.
>>
>> They're decent about the actual web hosting part (it's a static website
>> I have fully backed up and rsync, it's hard to screw that up and "fire
>> up an empty container and restore the data from a backup" generally _is_
>> the right answer to something going wrong there). They're cheap, they
>> don't meter bandwidth or have a hard storage limit, and I do _not_ have
>> time to try to administer my own servers these days (work has gotten
>> _exciting_ again, I'm trying to get this release out before I get
>> kidnapped by j-core stuff for a few months)...
>>
>> But it would be really really nice if I could run a mailman instance on
>> the container I ssh into instead of having them run lists.landley.net as
>> a shared "service" for me, which I have to administer through a web page
>> instead of the command line.
>>
>> Right, back to trying to test and checkin cut.c. The new version of
>> which can do this!
>
> the new cut seems to have broken the chmod tests (on systems that
> always use the toybox cut).
>
> 4 of the cut tests seem to be failing too.
>
>>   $ awk '{print $2" "$7" "$5}' LICENSE | head -n 5
>>   (C) Landley by
>>
>>   to distribute modify,
>>   with hereby fee
>>
>>   l$ ./cut -DF 2,7,5 LICENSE | head -n 5
>>   (C) Landley by
>>
>>   to distribute modify,
>>   with hereby fee
>>
>> (It's not awk, but it's better than nothing.)
>>
>> Rob
>
>
>
> --
> Elliott Hughes - http://who/enh - http://jessies.org/~enh/
> Android native code/tools questions? Mail me/drop by/add me as a reviewer.



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] ps -T

[Rob Landley]

On 10/19/2017 06:13 PM, enh wrote
>> On 09/20/2017 05:08 PM, enh wrote:
>>> ps -T doesn't really work if you have any filters. so ps -AT is fine,
>>> but ps -p  -T only shows the main thread.
>>
>> Alas, I don't personally use threads much so basically never test this.
>>
>>> why? because slots[SLOT_pid] is "wrong" in shared_match_process (where
>>> by wrong i mean "is the tid").
>>>
>>> why? because toybox reads (say) /proc/147047/task/147058/stat and sees
>>>
>>> 147058 (CompositorTileW) S 31782 6249 6249 0 -1 1077952576 4 0 0 0 0 0
>>> 0 0 20 0 11 0 1211910244 928649216 35602 18446744073709551615
>>> 94558515900416 94558627572512 140720560858928 140506510343072
>>> 140506833892356 0 0 4098 1073827581 1 0 0 -1 31 0 0 0 0 0
>>> 94558627579744 94558633602584 9455861888 140720560866826
>>> 140720560866928 140720560866928 140720560869342 0
>>>
>>> and copies 147058 into SLOT_pid because that code no longer knows the real 
>>> pid.

I added -H to iotop, fixed the off-by-one error in screen width
truncation, set the screen width to 72, and ran "top -H -O TID,SHR" and
cursored over to the SHR column and:

  PID USER   TID [SHR]%CPU  %MEM TIME+ THREAD  PROCE
 1865 landley   1865  84M 98.1  14.8 128:47.94 thunderbird thun+
 1865 landley   1895  84M  1.8  14.8  41:53.81 SoftwareVsyncTh thun+
 1865 landley723  84M  0.0  14.8   0:00.00 StreamT~s #3048 thun+
 1865 landley  30482  84M  0.0  14.8   2:14.82 DOM Worker  thun+
 1865 landley   9488  84M  0.0  14.8   7:16.20 DOM Worker  thun+
 1865 landley  16082  84M  0.0  14.8  15:33.85 DOM Worker  thun+
 1865 landley  15523  84M  0.0  14.8   0:00.49 DOM Worker  thun+
 1865 landley  12086  84M  0.0  14.8  17:24.69 DOM Worker  thun+
 1865 landley   3838  84M  0.0  14.8  18:14.68 DOM Worker  thun+
 1865 landley   2254  84M  0.0  14.8  18:26.28 DOM Worker  thun+
 1865 landley  30746  84M  0.0  14.8  18:30.12 DOM Worker  thun+

An they all seem to think they're PID 1865, but each knows its TID?

$ ls /proc/1865/task/
10228  1865  1875  1880  1887  1893  1899  1911  1934   30482  5471
12086  1871  1876  1881  1888  1894  1903  1912  19926  30746  6316
14369  1872  1877  1882  1889  1895  1904  1921  22017  32148  9488
15523  1873  1878  1885  1891  1897  1906  1927  2254   3838   9955
16082  1874  1879  1886  1892  1898  1908  1933  2770   5469

Which seems reasonable? It's got PID, it's got TID, what do I need to
fix here?

(Aside: thunderbird really, really, really doesn't like a local
linux-kernel folder with 500k messages in it. Or a pop3 inbox going back
to 2013, which is the last time I split it. But I think what it's mad
about right now is the BUG() I hit in the vfat code which I had to fsck
away to do some work, and thus the vfat maintainer couldn't reproduce
it. I've meant to reboot ever since, it happened about 5 times before I
fixed it, emergency-zapping a filesystem each time, and the memory
menagement on this box has gone all wonky since then. I've been meaning
to reboot to replace the keyboard anyway, but 8 desktops full of windows
full of tabs takes a while to unwind...)

>>> not sure how best to fix this.
>>
>> Hmmm... Reasonably straightforward to fix,

Not necessarily straightfowrward to reproduce.

$ ps -AT
PID   TID
32667 32667 ?00:26:22 chromium-browse
32667 32668 ?00:00:00 TaskSchedulerSe
32667 32669 ?00:00:20 Chrome_ChildIOT
32667 32670 ?00:00:00 GpuMemoryThread

They're different?

$ ls /proc/32667/task/
10087  32668  32670  32672  32674  5747  5754
32667  32669  32671  32673  32697  5748

And they're reasonable?

>> but my tree has local c
>> changes in ps.c. Looks like I'm adding -m to show maximum number of
>> lines (somebody asked, it's easy enough.)
> 
> i said "no" to the single internal request we had for that when we
> switched from traditional Android top to toybox top. easy, yes, but
> not obviously useful. the original Android top only had batch mode, so
> it was a bit more useful then. but "first N" isn't an obviously
> meaningful heuristic. "field X no lower than Y" would be more
> convincing. but that's no longer as easy :-)

I can yank it again. If you wanna design a filter syntax I can probably
implement something.

I've just started poking at bc ala shell $((blah)) math syntax, and I'm
likely to make a function or two that lets you substitute in variables
(via string substitution) and do math on the result.

That said... that's not the syntax we've got in find, or in test. And ps
has:

  -o  FIELDs instead of defaults, each with optional :size and =title

I could add XXX to that, I suppose? Might take a bit of
fiddling to make room.

That said, you still couldn't implement -m with a syntax like that.
Maximum number of fields to display isn't a -o field. :)

>> And -H to iotop (which is
>> where I left off; need to come up with a test for this and haven't got
>> one. Is chrome threads or processes?