Re: [Toybox] [landley/toybox] Help building toybox with the NDK/bionic (#43)

[enh]

On Fri, Dec 30, 2016 at 12:46 PM, Rob Landley  wrote:
> On 12/27/2016 02:49 PM, enh wrote:
>> On Tue, Dec 27, 2016 at 1:21 AM, Rob Landley  wrote:
>>> I'm trying to improve upon "Elliott will tell me if I screwed up the
>>> android build", which seems impolite somehow...
>>
>> although i'd rather triage Android build issues myself rather than
>> never hear of them because of dodgy workarounds committed upstream,
>> which is what happens in some other projects!
>
> True, but "works on musl and glibc, doesn't work on bionic" is the kind
> of thing I generally just poke you and make puppy eyes about anyway.
>
> I'm thinking more along the lines of "finding the sed problem earlier
> would have been nice".

wasn't that actually an LP32 problem?

>>> Long-term I'm hoping to make it easier to build system images. (I'm
>>> aware installing them on actual hardware with "verified boot" is its own
>>> can of worms, ala http://www.youtube.com/watch?v=OfWFmhucWlg but
>>> building and running under qemu should be easier than I currently find it.)
>>
>> or you can just flash your own AOSP -eng build to a Nexus device, "adb
>> root" and do what you like.
>
> I thought they'd renamed it Pixel?

renaming the teams/branches internally and fixing everyone's muscle
memory is a lot harder than changing the marketing label. but, yes,
for the next few years any time i say "Nexus" assume i said
"Nexus/Pixel".

> I've got a Nexus 5. It maxed out at Android M and you stopped shipping
> security updates for it in October. I'm reluctant to buy another device
> that may be discontinued before I get it home.

Nexus 5 was 32-bit only too, so any testing on that would be limited
to the ABI that's -- for platform code -- pretty much dead.

> (The whole "two year support horizon" combined with the implosion of
> cyanogenmod kinda undermines my whole "if you're selling a billion
> phones a year then there are a billion _old_ phones a year available for
> the next generation of developers to play with and learn on instead of
> just landfilling them" theory. Making it easier to build and install new
> system images might work around that, but there's a lot of work to do to
> get from here to there. I need to figure out how to carve out more time
> to work on this again. Endless travel between time zones got me out of
> the "set the alarm at 5am and get a couple hours in before work" habit...)
>
>> qemu is... pretty awful.
>
> It's slow, but using it I can script automated regression tests for a
> half-dozen architectures from a coffee shop without having to plug extra
> hardware into my netbook.
>
> The best part is since an emulator image starts in a known state, the
> mount or ifconfig tests failing to clean up after themselves (or the ps
> tests --bind mounting test data over /proc and getting interrupted
> halfway through so it didn't unmount it) don't screw up the next round
> of tests in a new emulator instance.
>
>> you _can_ use host bionic to run stuff on a regular Linux box, but
>> none of the "interesting" stuff (the stuff in toys/android/) will work
>> there anyway.
>
> Hence booting an android kernel under qemu. But that's something I need
> to work up to...
>
>>> I was thinking that the NDK is the thing changing right now, so it would
>>> get a new symbol to indicate the build environment it provides differs
>>> from the existing AOSP. But you know your build system design better
>>> than I do.
>>
>> it's the other way round: you tell the NDK what API level you're
>> targeting. (because you might legitimately want to say "don't let me
>> use anything that wasn't available before ICS, because i want to run
>> on everything >= ICS".)
>
> Ah, selectable API levels. Hmmm.
>
> Yay patch. I look forward to it.

 sent.
>>>
>>> $ git am 0001-Switch-to-bionic-sys-system_properties.h.patch
>>> Applying: Switch to bionic .
>>> error: patch failed: toys/android/getprop.c:17
>>> error: toys/android/getprop.c: patch does not apply
>>> error: patch failed: toys/android/setprop.c:17
>>> error: toys/android/setprop.c: patch does not apply
>>> Patch failed at 0001 Switch to bionic .
>>>
>>> The last commit toy toys/android that wasn't from you was on August 4th?
>>> Hmmm... I removed the #ifdef __ANDROID__ back in commit 5b493dc48db0 in
>>> 2015. (And then fixed a screwup in in e5fb6a28ffb0 the next day.)
>>>
>>> We appear to be out of sync? (Speaking of "Elliott will tell me if I
>>> screwed up the android build..." :)
>>
>> that kind of rings a bell, that you did something years ago that
>> caused a merge conflict in toys/android/ and i just skipped it. it
>> used to be useful to me that i could build the Android configuration
>> on non-Android, but i gave up fighting you and now i have the opposite
>> problem to you: i don't/can't test the Android toybox build against
>> glibc.
>
> I'm attempting to fix this, but being able to build an android version
> myself is a prerequisite to me being able 

Re: [Toybox] Stat %Z - What are valid values?

[Rob Landley]

On 12/30/2016 01:39 PM, enh wrote:
> 1928-04-26 17:31:55.746667836 18446744072394174731

Really we can partially blame posix here for not specifying whether
time_t is signed or unsigned. (If it's unsigned it can't represent times
before January 1, 1970, as the code on the left is doing. And there
_are_ times before then. So I think we have to treat it as signed and go
"32 bit timestamps gotta go away before 2038".)

Linus' opinion on unsigned time_t is here (spoiler: it's "no"):

  https://lkml.org/lkml/2011/8/31/246

> two patches attached.
...
> the other
> removes void* casts unnecessary since POSIX 2008 and fixes the
> strftime buffer length argument.

Why did you remove the comment block in the second patch? (Is it wrong?)

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] Stat %Z - What are valid values?

[Rob Landley]

I have negotiated a 3 day weekend for the holidays! Starting today. Ok,
catching up...

On 12/30/2016 01:39 PM, enh wrote:
> time_t on 32-bit Android is 32 bits.
> 
> that particular value looks like a sign-extension of 0xAFBEADCE, which
> is still some time in 2063.

Or if it's signed, that's -1346458162 which would be... sometime in the
1930's? hmmm... "./date -D %s -d -1346458162" is failing under glibc,
and failing _differently_ under musl. (Wheee.)

/me goes down tangent rathole debugging why. (THIS is why I need to
finish mkroot and get a toybox test environment set up that can test
stuff requiring root access under qemu under a known root filesystem and
kernel config. Is my code broken, or does strptime not parse negative
numerical arguments for %s format?)

(Answer: musl doesn't implement %s at all, and glibc doesn't allow the
%s value it converts to be negative. And of course posix doesn't specify
signed OR unsigned for time_t at
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_types.h.html
because actually providing enough detail to implement things would be
_useful_.)

> two patches attached. one avoids sign extension for all calls to
> `out`, fixing %Z for systems with a signed 32-bit time_t.

The 0001-Avoid* patch only has the first hunk, which defines two macros
there are never used and renames a function but not its users.

>> This seems suspiciously large and I'm wondering what the valid range for
>> this value is.

I think we all do at this point.

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] [landley/toybox] Help building toybox with the NDK/bionic (#43)

[Rob Landley]

On 12/27/2016 02:49 PM, enh wrote:
> On Tue, Dec 27, 2016 at 1:21 AM, Rob Landley  wrote:
>> I'm trying to improve upon "Elliott will tell me if I screwed up the
>> android build", which seems impolite somehow...
> 
> although i'd rather triage Android build issues myself rather than
> never hear of them because of dodgy workarounds committed upstream,
> which is what happens in some other projects!

True, but "works on musl and glibc, doesn't work on bionic" is the kind
of thing I generally just poke you and make puppy eyes about anyway.

I'm thinking more along the lines of "finding the sed problem earlier
would have been nice".

>> Long-term I'm hoping to make it easier to build system images. (I'm
>> aware installing them on actual hardware with "verified boot" is its own
>> can of worms, ala http://www.youtube.com/watch?v=OfWFmhucWlg but
>> building and running under qemu should be easier than I currently find it.)
> 
> or you can just flash your own AOSP -eng build to a Nexus device, "adb
> root" and do what you like.

I thought they'd renamed it Pixel?

I've got a Nexus 5. It maxed out at Android M and you stopped shipping
security updates for it in October. I'm reluctant to buy another device
that may be discontinued before I get it home.

(The whole "two year support horizon" combined with the implosion of
cyanogenmod kinda undermines my whole "if you're selling a billion
phones a year then there are a billion _old_ phones a year available for
the next generation of developers to play with and learn on instead of
just landfilling them" theory. Making it easier to build and install new
system images might work around that, but there's a lot of work to do to
get from here to there. I need to figure out how to carve out more time
to work on this again. Endless travel between time zones got me out of
the "set the alarm at 5am and get a couple hours in before work" habit...)

> qemu is... pretty awful.

It's slow, but using it I can script automated regression tests for a
half-dozen architectures from a coffee shop without having to plug extra
hardware into my netbook.

The best part is since an emulator image starts in a known state, the
mount or ifconfig tests failing to clean up after themselves (or the ps
tests --bind mounting test data over /proc and getting interrupted
halfway through so it didn't unmount it) don't screw up the next round
of tests in a new emulator instance.

> you _can_ use host bionic to run stuff on a regular Linux box, but
> none of the "interesting" stuff (the stuff in toys/android/) will work
> there anyway.

Hence booting an android kernel under qemu. But that's something I need
to work up to...

>> I was thinking that the NDK is the thing changing right now, so it would
>> get a new symbol to indicate the build environment it provides differs
>> from the existing AOSP. But you know your build system design better
>> than I do.
> 
> it's the other way round: you tell the NDK what API level you're
> targeting. (because you might legitimately want to say "don't let me
> use anything that wasn't available before ICS, because i want to run
> on everything >= ICS".)

Ah, selectable API levels. Hmmm.

 Yay patch. I look forward to it.
>>>
>>> sent.
>>
>> $ git am 0001-Switch-to-bionic-sys-system_properties.h.patch
>> Applying: Switch to bionic .
>> error: patch failed: toys/android/getprop.c:17
>> error: toys/android/getprop.c: patch does not apply
>> error: patch failed: toys/android/setprop.c:17
>> error: toys/android/setprop.c: patch does not apply
>> Patch failed at 0001 Switch to bionic .
>>
>> The last commit toy toys/android that wasn't from you was on August 4th?
>> Hmmm... I removed the #ifdef __ANDROID__ back in commit 5b493dc48db0 in
>> 2015. (And then fixed a screwup in in e5fb6a28ffb0 the next day.)
>>
>> We appear to be out of sync? (Speaking of "Elliott will tell me if I
>> screwed up the android build..." :)
> 
> that kind of rings a bell, that you did something years ago that
> caused a merge conflict in toys/android/ and i just skipped it. it
> used to be useful to me that i could build the Android configuration
> on non-Android, but i gave up fighting you and now i have the opposite
> problem to you: i don't/can't test the Android toybox build against
> glibc.

I'm attempting to fix this, but being able to build an android version
myself is a prerequisite to me being able to make both work. :)

> new patch attached.
> 
> you still won't be able to build getprop.c with the NDK, though,
> because its use of libselinux is quite ingrained.

It's still an improvement. Thanks.

>> So a patch on this would cc: Rich Felker, David Woodhouse, David
>> Howells, James Bottomley, Doug Gilbert, and Christoph Hellwig. Plus
>> maybe Al Viro and Andrew Morton.
>>
>> But the first part would be coming up with the suggested patch, which
>> involves comparing musl's and bionic's version of these files (bonic's
>> at
>> 

Re: [Toybox] Stat %Z - What are valid values?

[enh]

time_t on 32-bit Android is 32 bits.

that particular value looks like a sign-extension of 0xAFBEADCE, which
is still some time in 2063. (so i'd assume this device's clock is set
wrong, and i'd assume -- since this is presumably a 32-bit device with
a signed 32-bit time_t -- that that's going to be causing other
problems too.)

https://android.googlesource.com/platform/external/toybox/+/android-6.0.1_r77/toys/other/stat.c
doesn't look to different from today.

with 64-bit toybox:

angler:/ # date 06022064
Mon Jun  2 00:00:00 GMT 2064
angler:/ # rm /data/local/tmp/empty
angler:/ # touch /data/local/tmp/empty
angler:/ # ls -l /data/local/tmp/empty
-rw-rw-rw- 1 root root 0 2064-06-02 00:00 /data/local/tmp/empty
angler:/ # stat /data/local/tmp/empty
  File: `/data/local/tmp/empty'
  Size: 0 Blocks: 0 IO Blocks: 512 regular empty file
Device: fd00h/64768d Inode: 3014659 Links: 1
Access: (666/-rw-rw-rw-) Uid: (0/root) Gid: (0/root)
Access: 2064-06-02 00:00:11.746667836
Modify: 2064-06-02 00:00:11.746667836
Change: 2064-06-02 00:00:11.746667836
angler:/ # stat -c "%z %Z" /data/local/tmp/empty
2064-06-02 00:00:11.746667836 2979590411
angler:/ # ^D

then with 32-bit toybox:

angler:/ # ls -l /data/local/tmp/empty
-rw-rw-rw- 1 root root 0 1928-04-26 17:31 /data/local/tmp/empty
angler:/ # stat /data/local/tmp/empty
  File: `/data/local/tmp/empty'
  Size: 0 Blocks: 0 IO Blocks: 512 regular empty file
Device: fd00h/64768d Inode: 3014659 Links: 1
Access: (666/-rw-rw-rw-) Uid: (0/root) Gid: (0/root)
Access: 1928-04-26 17:31:55.746667836
Modify: 1928-04-26 17:31:55.746667836
Change: 1928-04-26 17:31:55.746667836
angler:/ # stat -c "%z %Z" /data/local/tmp/empty
1928-04-26 17:31:55.746667836 18446744072394174731

two patches attached. one avoids sign extension for all calls to
`out`, fixing %Z for systems with a signed 32-bit time_t. the other
removes void* casts unnecessary since POSIX 2008 and fixes the
strftime buffer length argument.

%z is still broken for systems with signed 32-bit time_t, but i think
that's just inherent on such systems.

On Fri, Dec 30, 2016 at 9:28 AM, darken  wrote:
> I've seen a value of "18446744072363093454" for stat %Z (seconds since
> epoch), for some files on a users device (Android 6.01).
>
> This seems suspiciously large and I'm wondering what the valid range for
> this value is.
> What range is valid for the filesystem to return and what value range can
> toybox handle?
>
> ~Matthias
>
> ___
> Toybox mailing list
> Toybox@lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net
>



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
From be5dd0c57df9d35bff7671c897e0b2dd384bdb84 Mon Sep 17 00:00:00 2001
From: Elliott Hughes 
Date: Fri, 30 Dec 2016 11:24:25 -0800
Subject: [PATCH] Avoid sign extension in stat.c.

---
 toys/other/stat.c | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/toys/other/stat.c b/toys/other/stat.c
index 9d64579..63dfba8 100644
--- a/toys/other/stat.c
+++ b/toys/other/stat.c
@@ -50,9 +50,19 @@ GLOBALS(
   int patlen;
 )
 
-// Force numeric output to long long instead of manually typecasting everything
-// and safely parse length prefix
-static void out(char c, long long val)
+// Force numeric output to unsigned long long instead of manually typecasting
+// everything. Takes unsigned because no struct stat field can be negative,
+// and in combination with the add_unsigned macro this lets us avoid
+// accidental sign extension of signed fields (time_t on LP32, say).
+// We also handle length prefixes here rather than repeating that everywhere.
+#define out(out_c, out_val) uout(out_c, add_unsigned(out_val))
+#define add_unsigned(x) \
+  (sizeof(x) == sizeof(char) ? (unsigned char)(x) : \
+   sizeof(x) == sizeof(short) ? (unsigned short)(x) : \
+   sizeof(x) == sizeof(int) ? (unsigned int)(x) : \
+   sizeof(x) == sizeof(long) ? (unsigned long)(x) : \
+   (unsigned long long)(x))
+static void uout(char c, unsigned long long val)
 {
   sprintf(toybuf, "%.*sll%c", TT.patlen, TT.pattern, c);
   printf(toybuf, val);
-- 
2.8.0.rc3.226.g39d4020

From 1c4a7a8ac7ff68e07df91f5822dd8145229e4bf9 Mon Sep 17 00:00:00 2001
From: Elliott Hughes 
Date: Fri, 30 Dec 2016 11:19:08 -0800
Subject: [PATCH] Remove unnecessary casts in stat.c, fix a claimed buffer
 length.

POSIX does have a name for the struct timespec in struct stat.
---
 toys/other/stat.c | 12 
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/toys/other/stat.c b/toys/other/stat.c
index b998a0e..9d64579 100644
--- a/toys/other/stat.c
+++ b/toys/other/stat.c
@@ -65,14 +65,10 @@ static void strout(char *val)
   printf(toybuf, val);
 }
 
-// Note: the atime, mtime, and ctime fields in struct stat are the start
-// of embedded struct 

[Toybox] Stat %Z - What are valid values?

[darken]

I've seen a value of "18446744072363093454" for stat %Z (seconds since
epoch), for some files on a users device (Android 6.01).

This seems suspiciously large and I'm wondering what the valid range for
this value is.
What range is valid for the filesystem to return and what value range can
toybox handle?

~Matthias
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net