Re: [Toybox] musl intentionally broke chrt

2017-09-02 Thread Rob Landley 
On 09/02/2017 04:08 PM, Owen Shepherd wrote:
> Per POSIX,  is in ths [PS] (Process Scheduling) option. You can
> identify whether [PS] is implemented at compile time, by the value of
> _POSIX_PRIORITY_SCHEDULING defined in .

I've found feature test macros to be a minefield for _years_, back to about:

http://lists.landley.net/pipermail/toybox-landley.net/2012-March/005251.html

But if you're saying "this is a compile-time probe you can use to work
around musl's breakage, as long as you're willing to first confirm that
all the other libc's out there _do_ provide this macro..." I'll put it
on the todo list.

But really, leaving musl broken here seems like what they explicitly
_want_. Maybe in chrt.c I can do an:

#ifndef _POSIX_THINGY
#warning musl commit  broke the system calls this depends on
#endif

And call that a solution.

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] musl intentionally broke chrt

2017-09-02 Thread Owen Shepherd 
Per POSIX,  is in ths [PS] (Process Scheduling) option. You can
identify whether [PS] is implemented at compile time, by the value of
_POSIX_PRIORITY_SCHEDULING defined in . This can be one of

* Undefined or -1, functionality never supported
* 0, functionally *maybe* supported (you can enquire with
sysconf(_SC_PRIORITY_SCHEDULING) at runtime). You can compile against it,
but it might not work.
* 200809L, supported per POSIX 2008 (Or potentially a different value for a
different version of POSIX)

(The sysconf call can return either -1 or 200809L)

musl doesn't define _POSIX_PRIORITY_SCHEDULING, so per POSIX is declaring
that it does not support that function grouping.

On Wed, Aug 30, 2017 at 9:42 AM Rob Landley  wrote:

> On 08/29/2017 10:32 AM, enh wrote:
> > On Tue, Aug 29, 2017 at 8:32 AM, enh  wrote:
> >> can't you just infer musl from the relatively small
> >
> > ...number of c libraries out there...
>
> It would work, but I'm uncomfortable with it.
>
> People keep threatening to port toybox to to BSD and MacOS, there is an
> active uClibc fork buildroot's using, it's posible newlib, klibc, or
> dietlibc will grow legs someday, far down on my todo list is maybe
> porting toybox to xv6, what library is fuchsia using, musl itself didn't
> publicly exist ~5 years ago...
>
> The posix API is insufficient so I'm using Linux as my API, but
> freebsd's Linux emulation layer is a viable target too, and I wanted the
> rest of it to be as generic as I could with target-specific stuff hidden
> in portability.h. That means C library, compiler, processor architecture...
>
> I'm not sure that's _achievable_ with musl refusing to identify itself
> because its' experience is universal and everyone else is wrong...
>
> Sigh. If I'm going to have the "else case" apply workarounds, I might as
> well make the workarounds apply to everybody. Calling the syscalls
> directly isn't that big a deal.
>
> But having musl stay broken because it broke itself seems pretty
> reasonable too. The fact is, Linux has a system call and the man page
> says we have a certain behavior, and musl isn't supporting that. If you
> build against a uClibc where you switched off regex support, grep won't
> work. If you build against a musl you haven't fixed, chrt won't work...
>
> (This is one of those "the choices are close enough that there's no
> obvious winner" things that's harder to make than a more important
> decision would be. :)
>
> Right now I've thrown it back on the todo list, which means "musl stays
> broken" is the default for now. Probably acceptable. Lemme finish this
> utf8 plumbing and then I need to fix ps again, then get back to... was
> lsof or dd next?
>
> Rob
>
> (Taking a break from endless GPS. For some reason it takes me about a
> day to switch mental contexts after doing serious GPS stuff, and I keep
> getting sucked back into "5 more minutes of GPS, just this one quick
> thing" that turns into 8 hours, so I'm saying no a bit because the
> toybox todo heap is teetering dangerously September 19th is the 3 month
> anniversary of last release which means time for a new one...)
> ___
> Toybox mailing list
> Toybox@lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net
>
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] musl intentionally broke chrt

2017-08-31 Thread enh 
On Thu, Aug 31, 2017 at 1:46 PM, Rob Landley  wrote:
> On 08/30/2017 10:25 AM, enh wrote:
>> no: the point of the C library is to hide the mapping from
>> functionality to system call. if you ever do port to BSD/macOS you'll
>> need to cope with their syscall differences if you don't let the C
>> library do it, and you're likely to hit places on Android where the
>> traditional system call isn't allowed because bionic doesn't use it
>> and seccomp disables the rest.
>
> Indeed. A broken C library was doing this, then stopped.
>
> On the one hand, this is a musl bug. (He disagrees with the system call
> Linux implemented, so he broke his wrapper to punish people who try to
> use it. If he didn't provide any wrapper I could probe at compile time,
> but musl provides broken stuff you can only tell is broken at runtime,
> so a build that supports cross compiling has no choice but to either
> bloat the code with runtine workarounds, stop supporting musl, or have
> config switches to manually select the behavior at compile time.)
>
> On the other hand, I remember gcc and glibc pointing fingers at each
> other for over FIVE YEARS about -gc-sections breaking stdout flushing at
> exit. The busybox maintainer I handed off to did a presentation
> explaining the problem in in 2010. Doesn't seem to be on youtube but if
> you can dig up code old enough to play "ogg" format (remember that?) the
> pdf and video are at:
>
>   http://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vlasenko.pdf
>
> http://free-electrons.com/pub/video/2010/elc/elc2010-vlasenko-dead-code.ogv
>
> (Fascinating problem, and really annoying to those of trying to just use
> the darn toolchain to built working code...)
>
> Rich is never going to admit he's wrong on this, or on the nommu fork()
> issue, or the meta-mistake of musl not having a #define so we can
> reliably work around his design mistakes ourselves. Me refusing to add a
> workaround for his design decisions reduces to the same "gcc and glibc
> devs pointing fingers at each other and each refusing to change" problem
> that sucks for end users.
>
> In this instance, chrt isn't core functionality, so I can punt for a
> while. But there will be a 4th instance of this, it's only matter of time...
>
>>> Right now I've thrown it back on the todo list, which means "musl stays
>>> broken" is the default for now. Probably acceptable. Lemme finish this
>>> utf8 plumbing and then I need to fix ps again, then get back to... was
>>> lsof or dd next?
>>
>> didn't you get in to utf8 because of my wc -m patch? :-)
>
> Working on it. It's one of those "I'd like to do what I consider the
> _proper_ fix" things that's honestly been a bit of a luxury these days.
>
> I wrote a for loop to go from 0 to UINT_MAX, and I'm comparing the
> mbrtowc(, str, 4, ) results to my contextless utf8towc(, str,
> len) output, and I'm fixing every deviation between the two. I'm
> currently trying to figure out why 0xeda080 _isn't_ 0xd800. (glibc
> translates wc 0xd800 as f8a08a83 but it's less than  so
> https://en.wikipedia.org/wiki/UTF-8 says it should be 3 bytes and I'm
> CONFUSED...)

U+d800 is a surrogate, so shouldn't be valid in utf8.

> (Context-related edge cases aside, I want utf8 support enabled
> regardless of locale and the glibc performance of its multibyte code is
> _terrible_, so this is worth doing anyway.)
>
> Rob



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] musl intentionally broke chrt

2017-08-31 Thread Rob Landley 
On 08/30/2017 10:25 AM, enh wrote:
> no: the point of the C library is to hide the mapping from
> functionality to system call. if you ever do port to BSD/macOS you'll
> need to cope with their syscall differences if you don't let the C
> library do it, and you're likely to hit places on Android where the
> traditional system call isn't allowed because bionic doesn't use it
> and seccomp disables the rest.

Indeed. A broken C library was doing this, then stopped.

On the one hand, this is a musl bug. (He disagrees with the system call
Linux implemented, so he broke his wrapper to punish people who try to
use it. If he didn't provide any wrapper I could probe at compile time,
but musl provides broken stuff you can only tell is broken at runtime,
so a build that supports cross compiling has no choice but to either
bloat the code with runtine workarounds, stop supporting musl, or have
config switches to manually select the behavior at compile time.)

On the other hand, I remember gcc and glibc pointing fingers at each
other for over FIVE YEARS about -gc-sections breaking stdout flushing at
exit. The busybox maintainer I handed off to did a presentation
explaining the problem in in 2010. Doesn't seem to be on youtube but if
you can dig up code old enough to play "ogg" format (remember that?) the
pdf and video are at:

  http://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vlasenko.pdf

http://free-electrons.com/pub/video/2010/elc/elc2010-vlasenko-dead-code.ogv

(Fascinating problem, and really annoying to those of trying to just use
the darn toolchain to built working code...)

Rich is never going to admit he's wrong on this, or on the nommu fork()
issue, or the meta-mistake of musl not having a #define so we can
reliably work around his design mistakes ourselves. Me refusing to add a
workaround for his design decisions reduces to the same "gcc and glibc
devs pointing fingers at each other and each refusing to change" problem
that sucks for end users.

In this instance, chrt isn't core functionality, so I can punt for a
while. But there will be a 4th instance of this, it's only matter of time...

>> Right now I've thrown it back on the todo list, which means "musl stays
>> broken" is the default for now. Probably acceptable. Lemme finish this
>> utf8 plumbing and then I need to fix ps again, then get back to... was
>> lsof or dd next?
> 
> didn't you get in to utf8 because of my wc -m patch? :-)

Working on it. It's one of those "I'd like to do what I consider the
_proper_ fix" things that's honestly been a bit of a luxury these days.

I wrote a for loop to go from 0 to UINT_MAX, and I'm comparing the
mbrtowc(, str, 4, ) results to my contextless utf8towc(, str,
len) output, and I'm fixing every deviation between the two. I'm
currently trying to figure out why 0xeda080 _isn't_ 0xd800. (glibc
translates wc 0xd800 as f8a08a83 but it's less than  so
https://en.wikipedia.org/wiki/UTF-8 says it should be 3 bytes and I'm
CONFUSED...)

(Context-related edge cases aside, I want utf8 support enabled
regardless of locale and the glibc performance of its multibyte code is
_terrible_, so this is worth doing anyway.)

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] musl intentionally broke chrt

2017-08-30 Thread Rob Landley 
On 08/30/2017 04:39 AM, Andre Renaud wrote:
> Hi Rob,
> Can you not call "pthread_setschedparam" using on pthread_self instead
> of calling sched_setscheduler?

The same commit broke a half-dozen related system calls, so I'd still
need to wrap sched_get_priority_max() and similar myself. (Not just -m.
The kernel barfs if you feed it an illegal value but I need range
checking to provide a reasonable error message saying what _would_ be
allowed...)

But a bigger reason is toybox doesn't currently use threads so I'm not
linking -lpthread (a requirement on glibc, which subtly changes the
behavior of lots of stuff, plus now it won't build on a toolchain built
with --disable-threads and it starts caring about allocating a thread
local storage block/register... I use "minimal bootstrapping cycle" as a
proxy for ease of porting to new hardware targets, ease of fully
understanding a system end to end (ala xv6's explicit educational goal),
and people keep gluing chunks of busybox into bootloaders and RTOS's
where you have to manually provide all the system interfaces the code
uses and if they might want to do the same with toybox someday keeping
it down to a dull roar is useful. I'm aware /proc and /sys are a
quagmire in that regard, but I'd prefer _not_ to open another can of
worms without good reason.

In general moving the system call wrapper over into the threading
library with added gratuitous locking does not strike me as an
improvement. I just want the system call. There's a function for the
system call, with a man page and everything. Musl still PROVIDES the
function, it's just broken at runtime so as with fork() I can't compile
time probe and provide my own wrapper in portability.h only if it wasn't
there; musl gives me broken crap I can only tell is broken at runtime
and refuses to let the _build_ make decisions.

There's a theme here...

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] musl intentionally broke chrt

2017-08-30 Thread enh 
On Wed, Aug 30, 2017 at 1:42 AM, Rob Landley  wrote:
> On 08/29/2017 10:32 AM, enh wrote:
>> On Tue, Aug 29, 2017 at 8:32 AM, enh  wrote:
>>> can't you just infer musl from the relatively small
>>
>> ...number of c libraries out there...
>
> It would work, but I'm uncomfortable with it.
>
> People keep threatening to port toybox to to BSD and MacOS, there is an
> active uClibc fork buildroot's using, it's posible newlib, klibc, or
> dietlibc will grow legs someday, far down on my todo list is maybe
> porting toybox to xv6, what library is fuchsia using, musl itself didn't
> publicly exist ~5 years ago...

1. YAGNI
2. if you ever do, you can just add them

> The posix API is insufficient so I'm using Linux as my API, but
> freebsd's Linux emulation layer is a viable target too, and I wanted the
> rest of it to be as generic as I could with target-specific stuff hidden
> in portability.h. That means C library, compiler, processor architecture...
>
> I'm not sure that's _achievable_ with musl refusing to identify itself
> because its' experience is universal and everyone else is wrong...
>
> Sigh. If I'm going to have the "else case" apply workarounds, I might as
> well make the workarounds apply to everybody. Calling the syscalls
> directly isn't that big a deal.

yes and no...

yes: this is certainly the reason why bionic only uses non-legacy
system calls. aarch64 doesn't have the legacy calls, and rather than
have code paths that differ depending on architecture, we do the same
thing everywhere. (so for example bionic never calls open; it's always
openat, regardless of architecture.)

no: the point of the C library is to hide the mapping from
functionality to system call. if you ever do port to BSD/macOS you'll
need to cope with their syscall differences if you don't let the C
library do it, and you're likely to hit places on Android where the
traditional system call isn't allowed because bionic doesn't use it
and seccomp disables the rest.

> But having musl stay broken because it broke itself seems pretty
> reasonable too. The fact is, Linux has a system call and the man page
> says we have a certain behavior, and musl isn't supporting that. If you
> build against a uClibc where you switched off regex support, grep won't
> work. If you build against a musl you haven't fixed, chrt won't work...
>
> (This is one of those "the choices are close enough that there's no
> obvious winner" things that's harder to make than a more important
> decision would be. :)

yeah, there are enough theoretical pulls in different directions here
that it makes sense to see which ones actually matter as time goes by.

> Right now I've thrown it back on the todo list, which means "musl stays
> broken" is the default for now. Probably acceptable. Lemme finish this
> utf8 plumbing and then I need to fix ps again, then get back to... was
> lsof or dd next?

didn't you get in to utf8 because of my wc -m patch? :-)

> Rob
>
> (Taking a break from endless GPS. For some reason it takes me about a
> day to switch mental contexts after doing serious GPS stuff, and I keep
> getting sucked back into "5 more minutes of GPS, just this one quick
> thing" that turns into 8 hours, so I'm saying no a bit because the
> toybox todo heap is teetering dangerously September 19th is the 3 month
> anniversary of last release which means time for a new one...)



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] musl intentionally broke chrt

2017-08-30 Thread Andre Renaud 
Hi Rob,
Can you not call "pthread_setschedparam" using on pthread_self instead of
calling sched_setscheduler?

Regards,
Andre
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] musl intentionally broke chrt

2017-08-30 Thread Rob Landley 
On 08/29/2017 10:32 AM, enh wrote:
> On Tue, Aug 29, 2017 at 8:32 AM, enh  wrote:
>> can't you just infer musl from the relatively small
> 
> ...number of c libraries out there...

It would work, but I'm uncomfortable with it.

People keep threatening to port toybox to to BSD and MacOS, there is an
active uClibc fork buildroot's using, it's posible newlib, klibc, or
dietlibc will grow legs someday, far down on my todo list is maybe
porting toybox to xv6, what library is fuchsia using, musl itself didn't
publicly exist ~5 years ago...

The posix API is insufficient so I'm using Linux as my API, but
freebsd's Linux emulation layer is a viable target too, and I wanted the
rest of it to be as generic as I could with target-specific stuff hidden
in portability.h. That means C library, compiler, processor architecture...

I'm not sure that's _achievable_ with musl refusing to identify itself
because its' experience is universal and everyone else is wrong...

Sigh. If I'm going to have the "else case" apply workarounds, I might as
well make the workarounds apply to everybody. Calling the syscalls
directly isn't that big a deal.

But having musl stay broken because it broke itself seems pretty
reasonable too. The fact is, Linux has a system call and the man page
says we have a certain behavior, and musl isn't supporting that. If you
build against a uClibc where you switched off regex support, grep won't
work. If you build against a musl you haven't fixed, chrt won't work...

(This is one of those "the choices are close enough that there's no
obvious winner" things that's harder to make than a more important
decision would be. :)

Right now I've thrown it back on the todo list, which means "musl stays
broken" is the default for now. Probably acceptable. Lemme finish this
utf8 plumbing and then I need to fix ps again, then get back to... was
lsof or dd next?

Rob

(Taking a break from endless GPS. For some reason it takes me about a
day to switch mental contexts after doing serious GPS stuff, and I keep
getting sucked back into "5 more minutes of GPS, just this one quick
thing" that turns into 8 hours, so I'm saying no a bit because the
toybox todo heap is teetering dangerously September 19th is the 3 month
anniversary of last release which means time for a new one...)
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] musl intentionally broke chrt

2017-08-29 Thread enh 
can't you just infer musl from the relatively small ?

#if defined(__linux__) && (!defined(__BIONIC__) && !defined(__GLIBC__)
&& whatever ulibc is if you haven't given up on that yet)
#define __MUSL__
#endif

?

On Mon, Aug 28, 2017 at 10:35 PM, Rob Landley  wrote:
> The point of chrt is to use the sched_setscheduler() syscall to change a
> PID's scheduler category (by default making it a realtime process), but
> the musl-libc maintainer decided he didn't like that system call and
> changed the wrapper to return -ENOSYS instead of making the system call:
>
> https://git.musl-libc.org/cgit/musl/commit?id=1e21e78bf7a5c24c217446d8760be7b7188711c2
>
> Unfortunately musl-libc has a policy that musl is never wrong and thus
> it doesn't provide a #define __MUSL__ that you can test against to work
> around this kind of breakage, the way every other libc does.
>
>   http://wiki.musl-libc.org/wiki/FAQ#Q:_why_is_there_no_MUSL_macro_.3F
>
> I did a quick stab at replacing sched_setscheduler() with a wrapper that
> calls syscall(NR_sched_setscheduler) but that musl commit broke _all_
> the sched functions, sched_get_priority_max() and so on. I'd have to
> wrap all of them, even on systems that provide non-broken wrappers for
> the Linux syscall.
>
> This is why we have the TOYBOX_MUSL_NOMMU_IS_BROKEN config symbol, for a
> similar case where for 20 years nommu linux let you tell if you were
> building for a system with mmu or without it by a simple compile time
> probe: build a hello world that links against fork(); if it's not there
> it's a nommu system. Rich decided he didn't like that, and provided a
> broken fork() (always returns -ENOSYS) for nommu systems so you can't
> tell at compile time whether it's mmu or not.
>
> I'm trying to figure out if I should just leave chrt broken on musl and
> add a FAQ entry about it. If there was an #ifdef __MUSL__ I'd have the
> build produce a warning and the runtime produce a message, but of course
> there isn't...
>
> Rob
> ___
> Toybox mailing list
> Toybox@lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net