Re: [Toybox] freadahead()

[Rob Landley]

On 9/22/22 18:59, enh wrote:
> On Tue, Apr 12, 2022 at 9:02 PM Rob Landley wrote:
> >     The number I'm looking for is "how many bytes can I read from the 
> FILE
> pointer
> >     without triggering another underlying read from the fd" because I
> can't re-read
> >     that data from the fd. So I need to fetch the data out of the buffer
> and handle
> >     it before calling the "takes the rest as a filehandle" function. In
> the case of
> >     patch.c, something like reading the remaining FILE * data into
> toybuf() and
> >     doing a write() to the output fd before calling sendfile().
> >
> > ah, right. that makes more sense. yeah, i'll add this.
> 
> Yay! Thank you.
> 
> well, that took a while, but better late than
> never: https://android-review.googlesource.com/c/platform/bionic/+/2227544

And now https://www.mail-archive.com/austin-group-l@opengroup.org/msg10171.html

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] freadahead()

[enh via Toybox]

On Tue, Apr 12, 2022 at 9:02 PM Rob Landley  wrote:

> On 4/12/22 20:39, enh wrote:> On Mon, Apr 11, 2022 at 11:25 PM Rob Landley
>  > > wrote:
> >
> > On 4/11/22 19:03, enh wrote:
> > > On Mon, Apr 11, 2022 at 3:19 PM Rob Landley  > 
> > > >> wrote:
> > > Anyway, if musl and bionic have __freadahead(), I can either
> #ifdef
> > the glibc
> > > hack for it in portability.h, or might just move xgetline()
> into
> > portability.c
> > > and have it do get_line() on systems that don't have
> __freadahead().
> > >
> > > you're still screwed on macOS? (oh, except macOS seems to be just
> the regular
> > > BSD header, with everything public and "roughly what you'd
> expect", so that's
> > > easily done manually too.)
> >
> > There's a bunch of stuff in the m4 and glib #ifdefs that show how to
> do it for
> > other libraries, although I only really care about 3 and can fall
> back to
> > get_line() for the others in portability.c. :)
> >
> > But part of my thinking is it's easier to convince the posix guys to
> standardize
> > the wrapper if the wrapper is more widely implemented. It's already
> in musl,
> > Dragonfly BSD, Z/OS...
> >
> > yeah, "standardize existing practice" is what they're supposed to do,
> but they
> > can be pretty slow even then. you might also want to
> > try https://www.openwall.com/lists/libc-coord/2020/01/30/1
>
> I'll poke Rich, the musl list is already on openwall and I'm pretty sure
> he's on
> there already. :)
>
> > Rich has always had a weird reluctance to have a #define admitting
> you're using
> > musl. (That's why my mcm-buildall.sh has to append it to features.h
> after each
> > toolchain build.) Instead he added __freadahead() so autoconf could
> probe for
> > that and use it, avoiding the staircase function entirely.
> >
> > so no-one's even mentioned this to glibc?
>
> Dunno. I approached busybox before coreutils for a reason. A lawyer I used
> to
> know described her profession as "sticking my hands in other people's
> toilets",
> which is pretty much how I feel about interacting directly with Gnu
> projects. I
> _can_ do it, but it's not my first choice. (There's some emotional scar
> tissue
> involved in this reluctance.) And when it does happen, I expect it drag on
> for a
> while, ala:
>
> https://lists.gnu.org/archive/html/coreutils/2022-04/msg00010.html
>
> > or they have, and they were happy with
> > people poking at their fields? (might be worth mentioning on libc-coord
> if
> > no-one's ever even mentioned it to them.)
>
> I was vaguely assuming one of the people who poked Rich to add the
> feature, or
> one of the gnu m4 developers, or one of the gnulib developers, would have
> raised
> this issue with the gnu libc developers at some point over the past 15
> years,
> but no I'm not personally aware of the communication history.
>
> I was sort of vaguely in touch with the eglibc developers for a bit, but
> when
> glibc did the "gcc has recaptured egcs! The FSF is once again firmly in
> control!" thing I backed away slowly maintaining eye contact until it was
> safe
> to run.
>
> > The number I'm looking for is "how many bytes can I read from the
> FILE pointer
> > without triggering another underlying read from the fd" because I
> can't re-read
> > that data from the fd. So I need to fetch the data out of the buffer
> and handle
> > it before calling the "takes the rest as a filehandle" function. In
> the case of
> > patch.c, something like reading the remaining FILE * data into
> toybuf() and
> > doing a write() to the output fd before calling sendfile().
> >
> > ah, right. that makes more sense. yeah, i'll add this.
>
> Yay! Thank you.
>

well, that took a while, but better late than never:
https://android-review.googlesource.com/c/platform/bionic/+/2227544


> Rob
>
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] freadahead()

[Rob Landley]

On 4/12/22 20:39, enh wrote:> On Mon, Apr 11, 2022 at 11:25 PM Rob Landley
 > wrote:
> 
> On 4/11/22 19:03, enh wrote:
> > On Mon, Apr 11, 2022 at 3:19 PM Rob Landley  
> > >> wrote:
> >     Anyway, if musl and bionic have __freadahead(), I can either #ifdef
> the glibc
> >     hack for it in portability.h, or might just move xgetline() into
> portability.c
> >     and have it do get_line() on systems that don't have __freadahead().
> >
> > you're still screwed on macOS? (oh, except macOS seems to be just the 
> regular
> > BSD header, with everything public and "roughly what you'd expect", so 
> that's
> > easily done manually too.)
> 
> There's a bunch of stuff in the m4 and glib #ifdefs that show how to do 
> it for
> other libraries, although I only really care about 3 and can fall back to
> get_line() for the others in portability.c. :)
> 
> But part of my thinking is it's easier to convince the posix guys to 
> standardize
> the wrapper if the wrapper is more widely implemented. It's already in 
> musl,
> Dragonfly BSD, Z/OS...
> 
> yeah, "standardize existing practice" is what they're supposed to do, but they
> can be pretty slow even then. you might also want to
> try https://www.openwall.com/lists/libc-coord/2020/01/30/1

I'll poke Rich, the musl list is already on openwall and I'm pretty sure he's on
there already. :)

> Rich has always had a weird reluctance to have a #define admitting you're 
> using
> musl. (That's why my mcm-buildall.sh has to append it to features.h after 
> each
> toolchain build.) Instead he added __freadahead() so autoconf could probe 
> for
> that and use it, avoiding the staircase function entirely.
> 
> so no-one's even mentioned this to glibc?

Dunno. I approached busybox before coreutils for a reason. A lawyer I used to
know described her profession as "sticking my hands in other people's toilets",
which is pretty much how I feel about interacting directly with Gnu projects. I
_can_ do it, but it's not my first choice. (There's some emotional scar tissue
involved in this reluctance.) And when it does happen, I expect it drag on for a
while, ala:

https://lists.gnu.org/archive/html/coreutils/2022-04/msg00010.html

> or they have, and they were happy with
> people poking at their fields? (might be worth mentioning on libc-coord if
> no-one's ever even mentioned it to them.)

I was vaguely assuming one of the people who poked Rich to add the feature, or
one of the gnu m4 developers, or one of the gnulib developers, would have raised
this issue with the gnu libc developers at some point over the past 15 years,
but no I'm not personally aware of the communication history.

I was sort of vaguely in touch with the eglibc developers for a bit, but when
glibc did the "gcc has recaptured egcs! The FSF is once again firmly in
control!" thing I backed away slowly maintaining eye contact until it was safe
to run.

> The number I'm looking for is "how many bytes can I read from the FILE 
> pointer
> without triggering another underlying read from the fd" because I can't 
> re-read
> that data from the fd. So I need to fetch the data out of the buffer and 
> handle
> it before calling the "takes the rest as a filehandle" function. In the 
> case of
> patch.c, something like reading the remaining FILE * data into toybuf() 
> and
> doing a write() to the output fd before calling sendfile().
> 
> ah, right. that makes more sense. yeah, i'll add this.

Yay! Thank you.

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] freadahead()

[Rob Landley]

On 4/12/22 15:15, dmccunney wrote:
> On Tue, Apr 12, 2022 at 2:25 AM Rob Landley  wrote:
> 
> And speaking of PostgreSQL:
> https://www.postgresql.org/about/news/trademark-actions-against-the-postgresql-community-2302/

My link was intended to demonstrate a historical pattern of behavior on the part
of the Free Software Foundation, and how their definition of "freedom" involves
coercion to force people into GPL licensing when they didn't actually want to.

I could also have used
https://www.linux.com/news/gpl-requirement-could-have-chilling-effect-derivative-distros
or http://landley.net/notes-2012.html#11-01-2012 or a bunch of others to
illustrate this point, but that link was the chronologically oldest one I could
find with a quick search. (He's explicitly explained why Readline in particular
is GPL not GPL, but the closest I could find was
https://www.oreilly.com/openbook/opensources/book/stallman.html#:~:text=Readline
which is just in-passing. There's an actual writeup on it somewhere. The
readline one was "bait", the binutils one was "trickery", the mepis one was
intimidation (threat of bankruptcy)... sort of escalating over the years. Don't
get me started on Bradley's SFC actions or the 1000+ message LKML thread...)

I'm not sure what your link is attempting to demonstrate. That other people file
IP suits too? That this isn't the only time that organization's wound up in 
court?

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] freadahead()

[enh via Toybox]

On Mon, Apr 11, 2022 at 11:25 PM Rob Landley  wrote:

> On 4/11/22 19:03, enh wrote:
> > On Mon, Apr 11, 2022 at 3:19 PM Rob Landley  > > wrote:
> > Anyway, if musl and bionic have __freadahead(), I can either #ifdef
> the glibc
> > hack for it in portability.h, or might just move xgetline() into
> portability.c
> > and have it do get_line() on systems that don't have __freadahead().
> >
> > you're still screwed on macOS? (oh, except macOS seems to be just the
> regular
> > BSD header, with everything public and "roughly what you'd expect", so
> that's
> > easily done manually too.)
>
> There's a bunch of stuff in the m4 and glib #ifdefs that show how to do it
> for
> other libraries, although I only really care about 3 and can fall back to
> get_line() for the others in portability.c. :)
>
> But part of my thinking is it's easier to convince the posix guys to
> standardize
> the wrapper if the wrapper is more widely implemented. It's already in
> musl,
> Dragonfly BSD, Z/OS...
>

yeah, "standardize existing practice" is what they're supposed to do, but
they can be pretty slow even then. you might also want to try
https://www.openwall.com/lists/libc-coord/2020/01/30/1?


> > yeah, iirc i added everything that glibc had but didn't even look at
> musl. (if
> > you'd asked me even this morning, i'd have told you musl was a strict
> subset of
> > glibc. TIL.)
>
> There are a few exceptions.
>
> > Honestly, I
> > have no sympathy for the glibc guys who implemented everything EXCEPT
> > __freadahead() because they want to force people to use
> > gnugnugnustallmanftaghnlib:
> >
> >
> > citation needed :-)
>
> If you want a list of what they DID implement, the man page I linked to
> lists
> them. (It's one of them group man pages for a bunch of related features.)
>

i know what they did do, because bionic got its subset via man7.org :-) i
just didn't realize until now that musl's set != glibc's set.


> If you want examples of gnu pressuring people to use their libraries to
> spread
> their license... they've been pretty open about it? Their darling used to
> be
> libreadline before that got cloned...
>
> https://www.gnu.org/licenses/why-not-lgpl.en.html
> https://news.ycombinator.com/item?id=20772053#:~:text=embraced
>
> There are more explicit on-topic references, but... sigh, stuff wanders by
> over
> the years that can be a bit hard to dig up again. They've been at this
> FOREVER...
>
> https://postgrespro.com/list/thread-id/1596439
>
> That whole
>
> https://www.computerworld.com/article/2732025/gpl-enforcement-sparks-community-flames.html
> thing was not my first licensing rodeo. I first encountered the GPL via
> the EMX
> project back in college like a decade before that:
>
>   https://en.wikipedia.org/wiki/EMX_(programming_environment)
>
> My first serious attempt to educate myself on IP law was over 20 years ago
> now,
> sometime before... Grrr, TMF keeps moving their archives around. Lemme
> splice
> together the week with my own archives...
>
> https://landley.net/writing/mirror/fool/rulemaker000501.htm
>
> https://www.fool.com/archive/portfolios/rulemaker/2000/05/02/get-your-copyrights-here.aspx
>
> https://www.fool.com/archive/portfolios/rulemaker/2000/05/03/you-gonna-patent-that.aspx
> http://landley.net/writing/mirror/fool/rulemaker000504.htm
> http://landley.net/writing/mirror/fool/rulemaker000505.htm
>
> I.E. sometime before that.
>
> I even drove to Boston to talk to Richard Stallman at his office in
> February
> 2001. (I'd already mailed him a VHS copy of the musical 1776, so I could
> explain
> he was John Adams and needed ESR's Franklin and Torvalds' Jefferson to get
> his
> agenda through. He didn't take that well, he thought he was Franklin.
> Sigh.)
>
> Anyway, I was very interested in the IP considerations and ramifications
> of all
> this stuff because licensing and development model were a big part what
> made
> Linux different, and I wanted to understand it. I think
> https://lkml.org/lkml/2002/6/3/199 is why I was cc'd on
>
> https://mail.gnome.org/archives/foundation-list/2004-September/msg00014.html
> but
> it's been a while?
>
> Oh, and at this LinuxWorld Expo in 2000:
>
>
> https://www.fool.com/archive/portfolios/rulemaker/2000/08/18/is-amds-chip-better-than-intels.aspx
>
> I attended a talk where Stallman was VERY UNHAPPY with the "application
> service
> provider" problem where people could run GPL code on a server and not share
> modifications because they weren't distributing the code! He already was
> trying
> to figure out how to change the license to stop this, but the problem is
> since
> they weren't distributing but merely using copyright didn't actually give
> him
> any leverage. (He later switched to "tivoization" where he was instead
> basically
> undermining first sale doctrine. That might also be where I learned that
> GPL
> can't rely on contract law for anything because there's no "privity of
> contract": you 

Re: [Toybox] freadahead()

[dmccunney]

On Tue, Apr 12, 2022 at 2:25 AM Rob Landley  wrote:

And speaking of PostgreSQL:
https://www.postgresql.org/about/news/trademark-actions-against-the-postgresql-community-2302/

> Rob
__
Dennis
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] freadahead()

[Rob Landley]

On 4/11/22 19:03, enh wrote:
> On Mon, Apr 11, 2022 at 3:19 PM Rob Landley  > wrote:
> Anyway, if musl and bionic have __freadahead(), I can either #ifdef the 
> glibc
> hack for it in portability.h, or might just move xgetline() into 
> portability.c
> and have it do get_line() on systems that don't have __freadahead(). 
> 
> you're still screwed on macOS? (oh, except macOS seems to be just the regular
> BSD header, with everything public and "roughly what you'd expect", so that's
> easily done manually too.)

There's a bunch of stuff in the m4 and glib #ifdefs that show how to do it for
other libraries, although I only really care about 3 and can fall back to
get_line() for the others in portability.c. :)

But part of my thinking is it's easier to convince the posix guys to standardize
the wrapper if the wrapper is more widely implemented. It's already in musl,
Dragonfly BSD, Z/OS...

> yeah, iirc i added everything that glibc had but didn't even look at musl. (if
> you'd asked me even this morning, i'd have told you musl was a strict subset 
> of
> glibc. TIL.)

There are a few exceptions.

> Honestly, I
> have no sympathy for the glibc guys who implemented everything EXCEPT
> __freadahead() because they want to force people to use
> gnugnugnustallmanftaghnlib:
> 
> 
> citation needed :-)

If you want a list of what they DID implement, the man page I linked to lists
them. (It's one of them group man pages for a bunch of related features.)

If you want examples of gnu pressuring people to use their libraries to spread
their license... they've been pretty open about it? Their darling used to be
libreadline before that got cloned...

https://www.gnu.org/licenses/why-not-lgpl.en.html
https://news.ycombinator.com/item?id=20772053#:~:text=embraced

There are more explicit on-topic references, but... sigh, stuff wanders by over
the years that can be a bit hard to dig up again. They've been at this 
FOREVER...

https://postgrespro.com/list/thread-id/1596439

That whole
https://www.computerworld.com/article/2732025/gpl-enforcement-sparks-community-flames.html
thing was not my first licensing rodeo. I first encountered the GPL via the EMX
project back in college like a decade before that:

  https://en.wikipedia.org/wiki/EMX_(programming_environment)

My first serious attempt to educate myself on IP law was over 20 years ago now,
sometime before... Grrr, TMF keeps moving their archives around. Lemme splice
together the week with my own archives...

https://landley.net/writing/mirror/fool/rulemaker000501.htm
https://www.fool.com/archive/portfolios/rulemaker/2000/05/02/get-your-copyrights-here.aspx
https://www.fool.com/archive/portfolios/rulemaker/2000/05/03/you-gonna-patent-that.aspx
http://landley.net/writing/mirror/fool/rulemaker000504.htm
http://landley.net/writing/mirror/fool/rulemaker000505.htm

I.E. sometime before that.

I even drove to Boston to talk to Richard Stallman at his office in February
2001. (I'd already mailed him a VHS copy of the musical 1776, so I could explain
he was John Adams and needed ESR's Franklin and Torvalds' Jefferson to get his
agenda through. He didn't take that well, he thought he was Franklin. Sigh.)

Anyway, I was very interested in the IP considerations and ramifications of all
this stuff because licensing and development model were a big part what made
Linux different, and I wanted to understand it. I think
https://lkml.org/lkml/2002/6/3/199 is why I was cc'd on
https://mail.gnome.org/archives/foundation-list/2004-September/msg00014.html but
it's been a while?

Oh, and at this LinuxWorld Expo in 2000:

https://www.fool.com/archive/portfolios/rulemaker/2000/08/18/is-amds-chip-better-than-intels.aspx

I attended a talk where Stallman was VERY UNHAPPY with the "application service
provider" problem where people could run GPL code on a server and not share
modifications because they weren't distributing the code! He already was trying
to figure out how to change the license to stop this, but the problem is since
they weren't distributing but merely using copyright didn't actually give him
any leverage. (He later switched to "tivoization" where he was instead basically
undermining first sale doctrine. That might also be where I learned that GPL
can't rely on contract law for anything because there's no "privity of
contract": you haven't got an agreement if you can't say who your agreement is
WITH. Alas the law has shifted a bit since then in truly horrible ways and the
FSF has been on the wrong side of that...)

So when I launched the first GPL enforcement suits in 2006 it didn't come out of
nowhere. Almost immediately after that the FSF broke the GPL into warring camps
and doomed it to irrelevance, and it took me something like 5 years to work
through the kubler-ross stages of grief. (Maintaining "last GPLv2" versions of
things involved both denial and anger. I did not bargain with the FSF ever
because Rudyard Kipling was 

Re: [Toybox] freadahead()

[enh via Toybox]

On Mon, Apr 11, 2022 at 3:19 PM Rob Landley  wrote:

> Elliott: could you add __freadahead() to bionic the way musl and z/os and
> so on
> have?
>
> I'm bothering the posix guys about this, thread starts here:
>
>   https://www.mail-archive.com/austin-group-l@opengroup.org/msg09468.html
>
> The problem is calling getline() then calling fileno() and reading from
> that can
> miss data in the FILE * buffer, and that's why patch.c is still using
> get_line()
> to do byte at a time reads so it can hand off to sendfile() when it runs
> out of
> hunks. It's also what wget wants for zlib support, it's likely to come up
> in
> Moritz' implementation of git, and so on. Basically anything with an
> smtp/mbox
> ancestry that needs a filehandle to process the "body" after the keyword:
> value
> headers.
>
> Unfortunately the ANSI C guys pointed out they don't do anything with file
> descriptors (no open() no read() no nothing), which is why fdopen() and
> fileno()
> (and for that matter getline()) live in posix, not ANSI C.
>
> But the Posix guys are so far saying (in private email, not to the list,
> sigh)
> that mucking about with the internals of an ANSI struct is thing that ANSI
> should do, not them. Except fileno() already does that, and ANSI hasn't got
> getline(), so the use case for this does not exist over there. You can ONLY
> access files through FILE * in pure ANSI C, so there's no context
> switching that
> can miss data...
>
> Anyway, if musl and bionic have __freadahead(), I can either #ifdef the
> glibc
> hack for it in portability.h, or might just move xgetline() into
> portability.c
> and have it do get_line() on systems that don't have __freadahead().


you're still screwed on macOS? (oh, except macOS seems to be just the
regular BSD header, with everything public and "roughly what you'd expect",
so that's easily done manually too.)

yeah, iirc i added everything that glibc had but didn't even look at musl.
(if you'd asked me even this morning, i'd have told you musl was a strict
subset of glibc. TIL.)


> Honestly, I
> have no sympathy for the glibc guys who implemented everything EXCEPT
> __freadahead() because they want to force people to use
> gnugnugnustallmanftaghnlib:
>

citation needed :-)

i'd be genuinely curious to know why glibc doesn't have this (and
especially "why musl does anyway"). i'll have to check what else is in
musl's  that isn't in glibc's. looks like these:

size_t __freadahead(FILE *);
const char *__freadptr(FILE *, size_t *);
void __freadptrinc(FILE *, size_t);



>   https://linux.die.net/man/3/__fpending
>
> They have _fbufsize() but not how much of the buffer is used. That's just
> sad.
>

that does seem of questionable utility now you bring it up. (though as you
pointed out on the posix list, unless you _know_ you can seek on a stream,
all of this is pretty dodgy...)


> You can ask this question for output, but not for input. (And given that
> you
> can't plug the output any data you wrote might already have been sent so
> being
> able to "unsend" it without seeking is never reliable... sigh.)
>
> Rob
>
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net