Re: IANA TZ / NerBSD TZ: tzalloc/tzfree and localtime_rz, mktime_z
On Thu, Jan 4, 2024 at 2:25 PM Steffen Nurpmeso wrote: > > enh wrote in > : > |for other precedent, bionic [Android] has tzalloc()/tzfree(), > |mktime_z(), localtime_rz(), and the timezone_t type since API level > |35: > | > |https://android.googlesource.com/platform/bionic/+/main/libc/include/time.h > > Not yet released? "Vanilla ice cream". yeah, it's in AOSP right now, but in terms of "phones in regular people's pockets", it'll be in this year's release. > It really is the better interface. indeed. i regret not exposing this years ago. i don't think the rust people realized their thread-safe house was built on the sand of a shared global, but at least this gives them an alternative going forward (for Android and NetBSD, anyway). > --steffen > | > |Der Kragenbaer,The moon bear, > |der holt sich munter he cheerfully and one by one > |einen nach dem anderen runter wa.ks himself off > |(By Robert Gernhardt)
Re: IANA TZ / NerBSD TZ: tzalloc/tzfree and localtime_rz, mktime_z
for other precedent, bionic [Android] has tzalloc()/tzfree(),
mktime_z(), localtime_rz(), and the timezone_t type since API level
35:
https://android.googlesource.com/platform/bionic/+/main/libc/include/time.h
On Wed, Jan 3, 2024 at 3:22 PM Steffen Nurpmeso via austin-group-l at
The Open Group wrote:
>
> Hello.
>
> A happy and healthy new year 2024 i wish.
>
> As stated in [1] the localtime() and mktime() series of functions
> have the inherent problem of not being thread-safe regarding
> possible changes to the time zone: in a pure POSIX environment
> changes to TZ always affect global data.
>
> If my memory serves correctly, about a decade ago the NetBSD
> project contacted the IANA TZ maintainer in order to upstream
> a new, truly thread-safe interface that addresses this issue.
> Since some time in 2014 (see [1]) the IANA TZ database, which
> includes the Public Domain aka open source code as is used by many
> projects to implement the time related programming interface,
> includes a new series of functions:
>
>timezone_t tzalloc(char const *TZ);
>void tzfree(timezone_t tz);
>
>struct tm *localtime_rz(timezone_t restrict zone,
>time_t const *restrict clock,
>struct tm *restrict result);
>struct tm *restrict tm);
>time_t mktime_z(timezone_t restrict zone,
>struct tm *restrict tm);
>
> [1] https://austingroupbugs.net/view.php?id=1788
>
> If POSIX would offer this interface, the open source (public
> domain) code and manual of which are available via the IANA TZ,
> truly "thread-safe" time programming becomes possible in POSIX.
>
> This is especially important if no CLOCK_TAI is available.
> For an example, here is what the widely used NTP server chrony
> performs in order to achieve its task:
>
> tm = gmtime();
> if (!tm)
> return tz_leap;
>
> stm = *tm;
>
> /* Temporarily switch to the timezone containing leap seconds */
> tz_env = getenv("TZ");
> if (tz_env) {
> if (strlen(tz_env) >= sizeof (tz_orig))
> return tz_leap;
> strcpy(tz_orig, tz_env);
> }
> setenv("TZ", leap_tzname, 1);
> tzset();
>
> /* Get the TAI-UTC offset, which started at the epoch at 10 seconds */
> t = mktime();
> if (t != -1)
> tz_tai_offset = t - when + 10;
>
> /* Set the time to 23:59:60 and see how it overflows in mktime() */
> stm.tm_sec = 60;
> stm.tm_min = 59;
> stm.tm_hour = 23;
>
> t = mktime();
>
> if (tz_env)
> setenv("TZ", tz_orig, 1);
> else
> unsetenv("TZ");
> tzset();
>
> if (t == -1)
> return tz_leap;
>
> if (stm.tm_sec == 60)
> tz_leap = LEAP_InsertSecond;
> else if (stm.tm_sec == 1)
> tz_leap = LEAP_DeleteSecond;
>
> *tai_offset = tz_tai_offset;
>
> I want to point out that setting an environment variable can be
> a costly operation, but moreover changing the timezone as such
> may involve several file system operations, being a potentially
> very expensive operation. (By the way the draft 4 uses "file
> system" as well as "filesystem".)
>
> With the new interface two timezone objects can be preallocated,
> and the operations are totally detached from global data and
> multithread-safe.
>
> My question: against which standard should an issue be opened?
>
> --steffen
> |
> |Der Kragenbaer,The moon bear,
> |der holt sich munter he cheerfully and one by one
> |einen nach dem anderen runter wa.ks himself off
> |(By Robert Gernhardt)
>
Re: system(NULL) overly restrictive?
On Wed, Oct 25, 2023 at 11:47 AM Gabriel Ravier wrote:
>
> On 10/24/23 13:59, Oğuz via austin-group-l at The Open Group wrote:
> > Oğuz
> >
> > On Tue, Oct 24, 2023 at 1:53 PM Jonathan Wakely wrote:
> >>
> >>
> >> On Tue, 24 Oct 2023 at 07:10, Oğuz via austin-group-l at The Open Group
> >> wrote:
> >>> On Tuesday, October 24, 2023, enh via austin-group-l at The Open Group
> >>> wrote:
> >>>> netbsd checks that _PATH_BSHELL is exectuable with access(2)
> >>>> (but doesn't actually _execute_ anything). apple's copy of freebsd has
> >>>> a local change similar to the netbsd one. glibc seems to actually try
> >>>> to _run_ a shell:
> >>>> ```
> >>>> [pid 3612818] execve("/bin/sh", ["sh", "-c", "exit 0"], 0x7fff98502fe8
> >>>> ```
> >>>
> >>> None of those guarantee that the shell is actually gonna work. I think
> >>> they all should just return 1.
> >>>
> >>>> but a
> >>>> POSIX system can have those kinds of features that could mean
> >>>> system(3) doesn't actually work _for this process_
> >>>
> >>> Such a system should provide the means for testing if a process is in
> >>> that state, and the means shouldn't involve auditing the shell executable.
> >>
> >> Why not? Checking if you can run /bin/sh seems like the most direct and
> >> accurate way of checking if the process can run /bin/sh. POSIX might
> >> require /bin/sh to be usable, so a "pure POSIX" system can safely just
> >> return non-zero from system(NULL). But a system that supports features
> >> like SELinux that aren't part of POSIX might want to do extra checks,
> >> which wouldn't be required on a system where /bin/sh is always available.
> >>
> >> Why require those systems to provide some other non-standard API for
> >> testing it when system(NULL) is specified by C to do that job?
> > system(NULL) is specified by both C and POSIX to return whether the
> > host environment features a command processor or not. It's merely a
> > feature test, as evident from the following paragraphs from POSIX
> > Issue 8 draft 3 P2196 L71808-71821:
> >
> >> Note that, system(NULL) is required to return non-zero, indicating that
> >> there is a command
> >> language interpreter. At first glance, this would seem to conflict with
> >> the ISO C standard which
> >> allows system(NULL) to return zero. There is no conflict, however. A
> >> system must have a
> >> command language interpreter, and is non-conforming if none is present. It
> >> is therefore
> >> permissible for the system() function on such a system to implement the
> >> behavior specified by
> >> the ISO C standard as long as it is understood that the implementation
> >> does not conform to
> >> POSIX.1-202x if system(NULL) returns zero.
> >>
> >> It was explicitly decided that when command is NULL, system() should not
> >> be required to check
> >> to make sure that the command language interpreter actually exists with
> >> the correct mode, that
> >> there are enough processes to execute it, and so on. The call system(NULL)
> >> could, theoretically,
> >> check for such problems as too many existing child processes, and return
> >> zero. However, it
> >> would be inappropriate to return zero due to such a (presumably) transient
> >> condition. If some
> >> condition exists that is not under the control of this application and
> >> that would cause any
> >> system() call to fail, that system has been rendered non-conforming.
> > On a system where certain users/processes can have limited/no shell
> > access, providing a separate interface like do_I_have_shell_access()
> > would be more appropriate than overloading system(NULL).
> >
> Isn't this a similar situation to e.g. `getuid`/`geteuid`/etc. failures
> ? I don't see the distinction between `getuid` and other associated
> functions saying that they "shall always be successful and no return
> value is reserved to indicate the error" and `system` stating "The
> system( ) function shall always return non-zero when command is NULL." -
> personally I would consider the note in `getuid`'s rationale stating:
>
> > It is possible for implementations to provide an extension where a
> process in a non-conforming environment will not be ass
system(NULL) overly restrictive?
C23 ("7.24.4.8 The system function") is stricter
than POSIX: "If the argument is a null pointer, the system function
returns nonzero only if a command processor is available".
POSIX: "If command is a
null pointer, system() shall return non-zero to indicate that a
command processor is available, or zero if none is available. [CX] The
system() function shall always return non-zero when command is NULL."
i'm assuming the intention here was "you're not a POSIX system without
a shell, so it's not possible for system(NULL) to fail to report that
a command processor is available" ... but is that true? what does
"available" mean?
openbsd and freebsd and musl and android's bionic libc always just
return 1. netbsd checks that _PATH_BSHELL is exectuable with access(2)
(but doesn't actually _execute_ anything). apple's copy of freebsd has
a local change similar to the netbsd one. glibc seems to actually try
to _run_ a shell:
```
[pid 3612818] execve("/bin/sh", ["sh", "-c", "exit 0"], 0x7fff98502fe8
```
i'd taken POSIX literally in bionic because "who cares?" --- i'd never
seen anyone actually try to use this. perhaps inspired by the man7.org
and macOS man pages, though, since then we've seen a few projects
(many admittedly seemingly copy & pastes of one another) try
system(NULL) before trying the command they _actually_ want.
which brings me back to (a) is POSIX being overly restrictive here?
and (b) what should i actually do in bionic? i'm unconvinced by the
access(X_OK) of netbsd/macOS: that seems neither one thing nor the
other. either `return 1` on the basis of "sure, there's a shell [but
if you want to know whether the command you want to run is going to
work, you'd better actually try it, because nothing else will
guarantee that]" seems fair enough, or "i actually ran a shell command
[like glibc does] and checked the result was what i expected [where
something like `exit 22` might be a bit more convincing]" seems fair
enough too, and at least "less useless".
i'm struggling to think of a POSIX way to end up in a situation where
your shell doesn't work; no chroots or containers or whatever. but a
POSIX system can have those kinds of features that could mean
system(3) doesn't actually work _for this process_, so "The
system() function shall always return non-zero when command is NULL"
seems like a bug?
"forgotten your password?" link broken?
the "forgotten your password?" link on https://www.opengroup.org/austin/lists.html takes me to a page to _reset_ my password which assumes i know my current password. (in the meantime, apologies for not having the right page numbers and line numbers in https://www.austingroupbugs.net/view.php?id=1775 --- the html doesn't have those, and i can't access the pdf without my password.)
Re: [1003.1(2016/18)/Issue7+TC2 0001457]: Add readlink(1) utility
On Fri, Feb 25, 2022 at 1:30 PM Thorsten Glaser wrote: > enh via austin-group-l at The Open Group dixit: > > >> OK, I did. Note that we're talking of options for realpath(1), not > >> readlink(1). It looks as if the toybox version (from what I can gather > >> from what I guess from that very stylised code) has no options to > realpath > >> at all - and that it is simply a clone of readlink -f, just as Steffen > >> indicated the busybox version is. > > > >correct. > > But useless as most Android scripts would get mksh’s realpath > if called without any options… > didn't i disable that, to avoid such confusion? yeah: https://cs.android.com/android/platform/superproject/+/master:external/mksh/src/funcs.c;l=130?q=realpath%20file:mksh (you're right in principle, though: this will be true for sufficiently old versions of Android.) > (some custom Android ROMs add busybox or GNU bash, but that’s > thankfully become rare) > > bye, > //mirabilos > -- > 22:20⎜ The crazy that persists in his craziness becomes a master > 22:21⎜ And the distance between the craziness and geniality is > only measured by the success 18:35⎜ "Psychotics are consistently > inconsistent. The essence of sanity is to be inconsistently inconsistent >
Re: [1003.1(2016/18)/Issue7+TC2 0001457]: Add readlink(1) utility
On Fri, Feb 25, 2022 at 12:20 PM Robert Elz via austin-group-l at The Open Group wrote: > Date:Fri, 25 Feb 2022 15:25:53 + > From:"Geoff Clare via austin-group-l at The Open Group" < > austin-group-l@opengroup.org> > Message-ID: <20220225152553.GA4559@localhost> > > | The point is it's a difference in behaviour between the two > | implementations. > > Yes, understood. My question (not to you, but to the universe) was > whether that difference needs to exist? That is, perhaps I can change > the NetBSD impl to at least behave wrt -e (and the fictional -E) the > same as the coreutils version. That seems like it would be a better > outcome, if it was possible, right? > > Right now I have no idea whether it is possible however (the code > is obviously possible, it is the compat issues/politics that are > questionable). > > | Rather than just making it unspecified whether > | the last component has to exist, it seemed to me that it would > | be more useful to have -e and -E options so that users have a > | way to ensure they get the same behaviour on both. > > Yes, if it turns out not to be possible to alter the default behaviour > of the default for our version, that would certainly be a reasonable > approach. > > | So my preferences are (in descending order): > > I'd insert: > > 0. Make the BSD version compatible with coreutils (at least for >the one issue that seems to matter - though I'm not sure what >its practical uses are). > > | 1. POSIX adds realpath with -e and -E, > > [...] > > | I wasn't proposing any other options be included for realpath > | (except perhaps -q, depending on whether it behaves the same in > | both implementations). > > Understood, I was just ruminating on the others on the assumption that > if I can attempt to make ours compatible, just how compat should I aim. > It won't be all the way, but perhaps more than just the -e issue. > > The -q seems to be (close enough to, given the other differences) the > same between the two (at least according to the coreutils man page). > It just makes some stderr output vanish (which incidentally might make > realpath do exit(1) without anything on stderr). > > e...@google.com said: > | in terms of "what's actually used in the wild", Android uses toybox > (0BSD > | licensed, so anyone can look :-) ) > > OK, I did. Note that we're talking of options for realpath(1), not > readlink(1). It looks as if the toybox version (from what I can gather > from what I guess from that very stylised code) has no options to realpath > at all - and that it is simply a clone of readlink -f, just as Steffen > indicated the busybox version is. > correct. > There would be no point including realpath(1) in posix if it is simply > readlink -f with no options possible at all - there has to be some > difference > in how it works to be worth including, I would have thought. > except readlink isn't in posix either :-) *that's* the portability issue i've seen in practice. in places where we do/did support macOS as well as linux, we'd end up with a python one-liner or whatever to make up for this. my assumption was "readlink(1) has a bunch of options, getting that into POSIX would be a nightmare ... but everyone seems to have got on just fine with a trivial realpath(1) that has no options [apart from the fact that it's also not in POSIX]". fwiw, current macOS does have readlink (with just -n), but still no realpath. > e...@google.com also said: > | one thing i haven't seen mentioned so far (but which i added to toybox > | myself, so i know it's definitely in use) is that existing realpath > | implementations support *multiple* file arguments on the command line, > not > | just one. > > Sure, as best I can tell, all implementations of both readlink and realpath > allow multiple file (path) args. > i don't think that's true of busybox? > kre > >
Re: [1003.1(2016/18)/Issue7+TC2 0001457]: Add readlink(1) utility
in terms of "what's actually used in the wild", Android uses toybox (0BSD licensed, so anyone can look :-) ) for both on-device *and* for the OS build itself on the host. toybox readlink ( https://github.com/landley/toybox/blob/master/toys/other/readlink.c) currently supports: usage: readlink FILE... With no options, show what symlink points to, return error if not symlink. Options for producing canonical paths (all symlinks/./.. resolved): -e Canonical path to existing entry (fail if missing) -f Full path (fail if directory missing) -m Ignore missing entries, show where it would be -n No trailing newline -q Quiet (no output, just error code) since toybox tends to add things _as they're needed_, rather than "because coreutils has them", that's probably "solid anecdata" about what gets used in the wild. one thing i haven't seen mentioned so far (but which i added to toybox myself, so i know it's definitely in use) is that existing realpath implementations support *multiple* file arguments on the command line, not just one. On Fri, Feb 25, 2022 at 6:56 AM Robert Elz via austin-group-l at The Open Group wrote: > OK. I have looked at the coreutils realpath man page (gnu licensing > stupidity means I cannot look at their code), and I can see the > possibility (subject to community agreement) of implementing > some of the options it has. Not all. > > I'm not sure a -E option is needed, if the whole path exists it > makes no difference, if just the last component is missing, I > can't really imagine and BSD usage requiring an error in that case > (and anyone who needs tgat could use -e if I implement this). > > The -m opion just looks stupid, at first glance it would mean > that any trailing part of a path could be missing (since if > a subdir is absent, necessarily it cannot have contents, but > there is also the possibility of /e1/e2/m1/../l1/m2/../e3/l2 > where ane e? name is an existing directory, any l? is an > existing symlink, and each m? is non-existing junk. > Dealing with nonsense like this isn't worth the effort, so > -m is unlikely to ever happen. Some other option (maybe > -p for partial) could exist to allow resolving smylinks > in a leading existing prefix of a path, omitting any more > existance checks beyond that (so no more symlink resolution) > but only if there is some demonstrated need for that. > > -L is not an possibility, I could easily add a do nothing -P > option though (just like our shell's cd command). > > The long name options won't happen, and I find it hard to > see a real need for the 2 --relative options in any case, it > seems likely they could better, and more flexibly, be handled > by post processing the results than built in. > > Then we get to the harder choices ... -z looks easy, and > harmless, but in practice probably mostly useless. So I'm > not sure about that one. -s just turns the whole thing into > a simple string processing program, something that could be > done with a fairly simple sed script, so I'm not sure about > that either. > > Their man page is also woefully short on error details. > Eg: what counts as "missing". Obviously ENOENT, but > what about ENOTDIR in the case of /d/f/m > where d is an existing directory (or path), f is some > non directory file, and m is (obviously) missing? > Or EACCESS for /d/n/m? where d is as previously, > n is an existing directory without x permission (perhaps > no r permission either) and m? is presumed missing as > we cannot determine if it is there, and even if r perm > exists, so we can read /d/n and discover m? exists, > we do not know if is is a symlink or not (cannot lstat). > > This is something of a mess, and while I hesitate to say > it, all too typical of linux/gnu software which only > specifies what happens in the cases they think of, and > all else is just "whatever happens" and isn't necassarily > even consistent in one utility. > > With this it would be difficult (and pure fluke if it > happened) for me to implement something compat, and I'd > hesitate to even suggest adding code to be compat, unless > I could promise that is what tge result woukd be. > > Of course if there was a proposed POSIX spec that made it > clear what is intended in all these odd cases, that might > make a big difference. > > kre > >
Re: Bug 1562 and other locale issues
yeah, having ended up the Android bionic libc maintainer via first maintaining Android's Java i18n libraries, this was basically the conclusion i came to too. it was interesting that -- despite the much larger effort that went into it -- even Java's mid-1990s API was insufficient for pretty mainstream examples in pretty mainstream languages (genitive dates in Russian, say) long before you get to the more exotic stuff. (rule of thumb: anything you think is the same across all human languages has at least one counterexample.) for the Java side, my "fix" was to rely on icu4c behind the scenes, since they started from similar API but added a lot of missing pieces since the mid-1990s. even icu's still a work in progress though, 20+ years later. (the "some languages have more than two day periods" example you mention, for example, was added only fairly recently.) for the C side, my approach has been "C.UTF-8" ... basically, everything's utf-8 (other than input to iconv()), and we use CLDR (via icu4c) for the functions (including stuff like towupper() that's a broken API), but there are no locales. sure, we could handle Turkish/Azeri dotted/dotless i, but i don't think we'd really be helping anyone if we did because (a) we'd introduce a whole world of pain for the common case of the command line, where now you have build breaks if you're running sed on source in one of those locales and (b) if you're actually serious about i18n, you're going to hit the edges of what the C/POSIX API can handle very quickly, and then you need icu4c. i don't think the amount of i18n you can actually achieve with these APIs is particularly worthwhile (you certainly can't write user-facing apps with them beyond a handful of "close enough to English" languages; you don't have to go very far east or south in Europe before it breaks down, let alone visit other continents), and although we could add more code and data, we'd actually just be leading developers down the wrong path ... sooner or later they'd have to rip it all out and start again with icu4c anyway. which, tbh, would be my recommendation for the 1990s "i18n" API in POSIX: rip it out [other than basic utf-8/character conversion support] and point people at icu4c, since i18n it's not just "a few extra things here and there" so much as "an entire project roughly equal in size and complexity to all of the rest of C/POSIX". POSIX is only a decade older, and not even that if you count the ancestors of icu4c, and a lot more work still goes into icu4c than POSIX. (to be clear: i'd mention alternatives to icu4c if i was aware of any, but i honestly don't know of anything comparably complete. i could mention icu4x [https://github.com/unicode-org/icu4x], but aiui that's mainly an exercise in removing the accidental complexity of icu4c because the inherent complexity of i18n is more than enough to be going on with!) On Sat, Feb 12, 2022 at 10:16 PM Robert Elz via austin-group-l at The Open Group wrote: > In general I don't comment much on anything related to i18n, > as it is (way) outside my area. > > However, what is clear is that much of what we have related to > locales is a total botch - thrown together in the 80's/90's > when it was clear that unix systems needed to be able to > operate in the non-English speaking world. > > But that all happened before the net made i18n a whole > different problem - originally one could mostly assume > that the system, or at least a user of the system, was > operating within one consistent locale - that might not > be ascii/English (American) but it would be something > stable (and would include English as at least some kind > of subset). The current "solution" has that general > philosophy embedded throughout. > > All this is simply inadequate now. The printf issue illustrates > some of the issues ... it isn't sufficient to distinguish between > bytes and characters, for the latter we also need to know the locale, > and the LC_* stuff isn't enough, as the locale of the format string > (that part written by the script writer who wrote the printf invocation) > the data (both strings and numerics) that is to be printed controlled > by that format, and the invoking user's desired output locale (that > supported by the display device - or desired as the format of the file) > might all be different, and the LC_* stuff just cannot cope with that. > > For this reason, I'd suggest not wasting time attempting to clarify > just what data is characters (as in might be multibyte) and what is > just uninterpreted bytes, as we really just don't know enough yet to > produce good answers. > > And as dismaying as it might be to some, it isn't this group's job > to invent the solution - that needs to be done by those who care > about it who actually use it all - and implementations need to be > done, tested, and proven satisfactory before any thought of standardising > anything happens. > > One obvious possible solution, at least for character
Re: cut -DF
as someone (the maintainer of the Android command line tools) who has to
deal with Unix n00bs *a lot*, i think the sales pitch for this is really
"this makes cut able to do what new users assume cut is for (but can't)".
(they also are very unlikely to have even heard of awk, and realistically
go straight from "cut's broken" to "i'll use python then".)
On Sat, Dec 4, 2021 at 7:08 AM shwaresyst via austin-group-l at The Open
Group wrote:
> Yes, there's a path; file an Enhancement Request in Mantis. However, if
> toybox wants to be more POSIX conforming it'll have to add an awk
> implementation anyways, eventually, so not sure such a request would get
> much traction for sponsorship. Those with awk already might not want to add
> it to their version of cut, as unnecessary duplication of functionality.
>
> On Sat, Dec 4, 2021 at 9:37 AM, Rob Landley via austin-group-l at The Open
> Group
> wrote:
> Since toybox doesn't have its own awk yet (and thus awk '{print $3 $4
> $5}'),
> back in 2017 toybox added the -D, -F, and -O options to cut:
>
> -D Don't sort/collate selections or match -fF lines without delimiter
> -F Select fields separated by DELIM regex
> -OOutput delimiter (default one space for -F, input delim for -f)
>
> -O is -d for output, -F is a regex version of -f, and -D says to show the
> raw
> matches in the order requested (and ONLY those matches, not passing through
> lines with no matches).
>
> This lets you do:
>
> $ echo one two three four five six seven eight nine | cut -DF 7,1-3,2
> seven one two three two
>
> Elliott Hughes (the Android base OS maintainer) asked if I could get the
> feature
> more widely adopted:
>
>
> http://lists.landley.net/pipermail/toybox-landley.net/2021-June/012453.html
>
> > your non-POSIX cut(1) extension covers 80% of the in-the-wild use of awk
> > anyway :-) if you still talk to any of the busybox folks, we should
> suggest
> > they copy that --- it would be nice for it to be a de facto standard so
> we
> > can get it into POSIX sometime around the 2040s... (and have made lives
> > better for the folks who don't care about standards and just want to "get
> > things done" in the intervening decades!)
>
> So I offered to implement it in busybox:
>
> http://lists.busybox.net/pipermail/busybox/2021-June/06.html
>
> And the busybox maintainer merged it here:
>
> https://git.busybox.net/busybox/commit/?id=0068ce2fa0e3
>
> Is there a path to try to get this option set into posix?
>
> Rob
>
>
Re: Posix issue 8 pending change to system().
good point. fair enough, i'm convinced... fixed by adding "--" to system() and popen() in bionic, with corresponding new tests: https://android-review.googlesource.com/c/platform/bionic/+/1881791 On Fri, Oct 22, 2021 at 1:38 AM Geoff Clare wrote: > Rob Landley wrote, on 22 Oct 2021: > > > > On 10/21/21 6:38 PM, enh wrote: > > > > > it's unclear whether anyone's actually hit this in practice? and > even if they > > > had, their portable workaround would be to prefix with "exec "? > > > > > > maybe try libc-co...@lists.openwall.com and see if there's any > consensus that > > > this is worth the potential trouble? i'm worried that someone is > > > _deliberately_ using this to pass extra flags to the shell, which > wouldn't > > > have a workaround if we did make this change :-( > > It's not possible to pass "extra" flags when the "--" is not there, > because system() only passes one argument after the "-c" to sh. > So you can pass flags _instead_of_ a command string, but not _extra_ > flags. Doing this will just get you an error message from sh about > the command string argument being missing (except on a system were sh > accepts -c with no command string as an extension, but I doubt any > such system exists). > > > > > Pinging you and Rich was my attempt at that, but if there's a dedicated > list... > > Note that I removed the libc-coord list from the Cc. > > -- > Geoff Clare > The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England >
Re: Anyone know how XCU/fuser got into POSIX, or why?
On Fri, Oct 29, 2021 at 5:45 PM Robert Elz wrote: > Date:Fri, 29 Oct 2021 17:22:44 -0700 > From: enh > Message-ID: zyrjcdvqgg6ick...@mail.gmail.com> > > | i've seen fuser used in scripts to provide a list of pids to iterate > ps or > | kill over, > > Written (in the script) how, and with an fuser actually implemented > the way POSIX says to do it?? > > That spec requires stdout to be just an unseparated list of pids, > no white space, no terminating newline, just digits. > /me reads https://pubs.opengroup.org/onlinepubs/9699919799.2013edition/utilities/fuser.html carefully... /facepalm huh. that's ... i have no words. but, yes, that's *actually* what the linux and macOS implementations actually do --- just no-one ever notices because they do get interleaved correctly. i'd never noticed, even after you'd said it, and had to use strace to confirm. > stderr gets the rest of the data. > > You only get anything rational if those two are correctly intermixed, > and both are sent to the same output file (no 2>/dev/null to discard > error messages for example, no simple pids=$(fuser ...) with stderr on > stderr of the shell, for errors, and the stdout via the command > substitution. > > Do the implementations really do all the fflush() calls needed to get > the output sequenced correctly, or perhaps simply set unbuffered mode > on both stdout and stderr, and write a char at a time ? > strace on linux suggests "a combination of the above" --- it looks like stdout is flushed and stderr is unbuffered?! > I really couldn't believe what I was reading when I read the way that > it is defined to work. And that has to be based upon what some real > implementation actually did. > i'm genuinely curious now whether the linux and macOS implementations are descendants of that improbable dinosaur, or have carefully re-created the nonsense because someone wrote it into POSIX... the latter would be a lot sadder than the former! > | it's definitely unfortunate that the -k option is not part of POSIX > > Sorry, no idea what that is, all I know of fuser is what POSIX says about > it. > -k automates the 80% use case of fuser --- "kill the things using this file^Wmount point". (the other 20% being "output diagnostics for some human to look at later".) > | _humans_ seem to use lsof instead. > > Yes, that I'd understand, I've used that too - though the *BSD's fstat > has similar functionality. I haven't bothered to install lsof on my > current system... > > kre > >
Re: Anyone know how XCU/fuser got into POSIX, or why?
i've seen fuser used in scripts to provide a list of pids to iterate ps or kill over, or just as an extra bit of diagnostic information for failures in automated tests. in particular, since it's available by default on macOS as well as linux (and explicitly mentioned in POSIX), it's seen as the portable option. it's definitely unfortunate that the -k option is not part of POSIX (and not available on macOS) though. _humans_ seem to use lsof instead. i'm not sure i've ever seen a human use fuser directly. fuser is on Android's to-do list for toybox, but really just for scripts; lsof was far more important. On Fri, Oct 29, 2021 at 4:17 PM Robert Elz via austin-group-l at The Open Group wrote: > There was a message on one of the NetBSD lists asking why > we don't have the POSIX standardised fuser command in NetBSD. > > I'd never heard of fuser (I've at least heard of most of the > older *nix commands) so I went and read its POSIX man page. > > What a disaster that thing is. > > Its Rationale however is a thing of beauty. I guess it wasn't > politic to explicitly include the kind of blatantly obvious > antecedent to that "This thing is garbage, but" > > I think it is (very) safe to say that NetBSD will never gain a > command that behaves like that (we have at least some level of > quality control). We do have an "fstat" command, which does much > the same job, and is rational. > > Is there any chance we can just make that hideous monstrosity > vanish from the doc?People who (for whatever reason) need > to keep such a command can of course do so, it doesn't need to > be in POSIX for that - but to me, other that for providing info > to a user when run with stdout and stderr both the default > tty output, the thing looks to be completely useless. If that > is its sole purpose, it doesn't need to be standardised. > > kre > >
Re: Pseudoterminal terminology in POSIX
On Tue, Aug 18, 2020 at 9:11 AM Dave Martin wrote: > On Wed, Aug 12, 2020 at 03:19:00PM +0200, Steffen Nurpmeso wrote: > > Joshua M. Clulow via austin-group-l at The Open Group wrote in > > : > > |On Tue, 11 Aug 2020 at 01:33, Michael Kerrisk man-pages via > > |austin-group-l at The Open Group wrote: > > |> On 8/9/20 1:18 AM, Larry Dwyer via Libc-alpha wrote: > > |>> How about the "control" side and the "terminal" side (of the paired > > |>> device files)? > > |> > > |> Thanks for the suggestion. As far as I'm concerned, that would > > |> also be an option worth considering. > > | > > |I work on the illumos project and a few of us have been having a > > |(not yet public) discussion about this lately as well. I think the > > |best one we could think of was: > > | > > |the "control" side for the result of posix_openpt() > > | > > |the "subordinate" side for the result of ptsname() and open(), > > > > You know, (In)Subordination has a very military touch, with > > exclamation mark many may have heard it. Also in traditional > > (white western world) education as such. Like in first the > > pizzle, then the bull pizzle, maybe. So to say. In my ears this > > sounds more aggressive and weird than slave, in a technical > > combination of master/slave, ever could. > > Also isn't it a bit submissive here; it is under control, but > > other than that. > > > > |"/dev/pts" still makes sense, etc > > | > > |we would rename our "/dev/ptmx" device file the "manager > > |driver" rather than the "master" > > | > > |We would strongly consider using the same shift as other projects, > > |but I think only if they actually make sense; e.g., the "terminal" > > |and "pseudoterminal" end doesn't really stand out as completely > > |clear. > > > > Manager sounds strange here, i always liked manager/worker > > terminologie for threads, and used them like that (and am the > > opinion that .. but that is something different and has sailed), > > but for a pseudo terminal? I think that if the standard wants to > > be future proof manager should be avoided. These guys induce > > strange, ruthless and devastating decisions which destroy life on > > earth (as we used to know it), so i for one do not want to be > > harassed by such a term. > > Was this discussion concluded yet? > no, but fwiw i've moved Android's libc over to pty/tty. if POSIX does make a change, i can easily change again. that said, i intend to keep pty and tty in the _code_, because they're a good deal less unclear than the almost meaningless master and slave were and -- despite having heard a *lot* of suggestions at this point -- i'm honestly not expecting anything better than pty/tty... > Question: was there ever an intention that a pty master-slave pair > should resemble two real terminals connected to each other? (e.g., two > serial ports on the same machine, cabled together). > > > POSIX seems pretty vague as to whether the pty master counts as a > terminal or not. In Linux, it has many but not all of the properties > of a terminal. It's not at all clear whether this is intentional, and > I don't know whether other implementations behave similarly. > > The main distinctions I'm aware of are that the pty master cannot become > the controlling terminal of any process, and that both master and slave > have rather weird dialin/hangup semantics which appear rather ad-hoc and > don't map nicely onto the behaviour of physical terminal lines. > > The master also has a few extra ioctls at its disposal for managing the > pair. > > Other stuff does work identically on the pty master and slave though, > such as setting termios modes. I have a vague memory of successfully > setting ECHO on both ends... > > > IMHO, the real problem here is that pty devices are underspecified, > and counterintuitive in some areas. Changing the nomenclature won't fix > that. > ...because of this. as far as i'm concerned you either know what you're doing (and pty and tty are helpful enough, and conveniently match the functions that apply to them ["is there a 'p' in the name?"]), or you're going to have to read the man page anyway, and the man page can cover all the gory details and weird historical cruft in a way that's never going to fit into something short enough to be used as a name. > Plus, renaming things won't kill off the old terminology, and with both > naming schemes in circulation, people are likely to be even more > confused than they were to start with, no? > that's what i like about pty and tty. not perfect, but definitely less bad than what we had in terms of "what is this, and what can i do with it?". but, yeah, you're going to be spending a lot of time with the documentation. > Cheers > ---Dave >
Re: Minutes of the 3rd August 2020 Teleconference
On Tue, Aug 4, 2020 at 8:52 AM Andrew Josey wrote:
>
> All
> Enclosed are the minutes of yesterdays teleconference
> regards
> Andrew
> --
>
> Minutes of the 3rd August 2020 Teleconference Austin-1050 Page 1 of 1
> Submitted by Andrew Josey, The Open Group. 4th August 2020
>
> Attendees:
> Don Cragun, IEEE PASC OR
> Jörg Schilling, FOKUS Franhofer
> Nick Stoughton, USENIX, ISO/IEC JTC 1/SC 22 OR
> Mark Ziegast, SHware Systems Dev.
> Eric Ackermann, HPI, University of Potsdam
> Eric Blake, Red Hat, The Open Group OR
> Geoff Clare, The Open Group
> Tom Thompson, IEEE
> Andrew Josey, The Open Group
> Richard Hansen
>
> Apologies:
>
> * General news
>
> We discussed terminology usage, in particuler terms such as
> master/slave, blacklist/whitelist. It was agreed some terminology
> for pseudo-terminals could be better described using more functionally
> descriptive terms, but the details of this are left to a future bug
> report. Andrew and Geoff took an action to investigate further
> and come back with an analysis.
i was actually planning on joining next week to bring this up. see the
thread at https://www.spinics.net/lists/linux-man/msg16641.html where
various libc maintainers and the man7.org man page maintainer have
been discussing this, with concrete suggestions of "pty"/"tty" for
short, or "pseudoterminal device"/"terminal device", or
"pseudoterminal end of the pseudoterminal device pair"/"terminal end
of the pseudoterminal device pair" if you're not into the whole
brevity thing.
see also https://android-review.googlesource.com/c/platform/bionic/+/1374048
for an example real-life-ish (libc and tests) switch to "pty" and
"tty".
the linux-man thread includes details of the existing practice for
various languages/libraries that we managed to dig up, with the TL;DR
being the suggestions above.
> A reminder that the Draft 1 review is now underway and runs until
> August 25 2020. The draft can be obtained from the login page of
> the Austin Group at: https://www.opengroup.org/austin/login.html .
>
>
> * Outstanding actions
>
> (Please note that this section has been flushed to shorten the minutes -
> to locate the previous set of outstanding actions, look to the minutes
> from 13th June 2019 and earlier)
>
> Bug 1254: "asynchronous list" description uses "command" instead of "AND-OR
> list" OPEN
> https://austingroupbugs.net/view.php?id=1254
> Action: Joerg to investigate how his shell behaves.
>
> Bug 700 - Nick to raise this issue with the C committee
> Bug 713 - Nick to raise with the C committee.
> Bug 739 - Nick to raise with the C committee.
> Bug 1216 - Eric to ask if The Open Group is willing to sponsor this
> interface, referencing bug note 4478.
>
>
> * Current Business
>
> Gettext draft.
>
> We continued this discussion and will continue on the next call.
>
> The gettext draft in the etherpad is at
>
> https://posix.rhansen.org/p/gettext_draft
> https://posix.rhansen.org/p/gettext_split
>
> Next Steps
> --
>
> The next calls are on:
>
> August 6th 2020 (Thursday)
> This call will be for 90 minutes.
>
> Aug 10th 2020 (Monday)
> This call will be for 60 minutes.
>
> Apologies in advance:
> None
>
> Calls are anchored on US time. (8am Pacific)
>
> Please check the calendar invites for dial in details.
>
> Bugs are at:
> https://austingroupbugs.net
>
>
> An etherpad is usually up for the meeting, with a URL using the date format
> as below:
>
> https://posix.rhansen.org/p/20xx-mm-dd
> username={first part of hostname} password=2115756#
>
>
>
> Andrew JoseyThe Open Group
> Austin Group Chair
> Email: a.jo...@opengroup.org
> Apex Plaza, Forbury Road,Reading,Berks.RG1 1AX,England
>
> To learn how we maintain your privacy, please review The Open Group Privacy
> Statement at http://www.opengroup.org/privacy.
> To unsubscribe/opt-out from this mailing list login to The Open Group
> collaboration portal at
> https://collaboration.opengroup.org/operational/portal.php?action=unsub=2481
>
>
>
>
>
Re: Draft minutes of the 5th August 2019 Teleconference
i'm not worried about FreeBSD so much as macOS and (even more so) iOS. i'm assuming they won't take the FreeBSD change, but don't know if anyone knows them to ask them? On Wed, Aug 7, 2019 at 2:58 PM Eric Blake wrote: > > On 8/7/19 4:43 PM, enh wrote: > > What's the plan for the qsort_r interface, given that glibc and BSD have > > mutually incompatible ones (which is why I didn't add it to Android)? > > Per http://austingroupbugs.net/view.php?id=900#c4112, FreeBSD was > planning to switch over to the glibc signature, making it easier to > standardize things as 'qsort_r' as presented in the bug, rather than as > 'posix_qsort_r'. But as there is still a 30-day window for Open Group > objections, we may very well receive an objection to the name 'qsort_r' > where we would have to go with 'posix_qsort_r'. > > -- > Eric Blake, Principal Software Engineer > Red Hat, Inc. +1-919-301-3226 > Virtualization: qemu.org | libvirt.org >
Re: Draft minutes of the 5th August 2019 Teleconference
What's the plan for the qsort_r interface, given that glibc and BSD have mutually incompatible ones (which is why I didn't add it to Android)? On Wed, Aug 7, 2019, 14:02 Eric Blake wrote: > On 8/6/19 4:48 AM, Geoff Clare wrote: > > These are the draft minutes from yesterday's call. Andrew will need > > to allocate the Austin-xxx document number and add the file to the > > document register after he returns. > > > > > > > Minutes of the 5th August 2019 Teleconference Austin-xxx Page 1 of 1 > > Submitted by Geoff Clare, The Open Group. 6th August 2019 > > Followup: > > > > Bug 1220: Add an API to query the name of a locale category of a locale > object OPEN > > http://austingroupbugs.net/view.php?id=1220 > > > > Action: Eric to ask if The Open Group is willing to sponsor this > interface. > > > ... > > > > Bug 1263: Add ppoll() OPEN > > http://austingroupbugs.net/view.php?id=1263 > > > > Action: Eric to ask if The Open Group is willing to sponsor this > interface. > > Now complete, along with earlier actions to ask about sponsorship of > qsort_r in bug 900 and reallocarray in bug 1218. I proposed a 30 day > window for any comments or objections, and will follow up in early > September (with the assumption that no objections is tacit approval that > we proceed with the new interfaces). > > > > Bug 374: malloc(0) and realloc(p,0) must not change errno on success > OPEN > > http://austingroupbugs.net/view.php?id=374 > > > > Geoff had noticed an overlap between changes suggested in this open bug > > and the changes needed to align with C17. > > > > We also noted that glibc does not conform to the change we made in > > 2008-TC1 to require that errno is set to an implementation-defined > > value if realloc(p,0) returns null. This matches the change made in > > C17 7.22.3.1 (overview) which says that if a null pointer is returned in > > the size 0 case it is "to indicate an error". However, 7.22.3.5 > (realloc) > > still says "If size is zero and memory for the new object is not > > allocated, it is implementation-defined whether the old object is > > deallocated" and "The realloc function returns a pointer to the new > > object [...], or a null pointer if the new object has not been allocated" > > which seems to imply a null pointer can be returned in this case without > > it being considered an error. > > > > Action: Eric to ask about this on the glibc mailing list. > > Also done; Florian Weimer has replied to the bug in note 4510, and in > fact,... > > > > > Action: Nick to draft a Clarification Request to WG14. > > ...says he already raised a similar question to WG14 in May 2018 > (although I do not have a URL handy to that thread). > > In fact, the call to standardize reallocarray() may also want to depend > on the outcome here. http://austingroupbugs.net/view.php?id=1218 > > -- > Eric Blake, Principal Software Engineer > Red Hat, Inc. +1-919-301-3226 > Virtualization: qemu.org | libvirt.org > >
Re: shell `var=value function`
On Mon, Jun 10, 2019 at 8:56 AM Chet Ramey wrote:
>
> On 6/10/19 11:38 AM, enh wrote:
> > the toybox project is adding a shell, and came across this issue that
> > i (Android native tools/libraries maintainer) have hit before because
> > mksh does exactly what POSIX says and bash does what POSIX probably
> > intended:
> > http://lists.landley.net/pipermail/toybox-landley.net/2019-June/010530.html
> >
> > repeated here, lightly edited and extended, because i know there are
> > plenty of shell folks here:
> >
> > I've reached the part of the posix shell stuff (section 2.9.1: simple
> > commands)
> > that specifies this behavior, and posix doesn't match bash:
> >
> > If no command name results, or if the command name is a special built-in
> > or
> > function, variable assignments shall affect the current execution
> > environment.
> > Otherwise, the variable assignments shall be exported for the execution
> > environment of the command and shall not affect the current execution
> > environment except as a side-effect of the expansions performed in step 4.
>
> This has been changed in the current published version of POSIX, which
> makes the variable persistance behavior unspecified:
>
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01
excellent!
> "If the command name is a function that is not a standard utility
> implemented as a function, variable assignments shall affect the current
> execution environment during the execution of the function. It is unspecified:
>
> Whether or not the variable assignments persist after the completion of
> the function
>
> Whether or not the variables gain the export attribute during the
> execution of the function
>
> Whether or not export attributes gained as a result of the variable
> assignments persist after the completion of the function (if variable
> assignments persist after the completion of the function)"
>
>
>
> >
> > A) This is not what bash does, or has ever done:
> >
> > $ hello() { echo boing=$BOING; }
> > $ BOING=123 hello
> > $ echo $BOING
>
> While this is true for bash in its default mode, bash implements the
> previously-mandated POSIX behavior in POSIX mode:
>
> $ cat x3
> set -o posix
> hello() { echo boing=$BOING; }
> BOING=123 hello
> echo after: $BOING
> $ ../bash-5.0-patched/bash x3
> boing=123
> after: 123
>
> I agree that the default bash behavior is more useful and often what is
> desired (that's why it's the default), and the next version of bash will
> probably make that the default in POSIX mode as well.
yeah, it also seems like the least surprising behavior... when i first
saw this with folks porting from bash to mksh to run on Android, i was
convinced it was a mksh bug because it didn't make any sense that
you'd get different behavior for a function than for an external
command!
> Chet
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
shell `var=value function`
the toybox project is adding a shell, and came across this issue that
i (Android native tools/libraries maintainer) have hit before because
mksh does exactly what POSIX says and bash does what POSIX probably
intended:
http://lists.landley.net/pipermail/toybox-landley.net/2019-June/010530.html
repeated here, lightly edited and extended, because i know there are
plenty of shell folks here:
I've reached the part of the posix shell stuff (section 2.9.1: simple commands)
that specifies this behavior, and posix doesn't match bash:
If no command name results, or if the command name is a special built-in or
function, variable assignments shall affect the current execution environment.
Otherwise, the variable assignments shall be exported for the execution
environment of the command and shall not affect the current execution
environment except as a side-effect of the expansions performed in step 4.
A) This is not what bash does, or has ever done:
$ hello() { echo boing=$BOING; }
$ BOING=123 hello
$ echo $BOING
$
B) doing it isn't useful because there's no reason to DO an assignment on the
same line rather than on the previous line unless you want to constrain the
lifetime of the variables. (The semicolon character exists, you can do X=y; echo
$X; It's literally one extra character.)
but here's mksh:
~$ mksh
$ hello() { echo boing=$BOING; }
$ BOING=123 hello
boing=123
$ echo $BOING
123
$
and ksh, since that seems to come up on this list:
$ ksh
~ [1]$ hello() { echo boing=$BOING; }
~ [2]$ BOING=123 hello
boing=123
~ [3]$ echo $BOING
123
~ [4]$
zsh also agrees. so it seems like bash is the exception (but bash is
also the most useful?).
Re: pthread_t and thrd_t
i think the requirement is only that the compiler accept -lpthread... "It is unspecified whether the libraries libc.a, libl.a, libm.a, libpthread.a, librt.a, [OB] [Option Start] libtrace.a, [Option End] libxnet.a, or liby.a exist as regular files. The implementation may accept as -l option-arguments names of objects that do not exist as regular files." Android has libpthread and librt in libc (though sadly libm is still separate). On Thu, Apr 18, 2019 at 8:46 AM Joel Sherrill wrote: > > > > On 4/18/2019 4:31 AM, Geoff Clare wrote: > > Shware Systems wrote, on 17 Apr 2019: > >> On Wednesday, April 17, 2019 Geoff Clare wrote: > >> > >>> ... does anyone know of an implementation > >>> that has pthreads and C11/C17 threads, with pthread_t and thrd_t defined > >>> as different types? > >> My implementation defines both as pointers, not int's, as 'typedef > >> thrd_t pthread_t;'. Since threads.h is 99% a subset of pthreads.h, > >> I see no reason to have them be different types, even as a breaking > >> change. What is missing is in pthreads.h, to account for the new > >> attributes C11 introduced, at least to mention them if support > >> isn't going to be added. > > What new attributes? As far as I can see the only attribute a > > thread has in C17 (although it doesn't use that word) is whether it > > has been detached. > > > >> The major hassle I see is POSIX wants the > >> the threading stuff in libpthread, and C11 wants it in libc if the > >> threading option is enabled. IMO C11 should have let interfaces in > >> optional headers all be in header-specific libraries or libc, as an > >> implentation-defined choice, but they didn't. > > Last I knew the C standard says nothing at all about how programs > > are built. > > Is there even a requirement to have libpthread in POSIX? > > --joel >
Re: pthread_t and thrd_t
for Android, i've been sitting this one out waiting for it to actually matter (i wasn't aware glibc had added this API --- i thought musl was the only implementation to date), but my intention is to just reuse the pthread types, not least because that lets me offer this in a backwards compatible way to old versions of Android that shipped before this stuff gets added. (it's not in Android Q.) looks like while i was sleeping FreeBSD implemented it too: https://github.com/freebsd/freebsd/tree/master/lib/libstdthreads /me wonders why no-one seems to have used pthread_key_t for tss_t, even when they have reused the pthreads types for the other stuff... On Wed, Apr 17, 2019 at 8:59 AM Geoff Clare wrote: > > I'm working on drafting the changes to align POSIX with C17 and I > have a question for the group about the types pthread_t and thrd_t. > > It seems to me that it would be best if these are defined as the > same type, so that there are no complications with calling both > pthread_self() and thrd_current() from: > > * The initial thread. > > * Threads created using pthread_create(). > > * Threads created using thrd_create(). > > * Threads created via a SIGEV_THREAD notification. > > If pthread_t and thrd_t were allowed to be different types, I > suppose this could be made to work by giving the initial thread > two IDs, one of each type, and forbidding calls to pthread_self() > from threads created using thrd_create() or calls to thrd_current() > from threads created using pthread_create() or via a SIGEV_THREAD > notification. However, this would be a nightmare for third-party > libraries if they want to use pthread_self() or thrd_current() in > a thread they don't create themselves. (I suppose they would have > to have two flavours of the library, one for pthreads and one for > C11/C17 threads.) > > I had a look at the current glibc source, and was initially alarmed > to see that it defines thrd_t as unsigned long int, since I was > sure I could remember pthread_t being a structure pointer, but I > see that it now has pthread_t as unsigned long int as well: > > https://sourceware.org/git/?p=glibc.git;a=blob_plain;f=sysdeps/nptl/threads.h;hb=HEAD > > https://sourceware.org/git/?p=glibc.git;a=blob_plain;f=sysdeps/nptl/bits/pthreadtypes.h;hb=HEAD > > So glibc will conform if we require that pthread_t and thrd_t are > defined as the same type, but does anyone know of an implementation > that has pthreads and C11/C17 threads, with pthread_t and thrd_t defined > as different types? > > -- > Geoff Clare > The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England >
Re: C++ condition_variable and timed_mutex with steady_clock and pthreads
On Sat, Oct 6, 2018 at 12:09 PM Mike Crowe wrote: > > On Wednesday 12 September 2018 at 13:47:59 -0700, Tom Cherry wrote: > > On Wed, Sep 12, 2018 at 11:56 AM Mike Crowe > > wrote: > > > > > > On Wednesday 12 September 2018 at 11:29:26 -0700, Tom Cherry wrote: > > > > On Sun, Sep 9, 2018 at 8:28 AM Mike Crowe > > > > wrote: > > > > > > > > > > On Thursday 30 August 2018 at 21:10:48 +0100, Mike Crowe wrote: > > > > > > C++11's std::condition_variable, std::timed_mutex and > > > > > > std::recursive_timed_mutex support waiting with a timeout specified > > > > > > using > > > > > > an arbitrary clock. It's common to use std::chrono::steady_clock > > > > > > (which > > > > > > corresponds to CLOCK_MONOTONIC) or std::chrono::system_clock (which > > > > > > corresponds to CLOCK_REALTIME) for these waits, and other clocks > > > > > > end up > > > > > > being converted to one of those when performing the wait. > > > > > > > > > > > > Unfortunately, I don't believe that it's possible to implement > > > > > > these on top > > > > > > of the pthread equivalents correctly for std::chrono::steady_clock > > > > > > (i.e. > > > > > > CLOCK_MONOTONIC) since: > > > > > > > > > > > > 1. pthread_mutex_timedlock only supports a time measured against > > > > > >CLOCK_REALTIME. > > > > > > > > > > > > 2. The clock used for pthread_cond_timedwait must be specified when > > > > > > the > > > > > >condition variable is created by pthread_cond_init. The clock to > > > > > > be used > > > > > >for future waits is not known at the point that the > > > > > >std::condition_variable constructor is called. > > > > > > > > > > > > I'm most interested in the std::condition_variable case. since I > > > > > > have code > > > > > > that wants to wait using std::chrono::steady_clock. > > > > > > > > > > > > There are a number of possible workarounds for these. I've > > > > > > described some > > > > > > in a blog post[1] and in defect 0001164[2]. But, my favourite > > > > > > solution is > > > > > > to introduce a variant of pthread_cond_timedwait that takes a an > > > > > > additional > > > > > > clockid_t parameter: > > > > > > > > > > > > int pthread_cond_timedwaitonclock(pthread_cond_t *cond, > > > > > >pthread_mutex_t *mutex, > > > > > >clockid_t clockid, > > > > > >const struct timespec > > > > > > *abstimeout); > > > > > > > > > > > > I've proposed[3] an implementation of this function for glibc (as > > > > > > pthread_cond_timedwaitonclock_np) and it was suggested that I ought > > > > > > to > > > > > > raise it here. (This led me to enter defect 0001164, but since that > > > > > > yielded > > > > > > no response I've finally got round to writing this email too.) > > > > > > > > > > > > The equivalent for mutex would probably be: > > > > > > > > > > > > int pthread_mutex_timedlockonclock(pthread_mutex_t *mutex, > > > > > > clockid_t clockid, > > > > > > const struct timespec > > > > > > *abstimeout); > > > > > > > > > > > > but I've not yet made any attempt to implement it in glibc. > > > > > > > > > > > > Would making the C++ standard library implementable on top of POSIX > > > > > > be > > > > > > considered a good enough reason to add such functions? > > > > > > > > > > > > Are these functions generic enough or perhaps too generic? Android > > > > > > had a > > > > > > pthread_cond_timedwait_monotonic function for a while, but > > > > > > deprecated it > > > > > > when they added support for pthread_condattr_setclock. > > > > > > > > We (libc team for Android) have observed this problem too and have > > > > seen it cause issues on multiple occasions, enough times that we > > > > actually submitted a workaround that always converts the input > > > > timespec to CLOCK_MONOTONIC and waits with that clock[1]. Of course > > > > this only ameliorates the problem, but does not completely solve it, > > > > since there is still a period in time before the conversion to > > > > CLOCK_MONOTONIC happens, that if CLOCK_REALTIME changes abruptly, > > > > we're back to the same bad behavior as before. > > > > > > > > For this reason, we added pthread_cond_timedwait_monotonic_np() back > > > > in the P release of Android[2] along with equivalent functions for the > > > > rest of the timed waits. I've also pushed various patches to libc++, > > > > one that builds upon these changes to our libc[3] and one that > > > > implements std::condition_variable from scratch, borrowing the > > > > implementation from our libc[4], however neither made progress. > > > > > > > > As for having a separate clock id parameter, we thought about that > > > > during the review of the above changes, but we both thought that 1) > > > > there's no strong reason to wait on any clock other than > > > > CLOCK_MONOTONIC for these functions and 2) the
Re: pthread_kill with signal 0
(like that guy the other day, my austin group username/password doesn't seem to work for "mantis", only for the download site...) On Sat, Sep 29, 2018 at 9:31 PM enh wrote: > > http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_kill.html > says: > > "As in kill(), if sig is zero, error checking shall be performed but no > signal shall actually be sent." > > but ESRCH isn't listed in ERRORS, only EINTR (presumably because of "Austin > Group Interpretation 1003.1-2001 #142 is applied, removing the [ESRCH] error > condition."). > > then, later: > > "A future version of this standard may require that pthread_kill() not fail > with [ESRCH] in the case of sending signals to an inactive thread (a > terminated thread not yet detached or joined), even though no signal will be > delivered because the thread is no longer running." > > which presumably should already have been removed as part of #142, since this > is already the first mention of ESRCH on the page? > > but what i'm here for today is to suggest that the "As in kill()" paragraph > be removed too... it makes sense for kill(2) because the argument is a pid_t, > but for pthread_kill(3) the pthread_t argument is basically a pointer (in > every implementation i'm aware of), which means there's no safe way to > implement this short of a global lock on a global thread list, which it seems > unreasonable for POSIX to require. > > [in Android we had so much trouble with apps causing chaos by not tracking > pthread_t lifetime that we've added FORTIFY-like checking that aborts a > process that hands an invalid pthread_t to any libc function. in the docs > (https://android.googlesource.com/platform/bionic/+/master/docs/status.md#invalid-handling-targetsdkversion-o) > we've said "The best alternative on Android is to use pthread_gettid_np at > some point that you know the thread to be alive, and then call kill/tgkill > with signal 0 (which checks whether a process exists rather than actually > sending a signal). That’s still not completely safe because if you're too > late the tid may have been reused, but your code is inherently unsafe without > a redesign anyway.".] > > anyway, since POSIX currently seems to say: > > 1. you can pass signal 0 ... > 2. but the function can't return ESRCH to tell you the thread doesn't exist > 3. so your code can't portably assume the return value means anything > > it seems to me that all we're doing here is confusing developers, and we > should remove the "if sig is zero" paragraph?
pthread_kill with signal 0
http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_kill.html says: "As in kill(), if sig is zero, error checking shall be performed but no signal shall actually be sent." but ESRCH isn't listed in ERRORS, only EINTR (presumably because of "Austin Group Interpretation 1003.1-2001 #142 is applied, removing the [ESRCH] error condition."). then, later: "A future version of this standard may require that pthread_kill() not fail with [ESRCH] in the case of sending signals to an inactive thread (a terminated thread not yet detached or joined), even though no signal will be delivered because the thread is no longer running." which presumably should already have been removed as part of #142, since this is already the first mention of ESRCH on the page? but what i'm here for today is to suggest that the "As in kill()" paragraph be removed too... it makes sense for kill(2) because the argument is a pid_t, but for pthread_kill(3) the pthread_t argument is basically a pointer (in every implementation i'm aware of), which means there's no safe way to implement this short of a global lock on a global thread list, which it seems unreasonable for POSIX to require. [in Android we had so much trouble with apps causing chaos by not tracking pthread_t lifetime that we've added FORTIFY-like checking that aborts a process that hands an invalid pthread_t to any libc function. in the docs ( https://android.googlesource.com/platform/bionic/+/master/docs/status.md#invalid-handling-targetsdkversion-o) we've said "The best alternative on Android is to use pthread_gettid_np at some point that you know the thread to be alive, and then call kill/tgkill with signal 0 (which checks whether a process exists rather than actually sending a signal). That’s still not completely safe because if you're too late the tid may have been reused, but your code is inherently unsafe without a redesign anyway.".] anyway, since POSIX currently seems to say: 1. you can pass signal 0 ... 2. but the function can't return ESRCH to tell you the thread doesn't exist 3. so your code can't portably assume the return value means anything it seems to me that all we're doing here is confusing developers, and we should remove the "if sig is zero" paragraph?
Re: Minutes of the 6th September 2018 Teleconference
On Sat, Sep 8, 2018 at 2:33 AM Andrew Josey wrote: > > > All > Enclosed are the minutes of this weeks call > regards > Andrew > > > > Minutes of the 6th September 2018 Teleconference Austin-884 Page 1 of 1 > Submitted by Andrew Josey, The Open Group. 8th September 2018 > > Attendees: > > Mark Ziegast, SHware Systems Dev. > Joerg Schillng, FOKUS Fraunhofer > Don Cragun ,IEEE PASC OR > Nick Stoughton, USENIX, ISO/IEC JTC 1/SC 22 OR > Geoff Clare, The Open Group > Martin Rehak, Oracle, The Open Group OR > Andrew Josey, The Open Group (partial) > Eric Blake, Red Hat > Drew DeVault, independent, (partial) just listening (and does not have a > microphone) > > Apologies: > David Clissold, IBM > > * General news > > A quick update on the status of the pipefail discussion from the mailing list. > A question of whether David Korn is still reachable, rumour has it that he > has now retired. > > Eric raised a request from MUSL libc for us to review bug id 900 (qsort_r) on > a future call. fwiw, although qsort_r has come up a few times for Android, we've avoided adding it precisely because of the two incompatible versions. because Android is a system with a Linux kernel there's some benefit to adopting GNU extensions, but because app developers are far more likely to care about iOS than other Linuxes, we have a lot of BSD extensions too. and cases like this, where BSD and GNU used the same name for different functions leave us in a bit of a bind. it's a pity that Annex K is such a mess. qsort_s would have solved this problem, but as long as no-one else is implementing Annex K, i'm not excited about Android implementing the few bits that do add some value such as qsort_s and scanf_s. > * Outstanding actions > > (Please note that this section has been flushed to shorten the minutes - > to locate the previous set of outstanding actions, look to the minutes > from 9 March 2018 and earlier) > > * Current Business > > Bug 1077: Recommend support for wide-character regcomp and regexec and/or > specify multi-byte behavior OPEN > http://austingroupbugs.net/bug_view_page.php?bug_id=1077 > > Andrew has completed the action to ping his Apple contact and is > awaiting a reply. > > Bug 1122: POSIX should include gettext() and friends OPEN > http://austingroupbugs.net/view.php?id=1122 > > Left open as an action is still in progress to flesh out a complete proposal. (while i'm here, i'd like to support what bhaible said in comments 0004080 and 0004081.) > Bug 1130: Address 0 does not make sense for the c command Accepted as Marked > http://austingroupbugs.net/view.php?id=1130 > > This item is tagged for TC3-2008 > > Suggested change: On page 2682, line 87448, delete "Address 0 shall > be valid for this command; it shall be interpreted as if address 1 > were specified." > > On page 2691, lines 87803-87805, change from: >For consistency with the a and r commands and better user >functionality, the i and c commands must also accept an address >of 0, in which case 0i is treated as 1i and likewise for the c >command. > to: > > For consistency with the a and r commands and better user > functionality, the i command also accepts an address of 0. > However, it is unspecified if 0i is treated as 1i > (which will fail if the buffer is empty), or means insert at > the beginning of the buffer (which will succeed even if the > buffer is empty). Earlier versions of this standard required > address 0 for the c command to be treated as 1 also, but this > requirement has been removed, though implementations are permitted > to do this as an extension. > > > Bug 1131: The synopsis of the i command is wrong and inconsistent with the > synopsis of the a command Accepted as marked. > http://austingroupbugs.net/view.php?id=1131 > > This bug is tagged for TC3-2008. > > On page 2684 line 87533 section ed, change: > Address 0 shall be valid for this command; it shall be interpreted > as if address 1 were specified. > to: > Address 0 shall be valid for this command; it is unspecified > whether it causes the inserted text to be placed at the beginning > of the buffer or it is interpreted as if address 1 were specified. > (These two allowed behaviors differ in the case that the buffer > is empty.) > > > Next Steps > -- > The next call is on Sep 13th 2018 (Thursday). > > Apologies in Advance: > Nick Stoughton (2018-09-20) > Andrew Josey (tentative 2018-09-13) > > Calls are anchored on US time. (8am Pacific) > This call will be for the regular 90 minutes. > > http://austingroupbugs.net > > An etherpad is usually up for the meeting, with a URL using the date format > as below: > > https://posix.rhansen.org/p/201x-mm-dd > username=posix password=2115756# > > > Andrew JoseyThe Open Group > Austin Group Chair > Email: a.jo...@opengroup.org > Apex Plaza, Forbury Road,Reading,Berks.RG1
