Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Shware Systems]

By process private I mean a method similar to how pax avoids name collisions 
when unpacking global extended headers as separate files, using the process id 
as part of the name of a subdirectory under $TMPDIR to hold those extractions 
as the default template. This convention isn't completely robust, but is 
adequate for the majority of scenarios.
There isn't an option for mkdir(), creat() or open() to limit accesses like 
this for any name. Only a few file system designs have the hooks to add a 
facility like this in a backwards compatible fashion anyways, so it's unlikely 
to become an option.

Atomically applies to the full sequence, before to the parts.

On Saturday, April 1, 2017 Martijn Dekker  wrote:

Op 29-03-17 om 18:30 schreef shwares...@aol.com:
> I agree something along these lines should be added. As worded now the
> assumption looks to be more the system is a single user one, or
> sandboxed to that effect, not a multi-user server.

Yes.

> It doesn't even have to be a different user; a separate process that
> happens to use the same tmpnam() template for a FIFO creation might
> cause a race condition.

This is true, but I figured a user who has nefarious processes running
under their own account has bigger problems to worry about than -o
noclobber.

> On some systems stdin and stdout may be implemented as FIFOs, so a
> shell could be expected to access them along with regular files in
> redirects. I'd have it be process private directories, not simply
> user private,

How does one go about making these in POSIX?

> therefore, or that non-regular files like FIFOs are required to be
> atomically unlinked before attempting the regular file creation.

I think "atomically" and "before ..." are mutually exclusive.

- M.

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Martijn Dekker]

Op 29-03-17 om 18:30 schreef shwares...@aol.com:
> I agree something along these lines should be added. As worded now the
> assumption looks to be more the system is a single user one, or
> sandboxed to that effect, not a multi-user server.

Yes.

> It doesn't even have to be a different user; a separate process that
> happens to use the same tmpnam() template for a FIFO creation might
> cause a race condition.

This is true, but I figured a user who has nefarious processes running
under their own account has bigger problems to worry about than -o
noclobber.

> On some systems stdin and stdout may be implemented as FIFOs, so a
> shell could be expected to access them along with regular files in
> redirects. I'd have it be process private directories, not simply
> user private,

How does one go about making these in POSIX?

> therefore, or that non-regular files like FIFOs are required to be
> atomically unlinked before attempting the regular file creation.

I think "atomically" and "before ..." are mutually exclusive.

- M.

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[SHwareSyst]

I agree something along these lines should be added. As worded now the  
assumption looks to be more the system is a single user one, or  sandboxed to 
that effect, not a multi-user server. It doesn't even have to be a  different 
user; a separate process that happens to use the same tmpnam()  template 
for a FIFO creation might cause a race condition. On some systems stdin  and 
stdout may be implemented as FIFOs, so a shell could be expected to access  
them along with regular files in redirects. I'd have it be process private  
directories, not simply user private, therefore, or that non-regular files 
like  FIFOs are required to be atomically unlinked before attempting the 
regular file  creation.
 
 
In a message dated 3/28/2017 9:00:43 P.M. Eastern Daylight Time,  
nore...@msnkbrown.net writes:


A  NOTE has been added to this issue.  
==  
http://austingroupbugs.net/view.php?id=1016  
==  
Reported By: izabera
Assigned To: 
==  
Project:   1003.1(2013)/Issue7+TC1
Issue ID:   1016
Category: Shell and Utilities
Type:Enhancement Request
Severity: Editorial
Priority: normal
Status:   Interpretation  Required
Name: Isabella 
Organization:   --- 
User Reference:   --- 
Section:   2.7.2 
Page Number:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag
_18_07_02

Line  Number:last paragraph  
Interp Status:  Approved  
Final Accepted Text: 
http://austingroupbugs.net/view.php?id=1016#c3493  
==  
Date Submitted: 2015-12-28  13:52 UTC
Last Modified:   2017-03-29 00:58  UTC
==  
Summary:   race condition with set  -C
==  

--  
(0003655) McDutchie (reporter) - 2017-03-29  00:58
http://austingroupbugs.net/view.php?id=1016#c3655  
--  
"Output redirection using the '>' format shall fail if the noclobber  option
is set (see the description of set -C) and the file named by the  expansion
of word exists and is either a regular file or a symbolic link  that
resolves to a regular file." [...] "Performing these operations  atomically
ensures that the creation of lock files and unique (often  temporary) files
is reliable."

If noclobber doesn't block on writing  to an existing FIFO or device file,
then an attacker on another user  account, if they guess the name before the
file is created, can subvert the  creation of a unique temporary file in a
shared directory (e.g. /tmp) by  putting a FIFO or device file in the way.
With a FIFO they could even steal  some data. So the statement about
reliable unique temporary files is  accurate only for private directories.
You may want to add this for  clarity. 

Issue History 
Date ModifiedUsername   Field   Change  
==  
2015-12-28 13:52 izaberaNew Issue  
2015-12-28 13:52 izabera Name => Isabella
2015-12-28  13:52 izaberaOrganization   => ---   
2015-12-28 13:52 izaberaUser  Reference=> --- 
2015-12-28 13:52 izabera Section => 2.7.2
2015-12-28 13:52 izaberaPage Number=>
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag
_18_07_02
2015-12-28  13:52 izaberaLine Number   => last paragraph  
2015-12-31  05:58 shware_systems Note Added: 0002990   
2015-12-31 12:52  jilles Note Added: 0002991  
2015-12-31 14:21 shware_systems Note Added: 0002992
2015-12-31 17:05 nick   Interp  Status => --- 
2015-12-31 17:05 nick Note Added: 0002993 
2015-12-31  17:05 nick   Description Updated
2016-10-20 16:40 geoffclare Note Added:  0003446 
2016-10-31 16:23 geoffclare Note  Added: 0003481   
2016-10-31 16:28 geoffclare  Note Edited: 0003481 
2016-10-31 16:33 geoffclare   Note Edited: 0003481 
2016-11-10 17:33  rhansenNote Added: 0003485
2016-11-10 17:36 rhansenNote Edited:  0003485 
2016-11-10 17:37 rhansen Note Edited: 0003485 
2016-11-11 15:24 rhansen Note Edited: 0003485 
2016-11-11 15:25  rhansenNote Edited: 0003485

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Garrett Wollman]

< said:

> Also, on Mac OS X and FreeBSD, 'link' turns out to act like 'ln': if a
> directory or a symlink to it exists, the file is created inside it. This
> is contrary to what the man page says, by the way. But in any case it
> means I can't use it.

Actually, no:

 When the utility is called as link, exactly two arguments must be sup-
 plied, neither of which may specify a directory.

What the text in the manual page says, albeit obliquely, is that the
results are unspecified if either argument to link(1) resolves to a
directory.  It happens to behave the same as ln(1) in this case, but
this is an artifact of implementation.

I agree that this seems to violate the standard, but FreeBSD, unlike
Mac OS, does not bear Unix branding and does not claim to implement
the XSI option -- the fact that there is a link(1) utility does not
mean that it is the XSI link(1) utility.

-GAWollman

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Shware Systems]

No, there is no app dir structure mandated
by POSIX. That is considered an administrative issue outside of the base POSIX 
scope, and is generally covered by packaging conventions like DEB and RPM as de 
facto standards. I don't think any platform implemented the proposed 1003.1j 
standard. Only /., /.., /dev and /tmp are required as fixed root directory 
names, per XBD 10. Each vendor can use whatever paths they prefer, besides 
those 4, for the symbolic names listed in XBD 8, such as $HOME or $LIB. In an 
embedded system a vendor might well put all utilities and shared libraries 
directly into / on a ROM and mount /tmp as an alias of a /dev/NVRAMFS device 
subdir.

As to mktemp, as long as it only creates files of regular type, or parent 
directories of one, there shouldn't be any problem I see with it using /tmp 
since ln is required to handle both those types. A platform that adds, as a 
contrived example, a "mimo" file type as a scatter/gather superset of fifos 
abstraction for persisting iovec_t templates, might need guard code for use of 
the complementary mkmimo utility in a script, rather than emulating the 
functionality with multiple use or redirects of fifos to a regular file in a cp 
pipe or for loop, as the portable option for a gather operation.

On Monday, November 7, 2016 Martijn Dekker  wrote:

Op 07-11-16 om 03:55 schreef Shware Systems:
> To last question, yes, but the effects are supposed to be documented so
> generic guard code that may invoke platform specific pre-ln attempt
> handling can be written. This is a compromise to disqualifying a system
> that defines additional file types from being considered conforming at
> all. In a script this might look like:
> if [ -e /app/$platform/linkhandler ] ;
> then { . .../linkhandler }
> else { do ln directly }; fi

Thanks for the reply. Where can I find more info about this? Is there a
standardised /app directory structure? I don't find it on any actual
system I've access to.

> To some extent it's also the operator's responsibility to sandbox use of
> non-standard file types outside directories the standard says portable
> applications need access to, such as $TMP, to avoid issues.

That makes sense. Many things would break if /tmp does not operate portably.

> A platform
> aware application might create such a file in a $TMP/appsubdir directory
> but shouldn't link it into /tmp after, iow, but to an ~/app/files type
> directory instead. That is more a training issue to me, not something
> the standard can reasonably address or make a requirement.

Unfortunately, by far the most common use case of 'mktemp' is to create
a temporary file in /tmp, so my cross-platform shell implementation of
it will have to be able to do that.

For the time being, unless anyone has concrete evidence or convincing
arguments to the contrary, I will assume that any issues with 'ln'
atomicity into /tmp are theoretical.

- M.

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Geoff Clare]

Stephane Chazelas  wrote, on 07 Nov 2016:
>
> 2016-11-07 16:57:34 +, Stephane Chazelas:
> [...]
> > OK, sorry, I had assumed rename() would fail if the target exits
> > already.
> [...]
> 
> BTW, in the spec of link(2):
> 
>  [EEXIST]
>   The path2 argument resolves to an existing
> directory entry or refers to a symbolic link.
> 
> Why the "or refers to a symbolic link"? That would still be a
> directory entry.

Because it says "resolves".  Pathname resolution follows symlinks
by default.  So if it did not mention symlinks explicitly here,
a symlink with a non-existent target would not cause EEXIST.

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Geoff Clare]

Stephane Chazelas  wrote, on 07 Nov 2016:
>
> 2016-11-07 16:20:08 +, Geoff Clare:
> [...]
> > > How so? "mv -i" with /dev/null as stdin ("no" answer to prompt)
> > > is not supposed to remove anything.
> > 
> > Only if it prompts.  If the existence check fails, mv will not
> > prompt and will just call rename().
> 
> OK, sorry, I had assumed rename() would fail if the target exits
> already.
> 
> So that's another "broken" command then if it can end-up removing
> the target without asking the user despite the fact "-i" was
> passed?
> 
> I suppose "mv" could use "link(src, dst) && unlink(src)" here to
> work around that (like it does copy+unlink when across file
> systems).

That would just exchange one race condition for another, although
I suppose it would be a smaller time window between link() and
unlink() than the one between the existence check and calling
rename() - particularly if the user takes time to answer the
prompt.

> Is that allowed by POSIX?

No; the standard clearly says "The mv utility shall perform actions
equivalent to the rename() function" in step 3.

> Do any implementations do it?

Only museum pieces from before rename() existed (I assume).

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Stephane Chazelas]

2016-11-07 16:57:34 +, Stephane Chazelas:
[...]
> OK, sorry, I had assumed rename() would fail if the target exits
> already.
[...]

BTW, in the spec of link(2):

 [EEXIST]
  The path2 argument resolves to an existing
  directory entry or refers to a symbolic link.

Why the "or refers to a symbolic link"? That would still be a
directory entry.

-- 
Stephane

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Stephane Chazelas]

2016-11-07 16:20:08 +, Geoff Clare:
[...]
> > How so? "mv -i" with /dev/null as stdin ("no" answer to prompt)
> > is not supposed to remove anything.
> 
> Only if it prompts.  If the existence check fails, mv will not
> prompt and will just call rename().

OK, sorry, I had assumed rename() would fail if the target exits
already.

So that's another "broken" command then if it can end-up removing
the target without asking the user despite the fact "-i" was
passed?

I suppose "mv" could use "link(src, dst) && unlink(src)" here to
work around that (like it does copy+unlink when across file
systems).

Is that allowed by POSIX? Do any implementations do it?

-- 
Stephane

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Geoff Clare]

Stephane Chazelas  wrote, on 07 Nov 2016:
>
> BTW, there's an issue in the spec for "mv":
> 
> > EXIT STATUS
> >
> >  The following exit values shall be returned:
> >
> >   0
> >  All input files were moved successfully.
> >  >0
> >  An error occurred.
> 
> In
> 
> mv -i a b
> 
> if the user says "no", "a" will not be moved successfully, and
> there will not have been any error.

Good catch.

> Should probably be something like:
> 
>    0
> All input files (approved by the user with -i) were
> moved successfully.

We should change it to match rm, which says:

Each directory entry was successfully removed, unless its removal
was canceled by a non-affirmative response to a prompt for
confirmation.

> Also, should failure to write the prompt or read the answer be
> considered an error?

I would say yes.

> Using:
> 
>mv -i a b  2>&- <&-
> 
> or
> 
>mv -i a b < /dev/null 2>&-

However, that's not a valid test because of the following text on the
execl() page:

If file descriptor 0, 1, or 2 would otherwise be closed after
a successful call to one of the exec family of functions,
implementations may open an unspecified file for the file descriptor
in the new process image. If a standard utility or a conforming
application is executed with file descriptor 0 not open for reading
or with file descriptor 1 or 2 not open for writing, the environment
in which the utility or application is executed shall be deemed
non-conforming, and consequently the utility or application might
not behave as described in this standard.

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

RE: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Schwarz, Konrad]

> -Original Message-
> From: Geoff Clare [mailto:g...@opengroup.org]
> Sent: Monday, November 07, 2016 5:20 PM
> To: austin-group-l@opengroup.org
> Subject: Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set
> -C
> > That seems to be in contradiction with it calling the link() system
> > call.
> >
> > I suspect that text is there for attempts to call "link" on a
> > directory.
> 
> If that were the case, I would have expected it to be worded like
> Joerg's man page quote.  Instead it just says "A user may need
> appropriate privileges to invoke the link utility."

When do you call link(1) in lieu of ln(1)?

BTW: the rationale for ln says:
This volume of POSIX.1-2008 does not allow the ln utility
to unlink existing destination paths by default for the
following reasons:

The ln utility has historically been used to provide locking for
shell applications, a usage that is incompatible with ln unlinking
the destination path by default.  There was no corresponding
technical advantage to adding this functionality.

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Geoff Clare]

Stephane Chazelas  wrote, on 07 Nov 2016:
>
> 2016-11-07 15:40:15 +, Geoff Clare:
> [...]
> > > Same problem with "mv" (which I think would work just
> > > as well (with LC_ALL=C mv -i < /dev/null 2> /dev/null))
> > 
> > No, mv -i doesn't work just as well - it has a race condition.
> > If a file is created in between the existence check and the
> > rename() call, mv will remove the file.
> 
> How so? "mv -i" with /dev/null as stdin ("no" answer to prompt)
> is not supposed to remove anything.

Only if it prompts.  If the existence check fails, mv will not
prompt and will just call rename().

> > > You could use "link" (Unix, not POSIX), or "ln -T" (GNU, not
> > > POSIX) or "mv -Tn" (GNU) instead.
> > 
> > The standard allows systems to make "link" available only to
> > processes with appropriate privileges, so that solution might
> > not be sufficiently portable.
> [...]
> 
> That seems to be in contradiction with it calling the link()
> system call.
> 
> I suspect that text is there for attempts to call "link" on a
> directory.

If that were the case, I would have expected it to be worded like
Joerg's man page quote.  Instead it just says "A user may need
appropriate privileges to invoke the link utility."

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Stephane Chazelas]

2016-11-07 16:10:01 +, Stephane Chazelas:
[...]
>mv -i a b < /dev/null 2>&-
> 
> seem to work with GNU or Solaris10 mv (in that it returns with
> an error when a prompt fails to be issued) but not with
> FreeBSD's
[...]

Sorry, I messed up my tests on Solaris. Solaris /bin/mv doesn't
issue a prompt when stdin is not a terminal (and renames the
file!). I did observe a non-zero exit status with that one but
that was because I had renamed the file in an earlier test.

/usr/xpg4/bin/mv behaves POSIXly, but still exits with 0 if the
prompt can't be issued or the answer can't be read.

-- 
Stephane

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Stephane Chazelas]

2016-11-07 15:57:25 +, Stephane Chazelas:
> 2016-11-07 15:40:15 +, Geoff Clare:
> [...]
> > > Same problem with "mv" (which I think would work just
> > > as well (with LC_ALL=C mv -i < /dev/null 2> /dev/null))
> > 
> > No, mv -i doesn't work just as well - it has a race condition.
> > If a file is created in between the existence check and the
> > rename() call, mv will remove the file.
> 
> How so? "mv -i" with /dev/null as stdin ("no" answer to prompt)
> is not supposed to remove anything.
[...]

However, at least with GNU mv, the exit code if the file exists
will be 0. So it can't be used here.

BTW, there's an issue in the spec for "mv":

> EXIT STATUS
>
>  The following exit values shall be returned:
>
>   0
>  All input files were moved successfully.
>  >0
>  An error occurred.

In

mv -i a b

if the user says "no", "a" will not be moved successfully, and
there will not have been any error.

Should probably be something like:

   0
  All input files (approved by the user with -i) were
  moved successfully.

Also, should failure to write the prompt or read the answer be
considered an error?

Using:

   mv -i a b  2>&- <&-

or

   mv -i a b < /dev/null 2>&-

seem to work with GNU or Solaris10 mv (in that it returns with
an error when a prompt fails to be issued) but not with
FreeBSD's

-- 
Stephane

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Stephane Chazelas]

2016-11-02 13:32:44 +, Martijn Dekker:
[...]
> If both 'mkdir' and 'ln' operate atomically, there could be a safe
> workaround for creating a regular file directly under /tmp. It would
> involve creating a (very) temporary directory under /tmp using 'mkdir
> -m700', then creating the file inside there, setting the mode, etc. with
> no need for atomicity, then attempting to 'ln' that file back to /tmp
> until we've got an available name. Do you think this could work?
[...]

I don't think you can use ln here.

ln "$tempdir/file" "$tempfile"

would create a "$tempfile/file" link if "$tempfile" existed and
was of type directory or a symlink eventually resolving to a
directory. Same problem with "mv" (which I think would work just
as well (with LC_ALL=C mv -i < /dev/null 2> /dev/null))

It would not clobber a file but could create one in unwanted
places like /etc/profile.d or /var/spool/cron/crontabs or just
/tmp/foo/ where the attacker could replace it with his own one.

You could use "link" (Unix, not POSIX), or "ln -T" (GNU, not
POSIX) or "mv -Tn" (GNU) instead.

-- 
Stephane

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Martijn Dekker]

Op 07-11-16 om 03:55 schreef Shware Systems:
> To last question, yes, but the effects are supposed to be documented so
> generic guard code that may invoke platform specific pre-ln attempt
> handling can be written. This is a compromise to disqualifying a system
> that defines additional file types from being considered conforming at
> all. In a script this might look like:
> if [ -e /app/$platform/linkhandler ] ;
> then { . .../linkhandler }
> else { do ln directly }; fi

Thanks for the reply. Where can I find more info about this? Is there a
standardised /app directory structure? I don't find it on any actual
system I've access to.

> To some extent it's also the operator's responsibility to sandbox use of
> non-standard file types outside directories the standard says portable
> applications need access to, such as $TMP, to avoid issues.

That makes sense. Many things would break if /tmp does not operate portably.

> A platform
> aware application might create such a file in a $TMP/appsubdir directory
> but shouldn't link it into /tmp after, iow, but to an ~/app/files type
> directory instead. That is more a training issue to me, not something
> the standard can reasonably address or make a requirement.

Unfortunately, by far the most common use case of 'mktemp' is to create
a temporary file in /tmp, so my cross-platform shell implementation of
it will have to be able to do that.

For the time being, unless anyone has concrete evidence or convincing
arguments to the contrary, I will assume that any issues with 'ln'
atomicity into /tmp are theoretical.

- M.

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Shware Systems]

To last question, yes, but the effects are supposed to be documented so generic 
guard code that may invoke platform specific pre-ln attempt handling can be 
written.
This is a compromise to disqualifying a system that defines additional file 
types from being considered conforming at all. In a script this might look like:
if [ -e /app/$platform/linkhandler ] ; 
then { . .../linkhandler }
else { do ln directly }; fi

To some extent it's also the operator's responsibility to sandbox use of 
non-standard file types outside directories the standard says portable 
applications need access to, such as $TMP, to avoid issues. A platform aware 
application might create such a file in a $TMP/appsubdir directory but 
shouldn't link it into /tmp after, iow, but to an ~/app/files type directory 
instead. That is more a training issue to me, not something the standard can 
reasonably address or make a requirement.

On Sunday, November 6, 2016 Martijn Dekker  wrote:

Op 02-11-16 om 13:32 schreef Martijn Dekker:
> If both 'mkdir' and 'ln' operate atomically, there could be a safe
> workaround for creating a regular file directly under /tmp. It would
> involve creating a (very) temporary directory under /tmp using 'mkdir
> -m700', then creating the file inside there, setting the mode, etc. with
> no need for atomicity, then attempting to 'ln' that file back to /tmp
> until we've got an available name. Do you think this could work?

No one replied to poke holes in this, so I went ahead and implemented
this workaround in the modernish shell library implementation of
'mktemp'. 

Just one thing still worries me a bit. Though 'ln' without the -f option
is never supposed to overwrite files, the spec also states:

| If the last operand specifies an existing file of a type not
| specified by the System Interfaces volume of POSIX.1-2008, the
| behavior is implementation-defined.
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ln.html

Does this mean I cannot actually rely on 'ln' not overwriting a file or
otherwise behaving unexpectedly?

Thanks,

- M.

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Martijn Dekker]

Op 02-11-16 om 10:51 schreef Stephane Chazelas:
> At the moment, there's no way (that I know) to create a temp file
> reliably with POSIX utilities (though it may be possible to create a
> temp dir reliably, which is generally a better idea than creating a temp
> file in a world writable directory anyway).

Hmmm. Interesting.

(Just for clarity: below I'm still talking about creating temporary
files for any use as in 'mktemp', and *not* about locking.)

If both 'mkdir' and 'ln' operate atomically, there could be a safe
workaround for creating a regular file directly under /tmp. It would
involve creating a (very) temporary directory under /tmp using 'mkdir
-m700', then creating the file inside there, setting the mode, etc. with
no need for atomicity, then attempting to 'ln' that file back to /tmp
until we've got an available name. Do you think this could work?

The lack of a good POSIX shell way to create a temporary file has irked
me for years. So, as part of my cross-platform POSIX shell library
"modernish", I've created a shell implementation of 'mktemp' which is a
superset of currently available 'mktemp' implementations. To create
regular files, it currently uses the 'set -C'-plus-output-redirection
technique discussed earlier. At the time of writing it I was assuming it
must be safe, but it isn't, and (given the discussions here) it seems
uncertain if it will ever be safe in the future[*]. If the workaround
described above is good, I will have to implement it.

It also has an -F flag to create a temporary FIFO. Should 'mkfifo' be
considered atomic?

Thanks,

- Martijn

[*] I don't get all the objections. Is there any good reason for *not*
ensuring that noclobber is atomic? It's not as if modern operating
systems make this hard to implement. Why not err on the side of safety?

RE: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Schwarz, Konrad]

> -Original Message-
> From: Geoff Clare [mailto:g...@opengroup.org]
> Sent: Tuesday, November 01, 2016 11:27 AM
> To: austin-group-l@opengroup.org
> Subject: Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set
> -C
> 

> > Why are we bothering to attempt to make > (with -C) atomic just to
> > solve a problem that already has a better solution ?
> 
> The problem is not limited to lock files.  That's just being used as an
> example because it's the case where problems are most likely to occur.

Well, the central argument of http://austingroupbugs.net/view.php?id=1016 is 
locking:

"One common use of set -C is to implement a simple file locking mechanism,
but this is impossible to do safely."

I agree with Robert Elz that the issue should be resolved by referring
to link(2)/ln(1) -- which has been atomic in Unix for a long time --,
and possibly state in the standard that set -C may not be atomic.

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Joerg Schilling]

Stephane Chazelas  wrote:

> 2016-11-02 10:13:37 +, Geoff Clare:
> [...]
> > We believe that all modern shells are already using O_EXEC.  The one
> > notable shell that doesn't is ksh88 but I don't count that as modern,
> > since ksh93 is its modern replacement.
> [...]
>
> Has ksh93 ever been certified as a POSIX conformant shell?
> Actually, have any other shell than ksh88 and bash been
> certified?

Has bash really been certified? Isn't macos certified with the 
non-POSIX-compliant bash-3.x?

> I regard ksh93 as an "experimental/research shell". Many of the
> new features are not really finalised (as acknowledged by the
> authors in the doc), there are a few POSIX compliance issues and
> bugs...

ksh93 indeed has many minor deviations from the standard and I also have the 
impression that ksh93 is mainly used to develop new ideas rather then trying 
to be POSIX compliant as a whole.

A POSIX compliant Korn Shell would need to revert using some aliases in favor 
of real builtin-commands. See e.g. the "times" alias that cannot be used as a 
POSIX implementation, e.g. because the output format is not compliant.

BTW: David, could you please say something on the current status and the 
current development platform/location?

> isn't mkdir the canonical way to do mutual exclusion locks in
> shell scripts? IIRC there were issues with ln on NFS for
> instance.

Could you please explain what you have in mind?

I would like to understand whether there really is a NFS problem or whether 
there is just a NFS bug in Linux.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ 
http://sourceforge.net/projects/schilytools/files/'

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Geoff Clare]

Robert Elz  wrote, on 01 Nov 2016:
>
> Without getting into the specifics of the proposed change yet, I'd
> like someone to explain just what is the fascination with using -C
> and '>' for making lock files?
> 
> The time honoured way of making lock files (since way, way, back) is via
> the use of link(2) (or ln(1)) which has always provided for atomic creation
> of a file name in a perfectly safe, and easy to use manor.
> 
> Why are we bothering to attempt to make > (with -C) atomic just to solve a
> problem that already has a better solution ?

The problem is not limited to lock files.  That's just being used as an
example because it's the case where problems are most likely to occur.

> IMO, -C should go back to its original use (from csh where noclobber mode
> originated) and have its intended function being to avoid accidental loss
> of improtant data due to accidental incorrect use of >filename (with the
> wrong filename) and simplify the definition, rather than complicating it.

If '>' with set -C is not atomic, then it cannot be relied on to perform
this function either. 

The complicated definition (option 1) is only that complicated because
it allows existing practice.

Option 2 is much simpler, and I would prefer that, but we are
constrained by our rules about existing practice, so we need the other
option as a fallback.

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Robert Elz]

Without getting into the specifics of the proposed change yet, I'd
like someone to explain just what is the fascination with using -C
and '>' for making lock files?

The time honoured way of making lock files (since way, way, back) is via
the use of link(2) (or ln(1)) which has always provided for atomic creation
of a file name in a perfectly safe, and easy to use manor.

Why are we bothering to attempt to make > (with -C) atomic just to solve a
problem that already has a better solution ?

IMO, -C should go back to its original use (from csh where noclobber mode
originated) and have its intended function being to avoid accidental loss
of improtant data due to accidental incorrect use of >filename (with the
wrong filename) and simplify the definition, rather than complicating it.

kre

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Jilles Tjoelker]

On Thu, Oct 20, 2016 at 04:40:36PM +, Austin Group Bug Tracker wrote:
> 
> A NOTE has been added to this issue. 
> == 
> http://austingroupbugs.net/view.php?id=1016 
> == 
> [snip]
> -- 
>  (0003446) geoffclare (manager) - 2016-10-20 16:40
>  http://austingroupbugs.net/view.php?id=1016#c3446 
> -- 
> (Response to http://austingroupbugs.net/view.php?id=1016#c2991)

> The FreeBSD sh method described here does not behave correctly in the case
> where the initial stat() fails and then open() with O_EXCL fails, and the
> file that was created in between was not a regular file (e.g. a FIFO).  It
> needs to open the file (without O_CREAT or O_EXCL) and use fstat() to see
> if it is a regular file.

> You then have the problem of what to do if this open() fails. 

I can reproduce this with many shells (FreeBSD sh, bash 4.4.0 in both
non-POSIX and POSIX mode, yash 2.30, ksh93 93u+ 2012-08-01).

My reproducer consists of two processes:
sh -c 'while :; do ln -s /dev/null testf 2>/dev/null; done'
sh -c 'set -C; while true >testf; do rm testf; done'

After a few minutes, the redirection fails with File exists or a similar
error.

With zsh (5.2 in default mode), this command can run for more than half
an hour without failing. This is because zsh does not do an initial
stat(), but first tries an open() with O_EXCL and failing that an open
without O_CREAT or O_EXCL and an fstat() check that it is not a regular
file.

This approach fixes your problem and is slightly simpler, but there is
now a problem when a non-regular file is deleted between the two open()
calls.

NetBSD sh has a slightly different approach, trying without O_CREAT or
O_EXCL first (with an fstat() check that it is not a regular file) and
using O_CREAT | O_EXCL if the initial open fails. This appears to have
the same issue as the FreeBSD sh method.

I think the most important part of this bug report is that > with set -C
will open a regular file exclusively. Almost all shells seem to
implement this part correctly (although they did not 10 years ago).

-- 
Jilles Tjoelker