Re: nmh 1.8RC2, xlbiff, and $HOME
Ralph wrote: > This close to a release, I think we should stick with requiring HOME to > be non-empty if it's set as otherwise there's too many paths to consider > which the test harness probably doesn't exercise. I'd rather crank out an RC3 than pass up the opportunity to solidify the behaviour here. Ken's point that different treatmet of unset vs empty HOME is a mistake is fairly compelling for me. David
Re: nmh 1.8RC2, xlbiff, and $HOME
Ralph Corderoy wrote in <20230131181958.1cfb121...@orac.inputplus.co.uk>: ... |But if HOME is empty we do not know their intent so to ignore it and use |pw_dir may not be what they think will occur. The wrong profile could |be read or the wrong .netrc used, upsetting the user. By the way my mailer does: ? xv var HOME #nodelete,import-environ-first,sync-environ,notempty: set HOME=/home/steffen (where xv temporarily enables *verbose* mode). If i recall correctly you, once you were still taking part, have been dissatisfied with TOCTOU issues for TMPDIR for which the same holds: ? xv var TMPDIR #default-value,import-environ-first,sync-environ,notempty: set TMPDIR=/tmp So i (fwiw) are in the camp $ HOME= s-nail -R# s-nail: Environment variable must not be empty: HOME s-nail: There are messages in the error ring, manageable via `errors' command #?0!0/NONE#ERROR|:/dev/null? var HOME set HOME=/home/steffen #?0!0/NONE#ERROR|:/dev/null? xit --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
Re: nmh 1.8RC2, xlbiff, and $HOME
Date:Tue, 31 Jan 2023 13:29:46 + From:Ralph Corderoy Message-ID: <20230131132946.44a5f20...@orac.inputplus.co.uk> | > Similarly, in XCU 4 in the description of the cd utility: | Yes, but also allowed there is âemptyâ which also triggers | implementation-defined. Yes, that's because of: | Also, POSIX's chdir(3) says |int chdir(const char *path); | ERRORS |The chdir() function shall fail if: |... |ENOENT A component of path does not name an existing |directory or path is an empty string. and since some shells simply do effectively chdir(getenv(HOME)); in this case, they get an error, if HOME is unset, or empty. Others check the value from getenv(HOME) and do something different if it is NULL, and still others go further and check for "" as the value, and either treat that the same as a NULL result, or in some other way that suits them. Note that POSIX documents what users can expect to have happen (what can be relied upon) and only rarely mandates some specific behaviour that the people who maintain it believe should happen, rather than what actually does. If HOME is unset or empty, there is no consistent behaviour from shells, so POSIX says that. In the case of ~ expansion, there is no system call involved (necessarily) just string manipulation, so the case of HOME='' isn't treated specially (though it may end up being in the next version of the standard, as this is a case where there are some implementation differences - or perhaps it might be decided that the implementations that don't do what the standard currently requires are simply buggy, and leave things as they are now). There is (as of yesterday) a POSIX bug report about this, and some other weird case issues with ~ expansions. | The possible errors are now | environment variable HOME is empty The current question is why that one in particular needs to be an error. kre
Re: nmh 1.8RC2, xlbiff, and $HOME
Date:Tue, 31 Jan 2023 13:22:19 +
From:Ralph Corderoy
Message-ID: <20230131132219.5e02b20...@orac.inputplus.co.uk>
| It looks to me like code assumes mypath isn't NULL, e.g. exmaildir(),
| so not bothering to call set_mypath() if MH is set doesn't look a goer.
Perhape not, but
void
set_mypath(void)
{
char *var = getenv("HOME");
if (var && *var != '\0') {
mypath = mh_xstrdup(var);
return;
}
errno = 0;
struct passwd *pw = getpwuid(getuid());
/* etc */
would be a fairly trivial change (making the function smaller).
It would also be possible to change
if (!*pw->pw_dir)
die("password entry has empty home directory");
to
if (*pw->pw_dir == '\0')
pw->pw_dir = "/"; /* or "/tmp" or even
"/no/home/dir" or ... */
if that seemed desirable,
kre
Re: nmh 1.8RC2, xlbiff, and $HOME
Date:Tue, 31 Jan 2023 18:19:58 +
From:Ralph Corderoy
Message-ID: <20230131181958.1cfb121...@orac.inputplus.co.uk>
| No, one cannot say of an unset HOME that it may be set by accident.
No, but it may have been unset by accident, where the intended value
for mh to use was something different than the pw_dir entry. No-one
can possibly know in that case, any more than in the case where HOME=''
| There is no ambiguity over the right course.
There is always ambiguity if you are doing something different than
what is documented to happen. Users may have expected something different
than you expect.
| In the same way that seeing ${foo:-bar} instead of
| ${foo:bar} in sh is a smell.
Nonsense, ${foo:-bar} (rather than ${foo-bar} - a version with just a :
would be a ksh, bash, or zsh extension - but in this case I assume just
a typo)) is the more common thing to use.
It matches the other common usage ${foo} which produces nothing when
foo is unset, or when foo='' (and "${foo}" produces a null string
in either of those cases). That a default value is to be used instead
of the empty string doesn't change that at all - explicitly using ${foo-bar}
is a much rarer use case (there are occasions when that's wanted, but
they are not very common).
There is no point comparing what shells do when HOME is unset or "",
whether the same thing in the two cases, or something different (and
seemingly bizarre implementations in unusual cases are often just
because the cases are unusual - don't happen often - and slowing down
the normal case to handle those specially isn't worth the effort).
A shell can do whatever it likes in these cases. So can nmh.
The question is what is best for nmh - and for that I agree with Ken,
treating an unset HOME and HOME='' differently is a mistake. All that
is required to make that OK is to document it. There is never a
need to deliberately make anything an error to satisfy user expectations,
if the user wants a complaint about HOME being an empty string, rather
than some other behaviour, all they need is one of
test -z "${HOME}" && { echo Error; exit ; }
or
test -z "${HOME-x}" && { echo Error; exit ; }
(the latter being a case where omitting the ':' is wanted)
My experience is that lots of users don't know "unset" exists in the
shell. They just do var= (or unnecessarily var='') when then want a
variable to "go away". Most of the time unset and empty variables mean
the same thing, so this just works.
kre
Re: nmh 1.8RC2, xlbiff, and $HOME
Hi Ken,
> > What's the intent of an empty HOME?
> > Is it set by accident when it's meant to be unset?
> > Is it empty by accident when it's meant to be non-empty?
> > Do they want HOME=/, HOME=$PWD, or are they expecting it to error.
> > Any choice could be not what the user intended so exit.
>
> I mean ... you could say the exact same things of an unset HOME!
No, one cannot say of an unset HOME that it may be set by accident.
Nor that it is empty by accident since it is neither empty or non-empty.
This being Unix, the user knows what they're doing and POSIX seemingly
allows it so we permit the lack of HOME and use pw_dir.
There is no ambiguity over the right course.
As is often the way with an environment variable, HOME overrides pw_dir
just as an argv option overrides an environment variable which overrides
a configuration-file setting.
But if HOME is empty we do not know their intent so to ignore it and use
pw_dir may not be what they think will occur. The wrong profile could
be read or the wrong .netrc used, upsetting the user.
> My point is that I cannot see a reason to treat unset HOME and empty
> HOME differently in nmh; if unset HOME uses pw_dir, then I would argue
> that an empty HOME should do the same thing.
I disagree. In the same way that seeing ${foo:-bar} instead of
${foo:bar} in sh is a smell.
> I was curious and some system call traces suggest that what happens
> under the hood is that when 'cd' is called with an empty HOME sh
> explicitly calls chdir(getcwd()).
Sometimes. I think it's more complex than that, e.g. perhaps PWD has
an effect, and it will obviously vary widely.
$ LC_ALL=C env HOME= PWD=/nonexist strace -fe getcwd,stat,chdir dash -c cd
stat("/nonexist", 0x7ffc101d1bb0) = -1 ENOENT (No such file or
directory)
getcwd("/home/tmp/1628693513.257008819", 4096) = 31
chdir("/home/tmp/1628693513.257008819") = 0
+++ exited with 0 +++
$
$ LC_ALL=C env HOME= PWD=$PWD strace -fe getcwd,stat,chdir dash -c cd
stat("/home/tmp/1628693513.257008819", {st_mode=S_IFDIR|0755, st_size=4096,
...}) = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
chdir("/home/tmp/1628693513.257008819") = 0
+++ exited with 0 +++
$
$ LC_ALL=C env -u PWD HOME= strace -fe getcwd,stat,chdir dash -c cd
getcwd("/home/tmp/1628693513.257008819", 4096) = 31
chdir("/home/tmp/1628693513.257008819") = 0
+++ exited with 0 +++
$
--
Cheers, Ralph.
Re: nmh 1.8RC2, xlbiff, and $HOME
>What's the intent of an empty HOME? >Is it set by accident when it's meant to be unset? >Is it empty by accident when it's meant to be non-empty? >Do they want HOME=/, HOME=$PWD, or are they expecting it to error. >Any choice could be not what the user intended so exit. I mean ... you could say the exact same things of an unset HOME! My point is that I cannot see a reason to treat unset HOME and empty HOME differently in nmh; if unset HOME uses pw_dir, then I would argue that an empty HOME should do the same thing. Bakul says: >FWIW, this is how /bin/sh behaves on FreeBSD: >[...] Fair enough; I was curious and some system call traces suggest that what happens under the hood is that when 'cd' is called with an empty HOME sh explicitly calls chdir(getcwd()). But to me that still doesn't make a case that empty HOME and unset HOME should be treated differently in nmh. I recognize that this is a case where reasonable people can disagree. --Ken
Re: nmh 1.8RC2, xlbiff, and $HOME
On Jan 31, 2023, at 8:32 AM, Ralph Corderoy wrote: > > Hi Ken, > >>> So an unset HOME is allowed by this function, it's an empty HOME >>> which isn't. >> >> It strikes me as strange that there is a difference between an unset >> HOME and an empty HOME in terms of behavior. I mean, yes, I can see >> how the code is written, the historical precedent and how we got here, >> but ... well, I'm trying to understand the justification for treating >> those differently. > > A non-empty HOME is clear; its content should be used. > > An unset HOME is clear; it cannot be used. > There's a ready fallback, getpwuid(3) and pw_dir so use that. > > What's the intent of an empty HOME? > Is it set by accident when it's meant to be unset? > Is it empty by accident when it's meant to be non-empty? > Do they want HOME=/, HOME=$PWD, or are they expecting it to error. > Any choice could be not what the user intended so exit. FWIW, this is how /bin/sh behaves on FreeBSD: $ cd /tmp $ sh -c 'cd; echo $PWD' /home/bakul $ unset HOME $ sh -c 'cd; echo $PWD' cd: HOME not set /tmp $ HOME=/bin sh -c 'cd; echo $PWD' /bin $ HOME='' sh -c 'cd; echo $PWD' /tmp $ HOME='foo' sh -c 'cd; echo $PWD' cd: foo: No such file or directory /tmp
Re: nmh 1.8RC2, xlbiff, and $HOME
Hi Ken, > > So an unset HOME is allowed by this function, it's an empty HOME > > which isn't. > > It strikes me as strange that there is a difference between an unset > HOME and an empty HOME in terms of behavior. I mean, yes, I can see > how the code is written, the historical precedent and how we got here, > but ... well, I'm trying to understand the justification for treating > those differently. A non-empty HOME is clear; its content should be used. An unset HOME is clear; it cannot be used. There's a ready fallback, getpwuid(3) and pw_dir so use that. What's the intent of an empty HOME? Is it set by accident when it's meant to be unset? Is it empty by accident when it's meant to be non-empty? Do they want HOME=/, HOME=$PWD, or are they expecting it to error. Any choice could be not what the user intended so exit. -- Cheers, Ralph.
Re: nmh 1.8RC2, xlbiff, and $HOME
>So an unset HOME is allowed by this function, it's an empty HOME which >isn't. It strikes me as strange that there is a difference between an unset HOME and an empty HOME in terms of behavior. I mean, yes, I can see how the code is written, the historical precedent and how we got here, but ... well, I'm trying to understand the justification for treating those differently. --Ken
Re: nmh 1.8RC2, xlbiff, and $HOME
Hi kre,
Thanks for your learned input on this. As I said in another reply just
now, where I listed set_mypath(), HOME being unset is fine as getpwuid()
is the fallback in which case pw_dir must be non-empty.
> > Alexander does point out that HOME is supposed to be valid according
> > to POSIX,
>
> That's actually not correct, all the text quoted requires is that HOME
> be set at login time - nothing at all about what can be done with it
> after that.
Agreed. Nearby, is:
8.1 Environment Variable Definition
...
Some are placed into the environment by the implementation at the
time the user logs in; all can be added or changed by the user or
any ancestor of the current process.
Does ‘changed’ include unsetting them? I'd expect the spec. to be
precise and list ‘unset’ to clarify if they can.
> Further POSIX reading will also find, in the description of ~
> expansion in the shell (XCU 2.6.1 in Issue 7 TC2 - the current version
> from 2018):
>
> If the login name is null (that is, the tilde-prefix contains only
> the tilde), the tilde-prefix is replaced by the value of the
> variable HOME. If HOME is unset, the results are unspecified.
...
> Setting HOME='' isn't even mentioned: in that case ~/foo expands to
> /foo (which is actually what you want
Agreed, it isn't mentioned.
> Similarly, in XCU 4 in the description of the cd utility:
>
> 1. If no directory operand is given and the HOME environment
> variable is empty or undefined, the default behavior is
> implementation-defined and no further steps shall be taken.
>
> Again, HOME being unset is allowed there, this time with
> implementation-defined results (which just means the implementation
> must say what happens in the doc).
Yes, but also allowed there is ‘empty’ which also triggers
implementation-defined.
Also, POSIX's chdir(3) says
int chdir(const char *path);
...
ERRORS
The chdir() function shall fail if:
...
ENOENT A component of path does not name an existing
directory or path is an empty string.
chdir(NULL) is invalid, obviously. But so is chdir("") so if an empty
HOME is really to be treated as the root directory then code can't do
h = getenv("HOME");
r = chdir(h ? h : "/");
but must instead
r = chdir(h && *h ? h : "/");
I expect lots of non-nmh code doesn't do that extra check so the chdir()
has no effect and the error return may not even be tested so an empty
HOME results in the current directory being used rather than /.
> Running with HOME unset is not even all that unusual
I think it is a bit. :-)
> "env -i command" results in that
Agreed, my self-at(1)-ing shell scripts run at(1) under ‘env -i’ but
bequeath HOME, LANG, LOGNAME, and PATH at the same time.
> (and I'd even retain: relative to / if HOME is not set, which it can
> be when there as been no login - like when you're in single user
> mode).
It would be nice for nmh to work in single-user mode as reading email
might be vital in fixing problems. As long as HOME is unset and pw_dir
is valid then all is well at the moment.
> I'd also suggest never issuing errors because the user's environment
> isn't like we think it ought to be, unless that actually prevents
> something functioning as intended.
The code I replaced mislead the user as to the cause of the exit.
That's fixed and no HOME is okay. The possible errors are now
environment variable HOME is empty
password entry not found
[perhaps with ‘getpwuid() failed’ and errno if it did]
password entry has empty home directory
This close to a release, I think we should stick with requiring HOME to
be non-empty if it's set as otherwise there's too many paths to consider
which the test harness probably doesn't exercise.
> But I agree that a relative Path in MH should be relative to HOME
Yes, I think it's too late to alter that.
Though ‘path: mail’ would be a more Unixy default. :-)
--
Cheers, Ralph.
Re: nmh 1.8RC2, xlbiff, and $HOME
Hi az,
> "Author: Ralph Corderoy
> Date: Thu May 13 13:46:20 2021 +0100
>
> sbr/path.c: add set_mypath() to factor out repeated code."
I think it's worth expanding on that.
commit d8ca46fabc26469be325b73a73dcc26e70681eb5
Author: Ralph Corderoy
Date: Thu May 13 13:46:20 2021 +0100
sbr/path.c: add set_mypath() to factor out repeated code.
set_mypath() sets the existing global ‘mypath’ which really holds
the home-directory's path. As getenv(3) and getpwuid(3)'s result
can be invalidated by further library calls, a value they return is
mh_xstrdup()'d; not all the old bits of code replaced by a call to
set_mypath() were doing this.
Also add more explicit error messages if set_mypath() has trouble
finding the home directory's path.
And here's the current code I wrote for set_mypath() which is causing
the trouble.
/* set_mypath sets the global mypath to the HOME environment variable if
* it is set and not empty, or else the password entry's home-directory
* field if it's found and not empty. Otherwise, the program exits
* with an error. The value found is copied with mh_xstrdup() as later
* library calls may invalidate returned values. */
void
set_mypath(void)
{
char *var = getenv("HOME");
if (var) {
if (!*var)
die("environment variable HOME is empty");
mypath = mh_xstrdup(var);
return;
}
errno = 0;
struct passwd *pw = getpwuid(getuid());
if (!pw) {
if (errno)
adios("", "getpwuid() failed"); /* "" prints errno! */
die("password entry not found");
}
if (!*pw->pw_dir)
die("password entry has empty home directory");
mypath = mh_xstrdup(pw->pw_dir);
}
So an unset HOME is allowed by this function, it's an empty HOME which
isn't.
An unset HOME falls back on getpwuid() which must dish up a non-empty
pw_dir. I think that's reasonble.
> a quick glance seems to indicate that only context_read in
> sbr/context_read.c looks at $MH for finding a home
And uip/install-mh.c in a similar manner.
> but it does (well attempts) that just a few lines AFTER set_mypath has
> bailed out on it...
>
> i think we need to fix that sequence up a little :-)
It's the long-standing arrangement going back to Git's
docs/historical/mh-6.8.5/sbr/m_getdefs.c which allows an empty HOME but
not an empty pw_dir, though it complains that the later is the lack of
a HOME ‘envariable’. :-)
if (mypath == NULL) {
if (mypath = getenv ("HOME"))
mypath = getcpy (mypath);
else
if ((pw = getpwuid (getuid ())) == NULL
|| pw -> pw_dir == NULL
|| *pw -> pw_dir == 0)
adios (NULLCP, "no HOME envariable");
else
mypath = getcpy (pw -> pw_dir);
if ((cp = mypath + strlen (mypath) - 1) > mypath && *cp == '/')
*cp = 0;
}
It looks to me like code assumes mypath isn't NULL, e.g. exmaildir(),
so not bothering to call set_mypath() if MH is set doesn't look a goer.
At least not without a considerably more analysis which would probably
inspire changes which would ‘disappoint’ RC-maker David. :-)
--
Cheers, Ralph.
Re: nmh 1.8RC2, xlbiff, and $HOME
Hi David, > I'd like to get Ralph's take on what we should do. Thanks. If my suggestion to Stephen of unsetting HOME works and is acceptable to him then I suggest we don't change nmh for this release. > > A further documentation issue: mh-profile(5) does not specify the > > treatment of a relative Path. I expected it to be relative to the > > directory of the profile file, but it seems to actually be relative > > to HOME. > > Right, it's relative to HOME. I'll add this clarification to the man > page: > A relative Path is relative to the user's $HOME directory. > unless there's another suggestion. As HOME needn't be set how about mh_profile(5) saying Path: Mail Locates nmh transactions in directory “Mail”. +A relative path starts from the user's home directory. This is the only mandatory profile entry. (profile, no default) I see the ENVIRONMENT section doesn't mention HOME. Perhaps, HOME If set, it is used as the user's home directory. If not set then getpwuid(3)'s pw_dir is used. -- Cheers, Ralph.
Re: nmh 1.8RC2, xlbiff, and $HOME
Hi Stephen, > I have investigated the failure of the xlbiff tests with nmh 1.8RC2. Thanks. > $ printf 'Path: /tmp\n' > /tmp/mh-profile-minimal > $ HOME= MH=/tmp/mh-profile-minimal /usr/bin/mh/mhparam path My suggestion for a quick fix to try is to not have HOME in the environment so getpwuid(3) is used. $ HOME= MH=$PWD/mh_profile mhparam path mhparam: environment variable HOME is empty $ $ env -u HOME MH=$PWD/mh_profile mhparam path /tmp > This is a regression. HOME is used only to set the default profile > file to "$HOME/.mh_profile". But nmh doesn't need HOME if MH is set. I think nmh *as a whole* does need HOME to be non-empty if it's set. -- Cheers, Ralph.
