Re: [Nmh-workers] strncpy(3), die, die, die.
> On Oct 24, 2016, at 2:56 PM, Ken Hornsteinwrote: > > I think just calling abort() is lousy behavior in general. Versus writing broken code? Handing back unpredictable behaviour to the end-user is a complete abdication of responsibility. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
[Nmh-workers] 1.7 release date
A couple of comments have come up about when to release 1.7. Given all the thrashing of string/buffer manipulation code that has taken place over the last week and a bit, I don't think we can even think about baking this code now for at least a couple of months. We have just hammered on the most security vulnerable part of the code base, having done no prior analysis, nor identifying any know gaping wounds in the code. This scares me. This is code rewrite for religious purposes, and that is ALWAYS wrong. How are we going to validate all these memory/buffer/string related changes to ensure they have not introduced NEW bugs? Ralph, what is your plan for code verification of these changes you are making? The current regression tests can't come anywhere near dealing with this. --lyndon ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
>> My concern there is our release cycles have been long > >What needs to get done before 1.7 gets sent off to make its fortune? >Is that soon enough that this could hold off? Um, good question. Some things for 1.7 have not quite been resolved. >> and I'd hate to have code that barfs on emails released for a few >> years. > >Perhaps a complainant could be told of the secret $NMHNOBARF to stop >TRUNCCPY from aborting? Though it would still complain for the first N >goes? Weeelll ... I'd be okay with that. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Having configure Check for cc's -pedantic -ansi.
Ralph wrote: > I found the build on Mac OS X failed. build_nmh prompted `[n]' for TLS, > which I accepted with Enter. That stopped --with-tls getting to > ./configure; good. But configure defaults to yes, I think, Yes, it did. I just committed a change to do this: By default (with_tls=''), enable TLS if header and libs were found. If TLS requested (--with-tls with_tls=yes), error if header/lib not found. If TLS disabled (--without-tls with_tls=no), don't enable it. That's analogous to what configure.ac does for OAUTH support. I reviewed the OAUTH support, it looks good to me. So that just leaves the SASL support. It currently defaults to no. I'm thinking that we could do the same thing with it: obey yes and no, and default to yes only if the header and lib are found. David ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
Ralph Corderoy wrote: > Perhaps a complainant could be told of the secret $NMHNOBARF to stop > TRUNCCPY from aborting? Though it would still complain for the first N > goes? i think the moment that the state of the program becomes undefined, you should abort. malloc and asprintf helpfully return a useless value (NULL) if they can't fit your result into a new heap blob. snprintf, strncpy, and strlcpy do not. the right thing to create on overflow is an empty string. if the caller doesn't check the return value, they're going to wonder where that empty string came from. this would teach callers to check return values. returning the front half of the source string is bad. and while returning it non-\0-terminated is worse, neither is acceptable. i once received a thousands-of-lines-long patch to bind8 to make it use snprintf and strlcpy. i rejected it, and replaced every caller whose starting conditions were not obvious from simple inspection with an "if" statement that crashed out of the current operation if the resulting string would not fit my assumptions. replacing overrun with truncation is not a big enough improvement to justify touching the code at all. -- P Vixie ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
Todd C. Miller wrote: > On Mon, 24 Oct 2016 16:40:36 -0400, valdis.kletni...@vt.edu wrote: > >> In other words - if the source string doesn't fit, it will create >> a non-NULL-terminated destination string for you. Repeat that, >> slowly, until it sinks in. > > It says nothing of the sort, please re-read the manual. The source > string doesn't fit the result is always NUL-terminated unless the > specified size is 0. it should just return a null string. partially useful results are bad. -- P Vixie ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
Hi Ken, > Although ... crud, I take your point that a small change is a lot > easier than a big change. Yes, as soon as a free() is needed then there's tracking the paths that exit the function, where the pointer gets copied to, etc. GC anyone? :-) > My concern there is our release cycles have been long What needs to get done before 1.7 gets sent off to make its fortune? Is that soon enough that this could hold off? > and I'd hate to have code that barfs on emails released for a few > years. Perhaps a complainant could be told of the secret $NMHNOBARF to stop TRUNCCPY from aborting? Though it would still complain for the first N goes? -- Cheers, Ralph. https://plus.google.com/+RalphCorderoy ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
>Agreed. But crapping out breaks the silence so that area can be worked >on. I think just calling abort() is lousy behavior in general. But maybe there's a middle ground; a lot of these cases are just because we didn't want to allocate a dynamic buffer. Maybe we should start using asprintf() a lot more? Although ... crud, I take your point that a small change is a lot easier than a big change. >At the moment, there's many strncpy calls and they can't all be >rewritten to be malloc or something else. Another alternative is >reporting on stderr, though spew from a loop would be annoying. Perhaps >the truncating-copy routine could stop reporting after a few. When >they're fixed, the next ones will project. My concern there is our release cycles have been long, and I'd hate to have code that barfs on emails released for a few years. A quick look suggests to me that we could legitimately barf on a lot of those calls, but I'd rather we took a careful look at each one that deals with actual on-wire email. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
Hi David, I hope this strncpy thread doesn't run on for ever and anon. It's delaying my brace-style email. > > I dunno, I think we'd need to think carefully if a particular use of > > strncpy() really warrants an abort vs a truncate. I mean, just > > crapping out on a really long line that other MUAs handle just fine > > seems rather unfriendly to me. What do others think? > > I'm not a fan of abort, either, unless it shows the context. That > might not be trivial with multibyte characters, but should be doable. What if we consider the s/strncpy/trunccpy/ a diagnostic aid for the existing code? The call site won't change much: the size will including the NUL so sizeof doesn't need adjusting; trunccpy will always store the NUL so the next line doesn't need to be the NUL assignment. The function can also be passed __FILE__ and __LINE__ for reporting when it truncates. And it can report details of the truncated too, assuming one-byte-per-rune (codepoint) encoding and escaping !isprint. A macro can handle passing size and source location for the common case. TRUNCCPY(buf, addr); -- Cheers, Ralph. https://plus.google.com/+RalphCorderoy ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
On Mon, 24 Oct 2016 22:10:46 +0100, Ralph Corderoy wrote: > Yes, asprintf(3) is very handy. Unfortunately, it's not standardised; C > nor POSIX. And rolling your own version around vsnprintf(3) can mean > doing the formatting twice; the first time to get the length. You've > then a malloc'd pointer to track. So it's a more of a change than I > want to tackle now. This is the kind of thing I've been doing. > http://git.savannah.gnu.org/cgit/nmh.git/commit/?id=9b3fc4790b2473b23c69c0e70 > 710e1e66a038f28=1 For what it's worth, you are welcome to use the BSD-licensed asprintf() sudo uses when the host system lacks it. It is based on the BSD vfprintf.c so you only need to do the formatting once. The only caveat is that it doesn't support floating point by default. I can certainly understand not wanting to make radical changes at this point. If you do find in the future that it would be useful, please let me know. - todd ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
Hi Todd, > Paul Vixie wrote: > > Copy or die, as the default behavior. malloc! Or death! > Both snprintf() and strlcpy() make it fairly easy to detect whe the > buffer was too small, which is more than I can say for strncpy(). It > is up to the programmer to actually check the return value. And there's the nub. It can be done with strncpy too; check strnlen(3)'s result afterwards. But the existing code does not check, and I'm not aiming to rewrite the code so I don't think the check should be added at the call-site. That also knocks out Paul's `copy' option above since if a local char[] is being used as the destination then I'm intending it remains that. I'm not arguing this shouldn't change in the future, but my aim is to do lots of small localised, readily verifiable by inspection, "fixes". That might be removing reundancy from the source, not just to remove the chance of error, but to save every read of it having to verify consistency. Or it may be, like here, to improve the run-time behaviour a bit. Better the truncation isn't silent. > That said, I certainly agree that proceeding with a truncated buffer > is the wrong thing to do. Many (but not all) systems these days > provide asprintf() which dynamically allocates its buffer which can > solve a lot of these problems. Yes, asprintf(3) is very handy. Unfortunately, it's not standardised; C nor POSIX. And rolling your own version around vsnprintf(3) can mean doing the formatting twice; the first time to get the length. You've then a malloc'd pointer to track. So it's a more of a change than I want to tackle now. This is the kind of thing I've been doing. http://git.savannah.gnu.org/cgit/nmh.git/commit/?id=9b3fc4790b2473b23c69c0e70710e1e66a038f28=1 So I'm really after moving some of the strncmp()s into the "really wants to copy the string whole, without any padding, and doesn't intend to truncate" camp by using a new routine for those thus identified. -- Cheers, Ralph. https://plus.google.com/+RalphCorderoy ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
On Mon, 24 Oct 2016 16:40:36 -0400, valdis.kletni...@vt.edu wrote: > In other words - if the source string doesn't fit, it will create > a non-NULL-terminated destination string for you. Repeat that, > slowly, until it sinks in. It says nothing of the sort, please re-read the manual. The source string doesn't fit the result is always NUL-terminated unless the specified size is 0. - todd ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
On Mon, 24 Oct 2016 19:59:10 -, P Vixie wrote: > I don't know what gcc or clang command line option to use to require this. If you declare the function with gcc's warn_unused_result attribute the compiler will warn when you don't check the result. For example: size_t strlcpy(char *dst, const char *src, size_t dstsize) __attribute__((warn_unused_result)); - todd ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
On Mon, 24 Oct 2016 13:08:22 -0600, "Anthony J. Bentley" said: > P Vixie writes: > > Strlcpy is completely bogus. > > What's silent about strlcpy? Just check the return value against the size > of the buffer. He didn't say it was silent. He said it was bogus. >From the manpage: https://www.freebsd.org/cgi/man.cgi?query=strlcpy=3 The strlcpy() and strlcat() functions return the total length of the string they tried to create. For strlcpy() that means the length of src. For strlcat() that means the initial length of dst plus the length of src. While this may seem somewhat confusing, it was done to make truncation detection simple. Note however, that if strlcat() traverses size characters without finding a NUL, the length of the string is considered to be size and the destination string will not be NUL-terminated (since there was no space for the NUL). This keeps strlcat() from running off the end of a string. In practice this should not happen (as it means that either size is incorrect or that dst is not a proper ``C'' string). The check exists to prevent potential security problems in incorrect code. In other words - if the source string doesn't fit, it will create a non-NULL-terminated destination string for you. Repeat that, slowly, until it sinks in. pgpFKulVISsfY.pgp Description: PGP signature ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
On October 24, 2016 9:08:22 PM GMT+02:00, "Anthony J. Bentley"wrote: >P Vixie writes: >> Strlcpy is completely bogus. > >What's silent about strlcpy? Just check the return value against the >size >of the buffer. I don't know what gcc or clang command line option to use to require this. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
P Vixie writes: > > > On October 24, 2016 6:37:16 PM GMT+02:00, Ken Hornsteinwrot > e: > > >I dunno, I think we'd need to think carefully if a particular use of > >strncpy() really warrants an abort vs a truncate. I mean, just > >crapping > >out on a really long line that other MUAs handle just fine seems rather > >unfriendly to me. What do others think? > > Copy or die, as the default behavior. > > Silent truncation should require explicit coding. > > Strlcpy is completely bogus. What's silent about strlcpy? Just check the return value against the size of the buffer. -- Anthony J. Bentley ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
On overflow, the string should be zero filled, or abort() should be called. Leaving a half useful result creates no incentive to check the return value. On October 24, 2016 9:11:09 PM GMT+02:00, "Todd C. Miller"wrote: >On Mon, 24 Oct 2016 18:59:36 -, P Vixie wrote: > >> Copy or die, as the default behavior. >> >> Silent truncation should require explicit coding. >> >> Strlcpy is completely bogus. > >Both snprintf() and strlcpy() make it fairly easy to detect whe the >buffer was too small, which is more than I can say for strncpy(). >It is up to the programmer to actually check the return value. > >That said, I certainly agree that proceeding with a truncated buffer >is the wrong thing to do. Many (but not all) systems these days >provide asprintf() which dynamically allocates its buffer which can >solve a lot of these problems. > > - todd -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
On October 24, 2016 6:37:16 PM GMT+02:00, Ken Hornsteinwrote: >I dunno, I think we'd need to think carefully if a particular use of >strncpy() really warrants an abort vs a truncate. I mean, just >crapping >out on a really long line that other MUAs handle just fine seems rather >unfriendly to me. What do others think? Copy or die, as the default behavior. Silent truncation should require explicit coding. Strlcpy is completely bogus. Vixie -- Sent from my Android device with K-9 Mail. Please excuse my brevity. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
Ken wrote: > I dunno, I think we'd need to think carefully if a particular use of > strncpy() really warrants an abort vs a truncate. I mean, just crapping > out on a really long line that other MUAs handle just fine seems rather > unfriendly to me. What do others think? I'm not a fan of abort, either, unless it shows the context. That might not be trivial with multibyte characters, but should be doable. David ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Having configure Check for cc's -pedantic -ansi.
Ralph wrote: > Hi David, > > Worked well on a machine where I hadn't built it before, but was likely > to have some of the optional development packages installed. Yeah, the script doesn't check those dependencies in advance. The build doesn't take very long on modern machines so I think that's OK. I'll add a suggestion to check the MACHINES file for dependencies if configure fails. > I went with the defaults for all but the first prompt. I should revisit the OAUTH, TLS, and SASL defaults based on what configure now does with them. > I then had to poke around for the log file. (With my "just paste the > commands I was told" hat on.) Perhaps its path can be printed at the > end? Added, if any of: it's not in the current directory (because the sources were downloaded), the build failed, or -v was used. > I wouldn't want to email the one here. tools/showbuildenv captures the > environment and that has personal information in it, e.g. paths can > contain real-life data. I'm wondering how useful env[] is, other than > for the uname values stuffed in there by showbuildenv. I replaced the showbuildenv with a few uname's and cat /etc/os-release if it exists. Looking at the buildbot hosts now, either that or none of the files searched for in showbuildenv are used. > It's a peer to configure, Makefile, etc? So have it at the top level? Sounds good to me. Any objection? > The `tail -4' output near the end, that has the ===..., has escape codes > for colour BTW. Feature! I added a fix to not do that if TERM is unset or dumb, and documented that in the comments at the top. > The second time `sh build_nmh' is run the `git clone' fails because > ./nmh exists and isn't empty. The user doesn't know that immediately Fixed. > This is due to the new `D' option to ar(1) for deterministic mode. I don't face this daily (on Fedora), though at least some of the buildbot hosts do. Not sure what to do about it. Excise it from, or override (assuming POSIX ar), ARFLAGS? > I've pushed 4ac9784480441d2bebe8a2ad9584165bbf2ee345 to have configure > edit flex's 2.6.1 output; it already had logic for fixing earlier > versions. Should we remove -Wall -Wextra from CFLAGS and leave all that up to the user? Then we could remove the flex dependencies. I'm tired of trying to clean up flex's droppings. > The `tags' target uses etags, not installed here. It seems that's a > default and nothing to do with nmh's use of autotools? Right, and looking quickly, I don't see a way to disable it: If any C, C++ or Fortran 77 source code or headers are present, then ‘tags’ and ‘TAGS’ rules will be generated for the directory David ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
Hi Ken, > You know, somehow I had missed all of these years that strncpy() pads > out the rest of the buffer, which is certainly not ideal! Yes, I keep forgetting it and rediscovering. :-) > I can only say that on my Linux systems, I don't have strlcpy() or > -lbsd. Here it's provided by package libbsd that was pulled in by openbsd-netcat, smbclient, ... http://libbsd.freedesktop.org But it's a trivial function. > However, the current situation isn't ideal. But this gets into some > deeper questions - like, if we are truncating a "string", does the > character set matter? Yes. > Also, what should we do when a string is truncated? I'm not so crazy > about our current behavior of calling exit() inside of a lot of > library functions. Me neither. But the doing that is arguably better than silently passing on truncated data, perhaps a string that's invalid under the encoding. > I suspect a lot of the time we don't care if truncation happens, or > more accurately there isn't really a great solution. Rewrite that bit of code where it can happen to use dynamic allocation? But that's a big leap from today's code for all strncpy calls. > I dunno, I think we'd need to think carefully if a particular use of > strncpy() really warrants an abort vs a truncate. There may be cases that truly aim for a truncate, yes. Either they'll be spotted and left alone. Or wrongly coverted, blow up, and be reverted. Given I think few want a truncate, I'm hoping not many reversions will be needed. > I mean, just crapping out on a really long line that other MUAs handle > just fine seems rather unfriendly to me. Agreed. But crapping out breaks the silence so that area can be worked on. At the moment, there's many strncpy calls and they can't all be rewritten to be malloc or something else. Another alternative is reporting on stderr, though spew from a loop would be annoying. Perhaps the truncating-copy routine could stop reporting after a few. When they're fixed, the next ones will project. -- Cheers, Ralph. https://plus.google.com/+RalphCorderoy ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] strncpy(3), die, die, die.
>This leaves s NUL terminated, but possibly silently truncated. Also, if >addr is a lot shorter than BUFSIZ, 8KiB here, then strncpy NULs out all >the rest of the 8KiB past the end of the string's terminator NUL. I'd >think that's not needed by most of the callers, though it's difficult to >know without examining each case. You know, somehow I had missed all of these years that strncpy() pads out the rest of the buffer, which is certainly not ideal! >I was thinking of switching to strlcpy(3) from BSD, also available on >Linux with -lbsd or similar. A stand-in if it's not available is >trivial. The autoconf less so, but seemingly do-able. >http://lists.windowmaker.org/dev/msg01783.html I can only say that on my Linux systems, I don't have strlcpy() or -lbsd. However, the current situation isn't ideal. But this gets into some deeper questions - like, if we are truncating a "string", does the character set matter? Also, what should we do when a string is truncated? I'm not so crazy about our current behavior of calling exit() inside of a lot of library functions. I suspect a lot of the time we don't care if truncation happens, or more accurately there isn't really a great solution. >«snprintf(s, sizeof s, "%s", addr)» is the equivalent. Both would >shorten the code back to one line, so less to review, but both would >still silently truncate. > >So how about our own function that takes (dest, src, size) and if >strlen(src) isn't less than size then it abort(3)s. So, it's a checking >strcpy, no more. A macro for the common case of size being dest's >sizeof will remove the last bit of repetition. I dunno, I think we'd need to think carefully if a particular use of strncpy() really warrants an abort vs a truncate. I mean, just crapping out on a really long line that other MUAs handle just fine seems rather unfriendly to me. What do others think? --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Minor gripe about send man page
Ken Hornsteinwrites: >>But it isn't clear, at least to me, which of those things send does >>and which post does. As I said, this is a minor gripe. > >Man, Norm, it's been that way for approximately forever ... you're just >noticing now? :-) Until I had to write a postproc the issue was of no interest to me. How send did its business was just an implementation detail. Then, when I did write a postproc, your sample bash script and a bit of instrumentation told me what I needed to know. That's why I called the gripe "minor". Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Having configure Check for cc's -pedantic -ansi.
Hi David, > Worked well on a machine where I hadn't built it before, but was > likely to have some of the optional development packages installed. The second time `sh build_nmh' is run the `git clone' fails because ./nmh exists and isn't empty. The user doesn't know that immediately though because it's at the end of the /tmp log file that's trap-rm'd on EXIT. Fair enough. We've only asked them to paste those two lines. Just FYI. -- Cheers, Ralph. https://plus.google.com/+RalphCorderoy ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Minor gripe about send man page
Hi Ken, > But to answer [Norm's] question ... the division of labor is kind of > like the division between show(1) and mhl(1); send(1) takes a message > and maybe does some minor processing on it (or splits it up), and then > passes the file off to post(8) to actually send it (post takes a > filename, _not_ a folder and message number). I had a skim of their man pages. So if one's writing a postproc then post(8)'s interface has to be implemented? I wonder if send(1)'s SYNOPSIS should separate out the arguments. Those it handles. Those that affect its behaviour and also get past to post. Those it ignores other than to pass to post. -- Cheers, Ralph. https://plus.google.com/+RalphCorderoy ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers