Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ken Hornstein]

>OK, I'll check-pick these across to the branch a little later on then.
>Let me know if that's not the right way to do it.

Thanks, Ralph!  I personally use git cherry-pick -x, and I think that's
what makes sense.  Does anyone disagree?

>On other news, I've moved from Arch Linux's AUR's `nmh' package to
>`nmh-git', which means whenever I re-run makepkg(1) for it the users
>here get updated to HEAD.  That's good, but it's not the 1.7 branch.  I
>suppose I could switch to that for a bit by modifying the git-repo URL
>it pulls.

Hm, I should do that for the MacOS brew recipie as well.

--Ken

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi Ken,

> I'm not really liking the idea of having a code fix that only appears
> on the release branch.  I'd rather have the same code in 1.7 and
> master.  So I say apply the re-write to 1.7.

OK, I'll check-pick these across to the branch a little later on then.
Let me know if that's not the right way to do it.

$ git log --oneline 1.7-release..HEAD sbr/fmt_scan.c
a091c28b cpstripped(): Rewrite multi-byte version.
8ba2c2e0 cpstripped(), single-byte: Simplify logic.
0a934c0f cpstripped: Split into two versions, each with a single #ifdef.
$

On other news, I've moved from Arch Linux's AUR's `nmh' package to
`nmh-git', which means whenever I re-run makepkg(1) for it the users
here get updated to HEAD.  That's good, but it's not the 1.7 branch.  I
suppose I could switch to that for a bit by modifying the git-repo URL
it pulls.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi Ken,

> So, I guess I'm a liar ... turns out MacOS X _does_ define
> __WCHAR_MAX__.  Although it seems like WCHAR_MAX is the correct macro
> according to POSIX, so maybe getcwidth.c should still switch?

Yes, I can't recall why I plumped for the __ version now.  I've pushed a
fix.  Indentifiers starting with double underscore are reserved so the
system has no fear of trampling on ours with its internal ones.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ken Hornstein]

>So, I'm back from vacation and trying to digest this all ... and the first
>thing I ran into is that the new ctype code in test/getcwidth.c uses
>__WCHAR_MAX__ ... which is not defined by MacOS X!

So, I guess I'm a liar ... turns out MacOS X _does_ define __WCHAR_MAX__.
Although it seems like WCHAR_MAX is the correct macro according to POSIX,
so maybe getcwidth.c should still switch?

--Ken

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ken Hornstein]

>> I guess we'd like a minimal-change fix for this on the 1.7
>> branch rather than flip to the re-write?
>
>I'd be fine with the re-write for 1.7, or not.  Let's see what Ken
>says.

I'm not really liking the idea of having a code fix that only appears on
the release branch.  I'd rather have the same code in 1.7 and master.  So
I say apply the re-write to 1.7.

--Ken

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ken Hornstein]

>Yeah, I think the code was assuming if it's not cntrl, and not space,
>then it must be print and therefore width >= 0.  But that's clearly not
>true, even here, where Norm's sunglasses don't have a negative width,
>e.g. U+0378.

So, I'm back from vacation and trying to digest this all ... and the first
thing I ran into is that the new ctype code in test/getcwidth.c uses
__WCHAR_MAX__ ... which is not defined by MacOS X!  It looks like POSIX
officially uses WCHAR_MAX.  I will admit that I am always confused why
some preprocessor symbols use __ around them and some do not (I am sure
there is a reason for when to use it, it just is not obvious to me).  But
since POSIX officially defines WCHAR_MAX, we should use that?

--Ken

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi David,

> > Sunglasses have a width of 1 here, that's why David and I don't see
> > the problem.
>
> I'm surprised that I didn't see the same behavior as Norm, because we
> use the same locale, en_US.utf8.  Any idea why?

I'm en_GB.utf8, but I don't see it either.  It's the wcwidth(3) answer
for a codepoint, and as Unicode continue to add POO WITH SUNGLASSES, so
the answers change with the version of one's system's database.

$ test/getcwidth --ctype | grep 1f576
 1f576   1  -pg--@--

So here it's `print', `graph', and `punct', with a width of 1.  Norm's
gang have a width of -1 as they haven't the foggiest what it is.
http://unicode.org/cldr/utility/character.jsp?a=1f576=Show says its
East Asian width is `Neutral', which is treated as `Narrow', so
getcwidth reporting 1 matches.

Nearby is http://unicode.org/cldr/utility/character.jsp?a=1f57a=Show
that says it's `Wide', but here I don't know anything about that yet,
thankfully.

$ test/getcwidth --ctype | grep 1f57a
 1f57a  -1  

One can poke about the local definitions.

$ test/getcwidth --ctype | awk '{print $2}' |
> sort -n | uniq -c
  57249 -1
   1723 0
  29884 1
  95464 2
$
$ test/getcwidth --ctype | awk '{print $3}' |
> LC_ALL=C sort | uniq -c
  57183 
 14 -psb
  15563 -pg--@--
 10 -pg---dxN---
 107528 -pgaN---
   2167 -pga-l--N---
  6 -pga-l-xN---
   1772 -pgau---N---
  6 -pgau--xN---
  4 -pgaul--N---
 60 c---
  6 c-s-
  1 c-sb
$ 

That says there are four runes that are both upper and lower!

$ printf '%b\n' $(test/getcwidth --ctype |
> awk '$3 ~ /ul/ {print "\\u" $1}')
Dž
Lj
Nj
Dz
$

And here's the first printable zero-width.

$ test/getcwidth --ctype | grep -m1 ' 0 .*p'
ad   0  -pg--@--

U+00AD is soft hyphen.  Unicode is said to be an ISO 8859-1 superset,
and U+AD was soft hyphen in that too, but visible, with a width of 1.
ISO used it at the end of the line to show a word had been broken, but
not by the author, allowing it to be stripped on re-formatting.  Unicode
changed that.  For them, it's a hint from the author to the renderer
that here's a potential point to break the word, thus, when rendered,
it's not visible and has zero width.  Toc toc toc!

Terminals get this wrong.  libvte-based terminals here think it has
width.

$ s="$(printf '\uad')"
$ scan -format "_%4(lit foo)_\n_%4(lit £)_\n_%4(lit $s)_" .
_foo _
_£   _
_­_   [Rune after first _ isn't a space.]
$

Dickey's venerable xterm(1) does better.

$ s="$(printf '\uad')"
$ scan -format "_%4(lit foo)_\n_%4(lit £)_\n_%4(lit $s)_" .
_foo _
_£   _
__   [All four are spaces.]
$

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi,

David wrote:
> > inc seems to have a similar, if not identical problem.
>
> It's identical:  inc uses the same code to print the scan line.

I was surprised recently when plodding through the source that inc(1)
actually uses the scan() function to do the inc-ing, with the scan as a
side effect.  IIRC.  And this means scan() has to bear in mind inc(1)
wants all the email even though the scan listing stopped a while ago.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[David Levine]

Ralph writes:

> Hi Leonardo,
>
> > At least regarding the `make check' (always on NetBSD/amd64 8.99.1),
> > all 104 tests are passed and 7 tests were not run.
>
> Great, thanks.

Yes, thanks to both of you.

> I guess we'd like a minimal-change fix for this on the 1.7
> branch rather than flip to the re-write?

I'd be fine with the re-write for 1.7, or not.  Let's see what Ken
says.

David

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[David Levine]

Ralph writes:

> Sunglasses have a width of 1 here, that's why David and I don't see the
> problem.

I'm surprised that I didn't see the same behavior as Norm, because
we use the same locale, en_US.utf8.  Any idea why?

David

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[David Levine]

Norm writes:

> inc seems to have a similar, if not identical problem.

It's identical:  inc uses the same code to print the scan line.

David

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Leonardo Taccari]

Hello Ralph,

Ralph Corderoy writes:
> [...]
> MACHINES says that's ftp/curl on FreeBSD, if you're interested.
> configure checks for its header file and library IIRC.  But I'm happy
> with the tests that were run.
> [...]

...indeed now I can report:


All 110 tests passed
(1 test was not run)


Thank you for the pointer!

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi Leonardo,

> At least regarding the `make check' (always on NetBSD/amd64 8.99.1),
> all 104 tests are passed and 7 tests were not run.

Great, thanks.

> Regarding the not run ones (all expected):
>
>  ./test/oauth/test-inc: skipped: no oauth support

MACHINES says that's ftp/curl on FreeBSD, if you're interested.
configure checks for its header file and library IIRC.  But I'm happy
with the tests that were run.

>  % scan -format '%(decode{subject})' +. 1
>  ?Sun’s out, savings ON—shop major appliance deals now
>
> So it no longer asserts and instead of sunglasses a `?' is printed!
> Yay!

Good.

David/Ken: I guess we'd like a minimal-change fix for this on the 1.7
branch rather than flip to the re-write?

> NetBSD valgrind isn't available so I can't probably help for these
> tests, sorry.

No problem.  Thanks for your help.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Leonardo Taccari]

Hello Ralph,

> [...]
> I've re-written the MULTIBYTE_ENABLED cpstripped() on git's master
> branch, not the 1.7 one.  Could you give it a go with what failed
> before, and ideally run a `make check' and if all's well a
> `NMH_VALGRIND=1 VALGRIND_ME=1 make check' to give it a work out, it
> takes a while, with your different definitions of Unicode widths.
> docs/README.developers explains those two variables.
> [...]

Sure! At least regarding the `make check' (always on NetBSD/amd64
8.99.1), all 104 tests are passed and 7 tests were not run. Regarding
the not run ones (all expected):

 [...]
 ./test/inc/test-deb359167: skipped: missing valgrind
 SKIP: test/inc/test-deb359167 
 [...]
 ./test/oauth/test-inc: skipped: no oauth support
 SKIP: test/oauth/test-inc 
 ./test/oauth/test-mhlogin: skipped: no oauth support
 SKIP: test/oauth/test-mhlogin 
 ./test/oauth/test-mhparam: skipped: no oauth support
 SKIP: test/oauth/test-mhparam 
 ./test/oauth/test-send: skipped: no oauth support   
 SKIP: test/oauth/test-send
 ./test/oauth/test-sendfrom: skipped: no oauth support   
 SKIP: test/oauth/test-sendfrom
 ./test/oauth/test-share: skipped: no oauth support  
 SKIP: test/oauth/test-share   
 [...]

(the former because valgrind isn't available here, the latter because
oauth support wasn't enabled).

Back to the problematic Norm's email (and some emails that I had
that exposed the same Norm's problem):

 % scan -format '%(decode{subject})' +. 1
 ?Sun’s out, savings ON—shop major appliance deals now

So it no longer asserts and instead of sunglasses a `?' is printed! Yay!

Regarding `NMH_VALGRIND=1 VALGRIND_ME=1 make check' unfortunately on
NetBSD valgrind isn't available so I can't probably help for these tests,
sorry.


Thank you very much Ralph for the investigation and for fixing this bug!

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi Leonardo,

> Sure! Here what I see with your patch applied:
>
>  % scan -format '%(decode{subject})' +. 1
>  mbtowc(0x20) = 1
>  cntrl:0  space:1  blank:1  print:1
>  mbtowc(0x1f576) = 4
>  cntrl:0  space:0  blank:0  print:0
>  wcwidth(0x1f576) = -1

Yeah, I think the code was assuming if it's not cntrl, and not space,
then it must be print and therefore width >= 0.  But that's clearly not
true, even here, where Norm's sunglasses don't have a negative width,
e.g. U+0378.

$ test/getcwidth --ctype | grep -m1 ' -1  [^cs]*$'
   378  -1  
$

That does mean I can reproduce the failed assertion using that rune.

$ uip/scan -format '%(lit £)' .
£
$ uip/scan -format '%(lit ͸)' .
scan: sbr/fmt_scan.c:280: cpstripped: Assertion `w >= 0' failed.

Sunglasses have a width of 1 here, that's why David and I don't see the
problem.

$ test/getcwidth --ctype | grep 1f576
 1f576   1  -pg--@--
$

I've re-written the MULTIBYTE_ENABLED cpstripped() on git's master
branch, not the 1.7 one.  Could you give it a go with what failed
before, and ideally run a `make check' and if all's well a
`NMH_VALGRIND=1 VALGRIND_ME=1 make check' to give it a work out, it
takes a while, with your different definitions of Unicode widths.
docs/README.developers explains those two variables.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Leonardo Taccari]

Ralph Corderoy writes:
> Hi Leonardo,
>
> [...]

Hello Ralph,

> That would do it.  Could you apply the attached patch and re-run?  I'm
> basically interested in how that locale classes it, e.g.  iswprint(3).
>

Sure! Here what I see with your patch applied:

 % scan -format '%(decode{subject})' +. 1
 mbtowc(0x20) = 1
 cntrl:0  space:1  blank:1  print:1
 mbtowc(0x1f576) = 4
 cntrl:0  space:0  blank:0  print:0
 wcwidth(0x1f576) = -1
 assertion "w >= 0" failed: file "sbr/fmt_scan.c", line 296, function 
"cpstripped"
 Abort (core dumped)
 Exit 134

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi Leonardo,

>  (gdb) print wide_char
>  $2 = 128374 L'\x1f576'
>  (gdb) quit
> 
> L'\x1f576' (in wide_char) is probably the `dark sunglasses' (U+1F576)
> unicode character

It is.  One of three non-ASCII characters in that subject.

$ uip/scan -file 8759.2.email -format '%(decode{subject})' |
> iconv -t ucs-4le |
> hexdump -ve '5/4 "  % 8x" /0 "\n"'
 1f57653756e  2019
73206f7574
2c20736176
696e677320
4f4e  20147368
6f70206d61
6a6f722061
70706c6961
6e63652064
65616c7320
6e6f77 a  
$

> and directly trying to:
> 
>  wcwidth(L'\x1f576')
> 
> ...returns `-1'.

That would do it.  Could you apply the attached patch and re-run?  I'm
basically interested in how that locale classes it, e.g.  iswprint(3).

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
diff --git sbr/fmt_scan.c sbr/fmt_scan.c
index c75db3ec..58977359 100644
--- sbr/fmt_scan.c
+++ sbr/fmt_scan.c
@@ -247,6 +247,7 @@ cpstripped (charstring_t dest, size_t max, char *str)
 while (*str != '\0' && len > 0 && max > 0) {
 #ifdef MULTIBYTE_SUPPORT
 	char_len = mbtowc(_char, str, len);
+fprintf(stderr, "mbtowc(%#x) = %d\n", wide_char, char_len);
 
 	/*
 	 * If mbrtowc() failed, then we have a character that isn't valid
@@ -259,6 +260,7 @@ cpstripped (charstring_t dest, size_t max, char *str)
 	if (char_len < 0) {
 	altstr = "?";
 	char_len = mbtowc(_char, altstr, 1);
+fprintf(stderr, "mbtowc(%#x) = %d\n", wide_char, char_len);
 	}
 
 	if (char_len <= 0) {
@@ -267,6 +269,8 @@ cpstripped (charstring_t dest, size_t max, char *str)
 
 	len -= char_len;
 
+fprintf(stderr, "cntrl:%d  space:%d  blank:%d  print:%d\n",
+iswcntrl(wide_char), iswspace(wide_char), iswblank(wide_char), iswprint(wide_char));
 	if (iswcntrl(wide_char) || iswspace(wide_char)) {
 	str += char_len;
 #else /* MULTIBYTE_SUPPORT */
@@ -288,6 +292,7 @@ cpstripped (charstring_t dest, size_t max, char *str)
 
 #ifdef MULTIBYTE_SUPPORT
 	w = wcwidth(wide_char);
+fprintf(stderr, "wcwidth(%#x) = %d\n", wide_char, w);
 	assert(w >= 0);
 	if (max >= (size_t) w) {
 	charstring_push_back_chars (dest, altstr ? altstr : str, char_len, w);
___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi again Norm,

I wrote:
> Could you run this scan command with 1.7-RC1

With Leonardo's reply, there's no need to bother.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Leonardo Taccari]

Hello Ralph,
I was bitten by something similar and so I've rebuilt nmh-1.7-RC1 with
debug symbols, attached some (hopefully) useful debug information (I've
used gdb though because AFAIK ltrace isn't available on NetBSD).

Ralph Corderoy writes:
> [...]
> Could you run this scan command with 1.7-RC1 to make sure it still has
> the assert(3) failure on the problem email.
>
> scan -format '%(decode{subject})' cur
> [...]

I've mhstore-ed Norm's problematic email as `1' in the current directory
and reproduced it via:

 % scan -format '%(decode{subject})' +. 1
 assertion "w >= 0" failed: file "sbr/fmt_scan.c", line 291, function 
"cpstripped"
 Abort (core dumped)
 Exit 134

Looking at the problematic char on gdb (this is on NetBSD/amd64 8.99.1):

 % gdb -core ./scan.core `which scan`
 Reading symbols from /usr/pkg/bin/scan...done.
 [New process 1]
 Core was generated by `scan'.
 Program terminated with signal SIGABRT, Aborted.
 #0  0x7e7fe9b2725a in _lwp_kill () from /usr/lib/libc.so.12
 (gdb) bt
 #0  0x7e7fe9b2725a in _lwp_kill () from /usr/lib/libc.so.12
 #1  0x7e7fe9b26e75 in abort () at /usr/src/lib/libc/stdlib/abort.c:74
 #2  0x7e7fe9b26a03 in __assert13 (file=file@entry=0x4171ff 
"sbr/fmt_scan.c", line=line@entry=291,
 function=function@entry=0x417600 <__func__.6577> "cpstripped", 
failedexpr=failedexpr@entry=0x4171f8 "w >= 0")
 at /usr/src/lib/libc/gen/assert.c:72
 #3  0x00407552 in cpstripped (dest=dest@entry=0x7e7fe9901560, max=136,
 str=0x7f7fffc11e41 "\360\237\225\266Sun\342\200\231s out, savings 
ON\342\200\224shop major appliance deals now",
 str@entry=0x7f7fffc11e40 " \360\237\225\266Sun\342\200\231s out, savings 
ON\342\200\224shop major appliance deals now")
 at sbr/fmt_scan.c:291
 #4  0x00407bf7 in fmt_scan (format=, 
scanlp=0x7e7fe9901560, width=136, dat=dat@entry=0x623c10 ,
 callbacks=callbacks@entry=0x0) at sbr/fmt_scan.c:450
 #5  0x00403dda in scan (inb=inb@entry=0x7e7fe9d83460, 
innum=innum@entry=1, outnum=outnum@entry=0,
 nfs=nfs@entry=0x7e7fe9901500 "%(decode{subject})", width=, 
width@entry=-1, curflg=, unseen=0,
 folder=0x7e7fe9918268 "/tmp/m", size=0, noisy=1, scanl=0x7f7fffc14408) at 
uip/scansbr.c:326
 #6  0x00403435 in main (argc=, argv=) at 
uip/scan.c:256
 (gdb) f 3
 #3  0x00407552 in cpstripped (dest=dest@entry=0x7e7fe9901560, max=136,
 str=0x7f7fffc11e41 "\360\237\225\266Sun\342\200\231s out, savings 
ON\342\200\224shop major appliance deals now",
 str@entry=0x7f7fffc11e40 " \360\237\225\266Sun\342\200\231s out, savings 
ON\342\200\224shop major appliance deals now")
 at sbr/fmt_scan.c:291
 291 assert(w >= 0);
 (gdb) list
 286
 287 prevCtrl = 0;
 288
 289 #ifdef MULTIBYTE_SUPPORT
 290 w = wcwidth(wide_char);
 291 assert(w >= 0);
 292 if (max >= (size_t) w) {
 293 charstring_push_back_chars (dest, altstr ? altstr : str, 
char_len, w);
 294 max -= w;
 295 str += char_len;
 (gdb) print w
 $1 = 
 (gdb) print wide_char
 $2 = 128374 L'\x1f576'
 (gdb) quit

L'\x1f576' (in wide_char) is probably the `dark sunglasses' (U+1F576)
unicode character and directly trying to:

 wcwidth(L'\x1f576')

...returns `-1'.

If you need any further information please let me know!

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi,

> Assertion failed: (w >= 0), function cpstripped, file
> ../nmh/sbr/fmt_scan.c, line 291.

char_len = mbtowc(_char, str, len);
if (char_len < 0) {
altstr = "?";
char_len = mbtowc(_char, altstr, 1);
}

I wonder if mbtowc()'s internal shift state needs resetting after the
first call returns -1 before it's asked to translate "?".

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi Norm,

> scan applied to an Email like the attached dumps core. I will send
> more particulars, on request.

Could you run this scan command with 1.7-RC1 to make sure it still has
the assert(3) failure on the problem email.

scan -format '%(decode{subject})' cur

And then repeat that but under the control of ltrace(1) to log scan's
calls to a couple of interesting C library routines.  ltrace might need
a package install, it's typically in a package of the same name.

ltrace -o /tmp/scan.lt \
-e 'mbtowc+wcwidth+iswcntrl+iswspace-@libc*' \
scan -format '%(decode{subject})' cur

The -o sets the output file for the log, that's what I'd like to see,
and the -e specifies the filter of what to log.  In this case, those
four functions, but not if called by libc internally.

By comparing to local output where I don't see the problem, I'm hoping
there will be a difference that explains the assertion failure.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[norm]

Ken Hornstein  writes:
>>Does the following command produce readable output, or a complaint?
>>
>>$ printf %s 
>>8J+VtlN1buKAmXMgb3V0LCBzYXZpbmdzIE9O4oCUc2hvcCBtYWpvciBhcHBsaWFuY2UgZGVhbHMgbm93
>> | base64 -d | iconv -f utf-8
>
>For me, that works fine.  Since I can reproduce it I'll gladly dig into it,
>but not for a week or so.

inc seems to have a similar, if not identical problem. I have decided to
revert back to nmh-1.6+dev_20150118, until the problem gets fixed, lest I lose
some important mail.

Of course, I will run any tests with nmh-1.7-RC1, that anybody asks me to.

Norman Shapiro

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[norm]

David Levine  writes:
>Does the following command produce readable output, or a complaint?
>
>$ printf %s 
>8J+VtlN1buKAmXMgb3V0LCBzYXZpbmdzIE9O4oCUc2hvcCBtYWpvciBhcHBsaWFuY2UgZGVhbHMgbm93
> | base64 -d | iconv -f utf-8

A readable output -- sort of.

Norman Shapiro

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ken Hornstein]

>Does the following command produce readable output, or a complaint?
>
>$ printf %s 
>8J+VtlN1buKAmXMgb3V0LCBzYXZpbmdzIE9O4oCUc2hvcCBtYWpvciBhcHBsaWFuY2UgZGVhbHMgbm93
> | base64 -d | iconv -f utf-8

For me, that works fine.  Since I can reproduce it I'll gladly dig into it,
but not for a week or so.

--Ken

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[David Levine]

Norm wrote:

> % locale
> LANG=en_US.utf8

So is mine.

Does the following command produce readable output, or a complaint?

$ printf %s 
8J+VtlN1buKAmXMgb3V0LCBzYXZpbmdzIE9O4oCUc2hvcCBtYWpvciBhcHBsaWFuY2UgZGVhbHMgbm93
 | base64 -d | iconv -f utf-8

David

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ralph Corderoy]

Hi Ken,

> Ooof, I get:

Like David, I don't.  The Subject is two lots of base64 that I think
decode to

  f0 9f 95 b6 53 75 6e e2  80 99 73 20 6f 75 74 2c  |Sun...s out,|
0010  20 73 61 76 69 6e 67 73  20 4f 4e e2 80 94 73 68  | savings ON...sh|
0020  6f 70 20 6d 61 6a 6f 72  20 61 70 70 6c 69 61 6e  |op major applian|
0030  63 65 20 64 65 61 6c 73  20 6e 6f 77 0a   |ce deals now.|
003d

What's your locale?

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[norm]

David Levine  writes:
>Norm wrote:
>
>> scan applied to an Email like the attached dumps core. I will send more
>> particulars, on request.
>
>Doesn't dump core for me, and valgrind doesn't notice anything unusual.
>
>I also tried with your .mh_profile that you posted 16 Nov 2016, with
>no problems.  If you have updated it since then, please send it to
>me or post it and I'll try that.

I attached it to this Email.

>If you haven't updated your profile, and you have valgrind
>installed, you might try prepending your scan invocation and see what
>it has to say:
>
>$ valgrind scan cur
>
>If you're not still running RedHat (or was it CentOS?), please let us
>know what distribution you're running.

I am running Red Hat Enterprise Linux Workstation release 6.9 .

Norman Shapiro
MH-Profile-Version: 1.0
Local-Mailbox: n...@dad.org
postproc: /home/norm/Mail/bin/postproc
Path: Mail
editor: neo
Aliasfile: aliases
repl:  -filter cite -nocc me -cc to -cc cc -anno
#: repl:  -nocc me -cc to -cc cc -anno
#: formatproc: /home/norm/lib/replyfilter
mhstore: -auto -clobber ask
nmh-storage: /t/Mime
sortm: -check -noall
Draft-folder: wind
send: -draftfolder +wind  -server smtp.tsoft.com
#:
#: All sequences are private, i.e. recorded in context
mh-sequences:
#:
Sequence-Negation: not
unseen-sequence: unseen
#:
#postproc:  /usr/local/nmh/lib/post
mhl: -width 1000
moreproc: mhless
#mhshow-show-text/html:  chromium-browser && chromium-browser %F && sleep 4
#:mhshow-show-text/html: google-chrome && google-chrome %F
mhshow-show-text/html: firefox %F && sleep 4
refile: -retainsequences
Alternate-Mailboxes:
   acte...@dad.org, americanhomeshi...@dad.org, cleanersup...@dad.org,
   staroff...@dad.org, thinkm...@dad.org, u...@dad.org, westernun...@dad.org,
   yout...@dad.org, ama...@dad.org, a...@dad.org, associated...@dad.org,
   beansh...@dad.org, brownad...@dad.org, c...@dad.org, d...@dad.org,
   et...@dad.org, exab...@dad.org, firecrac...@dad.org, fl...@dad.org,
   goo...@dad.org, h...@dad.org, installshi...@dad.org, i...@dad.org,
   kitc...@dad.org, kitchen...@dad.org, la...@dad.org, networkwo...@dad.org,
   n...@dad.dad.org, n...@dad.org, n...@gad.dad.org, n...@smtp.rawbw.com,
   o...@dad.org, og...@dad.org, p...@dad.org, rbl...@dad.org, red...@dad.org,
   rei...@dad.org, safe...@dad.org, s...@dad.org, s...@tsoft.com,
   sourcefo...@dad.org, spamassas...@dad.org, t...@dad.org, ubta...@dad.org,
   v...@dad.org, walgre...@dad.org, weat...@dad.org, wellsfa...@dad.org,
   your...@dad.org, empowe...@dad.org
___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[David Levine]

Norm wrote:

> scan applied to an Email like the attached dumps core. I will send more
> particulars, on request.

Doesn't dump core for me, and valgrind doesn't notice anything unusual.

I also tried with your .mh_profile that you posted 16 Nov 2016, with
no problems.  If you have updated it since then, please send it to
me or post it and I'll try that.

If you haven't updated your profile, and you have valgrind
installed, you might try prepending your scan invocation and see what
it has to say:

$ valgrind scan cur

If you're not still running RedHat (or was it CentOS?), please let us
know what distribution you're running.

David

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers

Re: [Nmh-workers] nmh-1.7-RC1: scan with complex subjects dumps core

[Ken Hornstein]

>scan applied to an Email like the attached dumps core. I will send more
>particulars, on request.

Ooof, I get:

Assertion failed: (w >= 0), function cpstripped, file ../nmh/sbr/fmt_scan.c, 
line 291.
Abort (core dumped)

Okay!  That will get worked on!

--Ken

___
Nmh-workers mailing list
Nmh-workers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/nmh-workers