Re: built-in pager and utf-8
On Thu, Jul 26, 2012 at 06:00:14PM +0200, Harald Weis wrote: On Mon, Jul 23, 2012 at 12:55:38PM -0500, Derek Martin wrote: I've worked some hours the last days to find fonts in my system which work for the pager. I've put the result in .Xdefaults: Great to hear. I'll be honest, I don't understand why it worked with your old font in some programs, and not in Mutt. But I guess as long as it works, there's nothing else to do. ;) ! The following fonts do not work with mutt's builtin pager, ! but they do work for w3m, less, joe and emacs: ! !XTerm*font: -misc-fixed-medium-r-normal--14-130-75-75-c-70-iso10646-1 !XTerm*font: -misc-fixed-medium-r-normal--15-140-75-75-c-90-iso10646-1 That seems bizarre to me. If you do try it, I use that in my xterms with the following resource setting: XTerm*font: -misc-fixed-medium-r-semicondensed-*-13-*-*-*-*-*-iso10646-* Where have you got this one from ? It's the same as the one you posted: -misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1 The only difference is mine has wild cards. It's the only font that I have installed that matches though, so the wild cards don't really change anything. libncursesw.so.8 versus libncurses.so.8 ! Could that make the difference ? Is +HAVE_CURS_SET or +HAVE_WC_FUNCS responsible here ? That seems like it should be unlikely, but it's possible. They should be the same library, except the 'w' version has wide character support. Normally you need the wide character support for mutt to work correctly with Unicode, and I would expect the same to be true with the other programs as well. It's possible that the lib is buggy though... Programs using the non-w version might work by virtue of the fact that they're just dumping the raw bytes to your terminal, which is interpreting the UTF-8 sequences correctly. Okay, basically the problem is solved. And I like the GNU Unifont. Me too. :) Thank you very much, Derek, for your great help. Glad to have helped, though I'm not really sure what I did. :) -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgp3ibJ1BIR5J.pgp Description: PGP signature
Re: built-in pager and utf-8
On Mon, Jul 23, 2012 at 12:55:38PM -0500, Derek Martin wrote:
I'm at a loss... Everything looks to be configured right on your
system, and the e-mail displays correctly in Mutt for me; the hex dump
was also not especially helpful. I would normally be inclined to
point to your font, but you say when you use w3m as your pager, it
Not only w3m, it works for less, joe and emacs as well !
works. That has me stumped. So, in spite of that, you might try
installing the GNU Unifont (if you haven't already), and using that
instead, and see if that helps -- though based on what you've said, I
wouldn't expect so.
I've worked some hours the last days to find fonts in my system
which work for the pager. I've put the result in .Xdefaults:
! The following fonts work with mutt's builtin pager:
!
!XTerm*font: -schumacher-clean-medium-r-normal--12-120-75-75-c-60-iso10646-1
!XTerm*font: -misc-DejaVu Sans Mono-medium-r-normal--0-0-0-0-m-0-iso10646-1
!XTerm*font: -misc-fixed-medium-r-normal-ja-13-120-75-75-c-120-iso10646-1
!XTerm*font: -misc-fixed-medium-r-normal-ko-18-120-100-100-c-180-iso10646-1
!XTerm*font: -misc-fixed-medium-r-semicondensed--12-110-75-75-c-60-iso10646-1
XTerm*font: -gnu-unifont-medium-r-normal--16-160-75-75-c-80-iso10646-1
! The following fonts work with mutt's builtin pager
! except for one symbol ('∧'):
!
!XTerm*font: -adobe-courier-medium-r-normal--14-100-100-100-m-90-iso10646-1
!XTerm*font: -adobe-courier-medium-r-normal--14-140-75-75-m-90-iso10646-1
!XTerm*font:
-bh-lucidatypewriter-medium-r-normal-sans-14-140-75-75-m-90-iso10646-1
! The following font works with mutt's builtin pager
! except for two special arrow symbols ('⇔' and '⇒'):
!
!XTerm*font: -misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1
! The following fonts do not work with mutt's builtin pager,
! but they do work for w3m, less, joe and emacs:
!
!XTerm*font: -misc-fixed-medium-r-normal--14-130-75-75-c-70-iso10646-1
!XTerm*font: -misc-fixed-medium-r-normal--15-140-75-75-c-90-iso10646-1
If you do try it, I use that in my xterms with the following resource
setting:
XTerm*font: -misc-fixed-medium-r-semicondensed-*-13-*-*-*-*-*-iso10646-*
Where have you got this one from ?
Presumably you've already made sure you're running with the latest
versions of ncursesw, iconv, and any other relevant system libraries
(i.e. your system is up to date). The last thing I can think to look
at is to also run ldd on the w3m binary, and compare the sorted output
to the sorted output of ldd mutt (perhaps with diff). This might
suggest a different library that one is using, which may be a problem.
me@pollux:~ % ldd `which mutt`
/usr/local/bin/mutt:
libncursesw.so.8 = /lib/libncursesw.so.8 (0x2810d000)
libgssapi.so.10 = /usr/lib/libgssapi.so.10 (0x2815b000)
libheimntlm.so.10 = /usr/lib/libheimntlm.so.10 (0x28164000)
libkrb5.so.10 = /usr/lib/libkrb5.so.10 (0x28169000)
libhx509.so.10 = /usr/lib/libhx509.so.10 (0x281c7000)
libcom_err.so.5 = /usr/lib/libcom_err.so.5 (0x281fd000)
libcrypto.so.6 = /lib/libcrypto.so.6 (0x281ff000)
libasn1.so.10 = /usr/lib/libasn1.so.10 (0x2835b000)
libroken.so.10 = /usr/lib/libroken.so.10 (0x283d)
libcrypt.so.5 = /lib/libcrypt.so.5 (0x283e)
libssl.so.6 = /usr/lib/libssl.so.6 (0x28406000)
libintl.so.9 = /usr/local/lib/libintl.so.9 (0x2844f000)
libiconv.so.3 = /usr/local/lib/libiconv.so.3 (0x28458000)
libc.so.7 = /lib/libc.so.7 (0x2855)
me@pollux:~ % ldd `which less`
/usr/bin/less:
libncurses.so.8 = /lib/libncurses.so.8 (0x280a8000)
libc.so.7 = /lib/libc.so.7 (0x280e8000)
me@pollux:~ % ldd `which joe`
/usr/local/bin/joe:
libm.so.5 = /lib/libm.so.5 (0x280e4000)
libncurses.so.8 = /lib/libncurses.so.8 (0x280fe000)
libutil.so.8 = /lib/libutil.so.8 (0x2813e000)
libiconv.so.3 = /usr/local/lib/libiconv.so.3 (0x2814d000)
libc.so.7 = /lib/libc.so.7 (0x28245000)
me@pollux:~ % ldd `which w3m`
/usr/local/bin/w3m:
libm.so.5 = /lib/libm.so.5 (0x2810)
libgc.so.1 = /usr/local/lib/libgc.so.1 (0x2811a000)
libssl.so.6 = /usr/lib/libssl.so.6 (0x28155000)
libcrypto.so.6 = /lib/libcrypto.so.6 (0x2819e000)
libncurses.so.8 = /lib/libncurses.so.8 (0x282fa000)
libc.so.7 = /lib/libc.so.7 (0x2833a000)
The next one filtered because too long otherwise.
me@pollux:~ % ldd `which emacs` |grep libncurses
libncurses.so.8 = /lib/libncurses.so.8 (0x28e13000)
libncursesw.so.8 versus libncurses.so.8 ! Could that make the difference ?
Is +HAVE_CURS_SET or +HAVE_WC_FUNCS responsible here ?
Okay, basically the problem is solved. And I like the GNU Unifont.
Thank you very much, Derek, for your great help.
Our mutt port is version 1.5 since 22-Jul-12.
I hesitate to upgrade again
Re: built-in pager and utf-8
Harald, I'm at a loss... Everything looks to be configured right on your system, and the e-mail displays correctly in Mutt for me; the hex dump was also not especially helpful. I would normally be inclined to point to your font, but you say when you use w3m as your pager, it works. That has me stumped. So, in spite of that, you might try installing the GNU Unifont (if you haven't already), and using that instead, and see if that helps -- though based on what you've said, I wouldn't expect so. If you do try it, I use that in my xterms with the following resource setting: XTerm*font: -misc-fixed-medium-r-semicondensed-*-13-*-*-*-*-*-iso10646-* Presumably you've already made sure you're running with the latest versions of ncursesw, iconv, and any other relevant system libraries (i.e. your system is up to date). The last thing I can think to look at is to also run ldd on the w3m binary, and compare the sorted output to the sorted output of ldd mutt (perhaps with diff). This might suggest a different library that one is using, which may be a problem. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. On Fri, Jul 20, 2012 at 04:59:36PM +0200, Harald Weis wrote: On Tue, Jul 17, 2012 at 12:27:37PM -0500, Derek Martin wrote: On Tue, Jul 17, 2012 at 04:12:03PM +0200, Harald Weis wrote: Output of mutt -v is: Mutt 1.4.2.3i (2007-05-26) As I've already suggested, at least if we don't get you fixed with this version, you should really consider getting the latest (1.5.21) and compiling from scratch. It's not that hard, and it may contain No problem for me. Have followed this advice and reported in post dated 8 Jul 2012. The port is mutt-devel-1.5.21_4 for us. Didn't help. So I recompiled my original mutt. bug fixes that eliminate your issues. Leaving this in for easy reference: System: FreeBSD 8.3-RELEASE (i386) [using ncurses 5.7] Compile options: -DOMAIN -DEBUG -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE -USE_FCNTL +USE_FLOCK +USE_POP +USE_IMAP +USE_GSS +USE_SSL -USE_SASL +HAVE_REGCOMP -USE_GNU_REGEX +COMPRESSED +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +HAVE_PGP -BUFFY_SIZE -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS +LOCALES_HACK +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_GETSID +HAVE_GETADDRINFO -ISPELL SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/local/share/mutt SYSCONFDIR=/usr/local/etc EXECSHELL=/bin/sh -MIXMASTER To contact the developers, please mail to mutt-...@mutt.org. To report a bug, please use the flea(1) utility. vvv.initials 1.3.28.nr.threadcomplete rr.compressed Does everybody think - when reading this message - that my case is hopeless ? It's not hopeless, but it's hard to guess how to help you from here. I know you may have answered some of these questions already, but it might be helpful to get the answers to all of the following questions in one place, to compare the different settings (and I've lost the beginning of the thread). So, if you still want help, please provide the answers/output to all of the following: What is the output of 'locale -a |grep $LANG' (without the single quotes) on your system? me@pollux:~ % locale -a | grep $LANG en_US.UTF-8 What is the output of 'locale' (without the single quotes) on your system? me@pollux:~ % locale LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_TIME=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_ALL= What is the output of 'locale' FROM WITHIN MUTT (press '!' to ask mutt for a shell prompt, then type localeenter)? LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_TIME=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_ALL= Press any key to continue... What terminal emulator are you using? Xterm since ages and tmux since several months now. Under tmux: me@pollux:~ % setenv | grep TERM TERM=screen XTERM_VERSION=X.Org 6.8.99.903(267) XTERM_LOCALE=en_US.UTF-8 TERMCAP=xterm|xterm-color|X11 terminal
Re: built-in pager and utf-8
On Tue, Jul 17, 2012 at 12:27:37PM -0500, Derek Martin wrote: On Tue, Jul 17, 2012 at 04:12:03PM +0200, Harald Weis wrote: Output of mutt -v is: Mutt 1.4.2.3i (2007-05-26) As I've already suggested, at least if we don't get you fixed with this version, you should really consider getting the latest (1.5.21) and compiling from scratch. It's not that hard, and it may contain No problem for me. Have followed this advice and reported in post dated 8 Jul 2012. The port is mutt-devel-1.5.21_4 for us. Didn't help. So I recompiled my original mutt. bug fixes that eliminate your issues. Leaving this in for easy reference: System: FreeBSD 8.3-RELEASE (i386) [using ncurses 5.7] Compile options: -DOMAIN -DEBUG -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE -USE_FCNTL +USE_FLOCK +USE_POP +USE_IMAP +USE_GSS +USE_SSL -USE_SASL +HAVE_REGCOMP -USE_GNU_REGEX +COMPRESSED +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +HAVE_PGP -BUFFY_SIZE -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS +LOCALES_HACK +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_GETSID +HAVE_GETADDRINFO -ISPELL SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/local/share/mutt SYSCONFDIR=/usr/local/etc EXECSHELL=/bin/sh -MIXMASTER To contact the developers, please mail to mutt-...@mutt.org. To report a bug, please use the flea(1) utility. vvv.initials 1.3.28.nr.threadcomplete rr.compressed Does everybody think - when reading this message - that my case is hopeless ? It's not hopeless, but it's hard to guess how to help you from here. I know you may have answered some of these questions already, but it might be helpful to get the answers to all of the following questions in one place, to compare the different settings (and I've lost the beginning of the thread). So, if you still want help, please provide the answers/output to all of the following: What is the output of 'locale -a |grep $LANG' (without the single quotes) on your system? me@pollux:~ % locale -a | grep $LANG en_US.UTF-8 What is the output of 'locale' (without the single quotes) on your system? me@pollux:~ % locale LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_TIME=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_ALL= What is the output of 'locale' FROM WITHIN MUTT (press '!' to ask mutt for a shell prompt, then type localeenter)? LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_TIME=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_ALL= Press any key to continue... What terminal emulator are you using? Xterm since ages and tmux since several months now. Under tmux: me@pollux:~ % setenv | grep TERM TERM=screen XTERM_VERSION=X.Org 6.8.99.903(267) XTERM_LOCALE=en_US.UTF-8 TERMCAP=xterm|xterm-color|X11 terminal emulator:ti@:te@:k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:k7=\E[18~:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:@7=\EOF:@8=\EOM:kI=\E[2~:kh=\EOH:kP=\E[5~:kN=\E[6~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:Km=\E[M:li#67:co#139:am:kn#12:km:mi:ms:xn:AX:bl=^G:is=\E[!p\E[?3;4l\E[4l\E:rs=\E[!p\E[?3;4l\E[4l\E:le=^H:AL=\E[%dL:DL=\E[%dM:DC=\E[%dP:al=\E[L:dc=\E[P:dl=\E[M:UP=\E[%dA:DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:ho=\E[H:cd=\E[J:ce=\E[K:cl=\E[H\E[2J:cm=\E[%i%d;%dH:cs=\E[%i%d;%dr:im=\E[4h:ei=\E[4l:ks=\E[?1h\E=:ke=\E[?1l\E:kD=\E[3~:sf=\n:sr=\EM:st=\EH:ct=\E[3g:sc=\E7:rc=\E8:eA=\E(B\E)0:as=\E(0:ae=\E(B:ml=\El:mu=\Em:up=\E[A:nd=\E[C:md=\E[1m:me=\E[m:mr=\E[7m:so=\E[7m:se=\E[27m:us=\E[4m:ue=\E[24m:vi=\E[?25l:ve=\E[?25h:ut:Co#8:pa#64:op=\E[39;49m:AB=\E[4%dm:AF=\E[3%dm:kb=\010: XTERM_SHELL=/bin/tcsh Under (pure) xterm: me@pollux:~ % setenv | grep TERM TERM=xterm-color XTERM_VERSION=X.Org 6.8.99.903(267) XTERM_LOCALE=en_US.UTF-8 TERMCAP=xterm|xterm-color|X11 terminal emulator:ti@:te@:k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:k7=\E[18~:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:@7=\EOF:@8=\EOM:kI=\E[2~:kh=\EOH:kP=\E[5~:kN=\E[6~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:Km=\E[M:li#40:co#80:am:kn#12:km:mi:ms:xn:AX:bl=^G:is=\E[!p\E[?3;4l\E[4l\E:rs=\E[!p\E[?3;4l\E[4l\E:le=^H:AL=\E[%dL:DL=\E[%dM:DC=\E[%dP:al=\E[L:dc=\E[P:dl=\E[M:UP=\E[%dA:DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:ho=\E[H:cd=\E[J:ce=\E[K:cl=\E[H\E[2J:cm=\E[%i%d;%dH:cs=\E[%i%d;%dr:im=\E[4h:ei=\E[4l:ks=\E[?1h\E=:ke=\E[?1l\E:kD=\E[3~:sf=\n:sr=\EM:st=\EH:ct=\E[3g:sc=\E7:rc=\E8:eA=\E(B\E)0:as=\E(0:ae=\E(B:ml=\El:mu=\Em:up=\E[A:nd=\E[C:md=\E[1m:me=\E[m:mr=\E[7m:so=\E[7m:se=\E[27m:us=\E[4m:ue=\E[24m:vi=\E[?25l:ve=\E[?25h:ut:Co#8:pa#64:op=\E[39;49m:AB=\E[4%dm:AF=\E[3%dm:kb=\010: XTERM_SHELL=/bin/tcsh (Sorry for the long TERMCAP lines, joe's wrap option does not work in this
Re: built-in pager and utf-8
On Sun, Jul 08, 2012 at 06:58:20PM +0200, Harald Weis wrote: On Tue, Jul 03, 2012 at 11:15:03AM +0900, Daniel P. Wright wrote: I am running mutt 1.5.21 compiled from the mutt repo (no ports) on FreeBSD with no trouble. The parameters I passed to prepare are as follows: ./prepare --prefix=/usr/local --enable-locales-fix --disable-fcntl --enable-hcache --with-libiconv-prefix=/usr/local --with-slang=/usr/local (I am using slang over ncurses because of weird colour issues I was getting with curses; that part is optional) Having said that, I just tried to build mutt from ports using defaults options (version 1.4.2.3i, curses, not slang; locales fix enabled), and it displays unicode fine. If you did want to use 1.5.21, the mutt-devel port is usually up-to-date. Did you check your locale settings? Mine are as follows: LANG=ja_JP.UTF-8 LC_ALL=ja_JP.UTF-8 LC_CTYPE=ja_JP.UTF-8 You probably don't want Japanese, but some sort of locale setting with UTF-8 in the name ought to do you. I can confirm these settings are working on my FreeBSD machine with both the mutt and mutt-devel ports, and with my own version compiled from sources. Hope that helps. -Dani. Following the FreeBSD Handbook, I use login classes set in /etc/login.conf. If I remember correctly, nothing else is required. me@pollux:~ % setenv |grep UTF MM_CHARSET=UTF-8 LANG=en_US.UTF-8 XTERM_LOCALE=en_US.UTF-8 -- No change whatsoever if I add in .cshrc the following lines. setenv LC_ALL en_US.UTF-8 setenv LC_CTYPEen_US.UTF-8 -- Not surprising. muttrc contains the following lines. me@pollux:~ % egrep charset .mutt/muttrc #set charset=utf-8# never EVER EVER EVER set $charset yourself #set send_charset=us-ascii:iso-8859-15:utf-8 # A list of character sets # for outgoing messages (Default: us-ascii:iso-8859-1:utf-8) charset-hook ^unknown-8bit$ WINDOWS-1252 charset-hook ^x-user-defined$ WINDOWS-1252 charset-hook ^ISO-8859-1$ WINDOWS-1252 charset-hook ^US-ASCII$ WINDOWS-1252 charset-hook ^none$ WINDOWS-1252 charset-hook ^ISO-8859-8-1$ ISO-8859-8 charset-hook ^GB2312$ GB18030 -- The charset and charset-hook lines come from this mailing list. Output of mutt -v is: Mutt 1.4.2.3i (2007-05-26) Copyright (C) 1996-2002 Michael R. Elkins and others. Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'. Mutt is free software, and you are welcome to redistribute it under certain conditions; type `mutt -vv' for details. System: FreeBSD 8.3-RELEASE (i386) [using ncurses 5.7] Compile options: -DOMAIN -DEBUG -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE -USE_FCNTL +USE_FLOCK +USE_POP +USE_IMAP +USE_GSS +USE_SSL -USE_SASL +HAVE_REGCOMP -USE_GNU_REGEX +COMPRESSED +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +HAVE_PGP -BUFFY_SIZE -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS +LOCALES_HACK +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_GETSID +HAVE_GETADDRINFO -ISPELL SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/local/share/mutt SYSCONFDIR=/usr/local/etc EXECSHELL=/bin/sh -MIXMASTER To contact the developers, please mail to mutt-...@mutt.org. To report a bug, please use the flea(1) utility. vvv.initials 1.3.28.nr.threadcomplete rr.compressed -- Installing mutt-devel does not help. No change, with the exception that mutt-devel does not use .mailcap, God knows why. So back to mutt. Everything okay, excepting the builtin pager not displaying the famous (multi-byte) characters. Finally, using the enter-command function in the index menu, I set the pager to w3m. That works! What is the missing option here? Does everybody think - when reading this message - that my case is hopeless ? Harald :-)
Re: built-in pager and utf-8
Harald Weis wrote: Does everybody think - when reading this message - that my case is hopeless ? I don't see why. I don't have any problems reading UTF-8 in the pager - see attached. This is a pretty bog standard mutt on Debian; mutt -v gives Mutt 1.5.21 (2010-09-15) Copyright (C) 1996-2009 Michael R. Elkins and others. Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'. Mutt is free software, and you are welcome to redistribute it under certain conditions; type `mutt -vv' for details. System: Linux 3.2.0-3-amd64 (x86_64) ncurses: ncurses 5.9.20110404 (compiled with 5.9) libidn: 1.25 (compiled with 1.25) hcache backend: tokyocabinet 1.4.47 Compile options: -DOMAIN +DEBUG -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE +USE_FCNTL -USE_FLOCK +USE_POP +USE_IMAP +USE_SMTP -USE_SSL_OPENSSL +USE_SSL_GNUTLS +USE_SASL +USE_GSS +HAVE_GETADDRINFO +HAVE_REGCOMP -USE_GNU_REGEX +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME +CRYPT_BACKEND_GPGME -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS -LOCALES_HACK +COMPRESSED +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_LIBIDN +HAVE_GETSID +USE_HCACHE -ISPELL SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/share/mutt SYSCONFDIR=/etc EXECSHELL=/bin/sh MIXMASTER=mixmaster attachment: mutt-utf8.png
Re: built-in pager and utf-8
On Tue, Jul 17, 2012 at 04:12:03PM +0200, Harald Weis wrote: Following the FreeBSD Handbook, I use login classes set in /etc/login.conf. If I remember correctly, nothing else is required. me@pollux:~ % setenv |grep UTF MM_CHARSET=UTF-8 LANG=en_US.UTF-8 XTERM_LOCALE=en_US.UTF-8 This is not the best way to get this info, FWIW. See below. No change whatsoever if I add in .cshrc the following lines. setenv LC_ALL en_US.UTF-8 setenv LC_CTYPEen_US.UTF-8 You don't need to set these, if they're the same as LANG. I suggest you make sure they're not set, to make sure your environment is working correctly. charset-hook ^unknown-8bit$ WINDOWS-1252 charset-hook ^x-user-defined$ WINDOWS-1252 charset-hook ^ISO-8859-1$ WINDOWS-1252 charset-hook ^US-ASCII$ WINDOWS-1252 charset-hook ^none$ WINDOWS-1252 charset-hook ^ISO-8859-8-1$ ISO-8859-8 charset-hook ^GB2312$ GB18030 It shouldn't matter, but for the next tests, try commenting these out. Output of mutt -v is: Mutt 1.4.2.3i (2007-05-26) As I've already suggested, at least if we don't get you fixed with this version, you should really consider getting the latest (1.5.21) and compiling from scratch. It's not that hard, and it may contain bug fixes that eliminate your issues. Leaving this in for easy reference: System: FreeBSD 8.3-RELEASE (i386) [using ncurses 5.7] Compile options: -DOMAIN -DEBUG -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE -USE_FCNTL +USE_FLOCK +USE_POP +USE_IMAP +USE_GSS +USE_SSL -USE_SASL +HAVE_REGCOMP -USE_GNU_REGEX +COMPRESSED +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +HAVE_PGP -BUFFY_SIZE -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS +LOCALES_HACK +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_GETSID +HAVE_GETADDRINFO -ISPELL SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/local/share/mutt SYSCONFDIR=/usr/local/etc EXECSHELL=/bin/sh -MIXMASTER To contact the developers, please mail to mutt-...@mutt.org. To report a bug, please use the flea(1) utility. vvv.initials 1.3.28.nr.threadcomplete rr.compressed Does everybody think - when reading this message - that my case is hopeless ? It's not hopeless, but it's hard to guess how to help you from here. I know you may have answered some of these questions already, but it might be helpful to get the answers to all of the following questions in one place, to compare the different settings (and I've lost the beginning of the thread). So, if you still want help, please provide the answers/output to all of the following: What is the output of 'locale -a |grep $LANG' (without the single quotes) on your system? What is the output of 'locale' (without the single quotes) on your system? What is the output of 'locale' FROM WITHIN MUTT (press '!' to ask mutt for a shell prompt, then type localeenter)? What terminal emulator are you using? How do you start the terminal emulator you're running Mutt in (i.e. do you start it manually from the command line, or are you clicking an icon in your desktop environment)? If the answer to the previous question was by clicking on an icon, did you try starting your terminal emulator from the command line, where you know your locale is set correctly? (If not, do so.) What font is your emulator using, and how is it configured/selected? What is the output of 'ldd `which mutt`' on your system? Can you resend a sample e-mail that doesn't display correctly in the pager? (I want to hex dump it to look at the raw bytes.) -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgpkonGxBnQJa.pgp Description: PGP signature
Re: built-in pager and utf-8
On Tue, Jul 03, 2012 at 11:15:03AM +0900, Daniel P. Wright wrote: I am running mutt 1.5.21 compiled from the mutt repo (no ports) on FreeBSD with no trouble. The parameters I passed to prepare are as follows: ./prepare --prefix=/usr/local --enable-locales-fix --disable-fcntl --enable-hcache --with-libiconv-prefix=/usr/local --with-slang=/usr/local (I am using slang over ncurses because of weird colour issues I was getting with curses; that part is optional) Having said that, I just tried to build mutt from ports using defaults options (version 1.4.2.3i, curses, not slang; locales fix enabled), and it displays unicode fine. If you did want to use 1.5.21, the mutt-devel port is usually up-to-date. Did you check your locale settings? Mine are as follows: LANG=ja_JP.UTF-8 LC_ALL=ja_JP.UTF-8 LC_CTYPE=ja_JP.UTF-8 You probably don't want Japanese, but some sort of locale setting with UTF-8 in the name ought to do you. I can confirm these settings are working on my FreeBSD machine with both the mutt and mutt-devel ports, and with my own version compiled from sources. Hope that helps. -Dani. Following the FreeBSD Handbook, I use login classes set in /etc/login.conf. If I remember correctly, nothing else is required. me@pollux:~ % setenv |grep UTF MM_CHARSET=UTF-8 LANG=en_US.UTF-8 XTERM_LOCALE=en_US.UTF-8 -- No change whatsoever if I add in .cshrc the following lines. setenv LC_ALL en_US.UTF-8 setenv LC_CTYPEen_US.UTF-8 -- Not surprising. muttrc contains the following lines. me@pollux:~ % egrep charset .mutt/muttrc #set charset=utf-8# never EVER EVER EVER set $charset yourself #set send_charset=us-ascii:iso-8859-15:utf-8 # A list of character sets # for outgoing messages (Default: us-ascii:iso-8859-1:utf-8) charset-hook ^unknown-8bit$ WINDOWS-1252 charset-hook ^x-user-defined$ WINDOWS-1252 charset-hook ^ISO-8859-1$ WINDOWS-1252 charset-hook ^US-ASCII$ WINDOWS-1252 charset-hook ^none$ WINDOWS-1252 charset-hook ^ISO-8859-8-1$ ISO-8859-8 charset-hook ^GB2312$ GB18030 -- The charset and charset-hook lines come from this mailing list. Output of mutt -v is: Mutt 1.4.2.3i (2007-05-26) Copyright (C) 1996-2002 Michael R. Elkins and others. Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'. Mutt is free software, and you are welcome to redistribute it under certain conditions; type `mutt -vv' for details. System: FreeBSD 8.3-RELEASE (i386) [using ncurses 5.7] Compile options: -DOMAIN -DEBUG -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE -USE_FCNTL +USE_FLOCK +USE_POP +USE_IMAP +USE_GSS +USE_SSL -USE_SASL +HAVE_REGCOMP -USE_GNU_REGEX +COMPRESSED +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +HAVE_PGP -BUFFY_SIZE -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS +LOCALES_HACK +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_GETSID +HAVE_GETADDRINFO -ISPELL SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/local/share/mutt SYSCONFDIR=/usr/local/etc EXECSHELL=/bin/sh -MIXMASTER To contact the developers, please mail to mutt-...@mutt.org. To report a bug, please use the flea(1) utility. vvv.initials 1.3.28.nr.threadcomplete rr.compressed -- Installing mutt-devel does not help. No change, with the exception that mutt-devel does not use .mailcap, God knows why. So back to mutt. Everything okay, excepting the builtin pager not displaying the famous (multi-byte) characters. Finally, using the enter-command function in the index menu, I set the pager to w3m. That works! What is the missing option here? -- Harald
Re: built-in pager and utf-8
On Mon, Jul 02, 2012 at 10:19:56AM +0900, Daniel P. Wright wrote: The example works fine in my mutt. You might find it's a result of your compile settings -- try mutt -v to see them. In particular, look for +HAVE_WC_FUNCS. If you don't have widechar funcs, try recompiling against (or finding a package which is compiled against) ncursesw or slang. Copied from mutt -v: +ENABLE_NLS -LOCALES_HACK +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR What else could be wrong?
Re: built-in pager and utf-8
On Sun, Jul 01, 2012 at 04:59:51PM +0200, Harald Weis wrote: My muttrc sets editor to joe. Mutt's built-in pager does not seem to understand utf-8. Example copied from a gnu-emacs mailing list: The first time is characterized with this system of equations: t₁ = 65536×h₁ + l₁ 0 ≤ h₁ 65536 0 ≤ l₁ 65536 The second time with this similar system: t₂ = 65536×h₂ + l₂ 0 ≤ h₂ 65536 0 ≤ l₂ 65536 Is this right? It seems well-formed, but using 'uncommon' glyphs. In a tty I get a lot of '?' (no glyph for this codepoint - there are only 512 available at most for console fonts), in a term I can see that they are less-than-or-equal and small/subscript '1' and '2'. Also a *lot* of whitespace at the end of most lines! (I highlight redundant whitespace in vim :) What I can't guess is what *you* are seeing which causes you to query it ? I know that the list archive doesn't display it correctly, but that is common for UTF-8 messages in archives. Also, I'm using 1.5.21, you seem to be using 1.4 - no idea if that makes a difference. ĸen [ ken, using what should look like a small russian or greek K and with an o-umlaut or o-diaeresis in .sig ] -- das eine Mal als Tragödie, das andere Mal als Farce
Re: built-in pager and utf-8
On Mon, Jul 02, 2012 at 02:14:21PM +0100, Ken Moffat wrote: On Sun, Jul 01, 2012 at 04:59:51PM +0200, Harald Weis wrote: My muttrc sets editor to joe. Mutt's built-in pager does not seem to understand utf-8. Example copied from a gnu-emacs mailing list: The first time is characterized with this system of equations: t₁ = 65536×h₁ + l₁ 0 ≤ h₁ 65536 0 ≤ l₁ 65536 The second time with this similar system: t₂ = 65536×h₂ + l₂ 0 ≤ h₂ 65536 0 ≤ l₂ 65536 Is this right? It seems well-formed, but using 'uncommon' glyphs. In a tty I get a lot of '?' (no glyph for this codepoint - there are only 512 available at most for console fonts), in a term I can see that they are less-than-or-equal and small/subscript '1' and '2'. Also a *lot* of whitespace at the end of most lines! (I highlight redundant whitespace in vim :) I assume if you paste the equations with a mouse to emacs, the subscripts would be displayed correctly. What I can't guess is what *you* are seeing which causes you to query it ? I see character-size rectangles in place of the subscripts. Everything else is okay. Also, I'm using 1.5.21, you seem to be using 1.4 - no idea if that makes a difference. mutt-1.4.2.3_5 needs updating (port has 1.4.2.3_6) That's all I could do on this FreeBSD system.
Re: built-in pager and utf-8
On Mon, Jul 02, 2012 at 05:09:21PM +0200, Harald Weis wrote: I assume if you paste the equations with a mouse to emacs, the subscripts would be displayed correctly. What I can't guess is what *you* are seeing which causes you to query it ? I see character-size rectangles in place of the subscripts. Everything else is okay. In a terminal emulator window, this usually means that the font you're using has no glyph for the character (though, I believe as another poster mentioned, on the console you'll see question marks instead). Try using a more comlete font for whatever terminal you're using Mutt in. You're using xterm, I bet? There's a mostly-complete Unicode font maintained by the GNU people... Try that. Also, I'm using 1.5.21, you seem to be using 1.4 - no idea if that makes a difference. mutt-1.4.2.3_5 needs updating (port has 1.4.2.3_6) That's all I could do on this FreeBSD system. You should probably upgrade to 1.5.21, it has a lot of bug fixes and enhancements over the 1.4 series. You can always compile it from source... -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgpjNOeGaVYYX.pgp Description: PGP signature
Re: built-in pager and utf-8
Mutt's built-in pager does not seem to understand utf-8. Example copied from a gnu-emacs mailing list: [...] t₁ = 65536×h₁ + l₁ 0 ≤ h₁ 65536 0 ≤ l₁ 65536 Both display here just fine under mutt-1.5.21-r9 (gentoo). Make sure all the components of the displaying chain are set to UTF-8; in mutt, you'll need to configure 'set charset=UTF-8'; your locale must use UTF-8 (see the LC_* environment variables), your terminal emulator must support UTF-8, and ultimately your font as well. -- --|-- | Patrice Levesque http://ptaff.ca/ mutt.wa...@ptaff.ca | --|-- -- signature.asc Description: Digital signature
Re: built-in pager and utf-8
On Sun, Jul 01, 2012 at 11:48:37AM -0400, Patrice Levesque wrote: in mutt, you'll need to configure 'set charset=UTF-8' NO YOU DO NOT, and in fact you should generally never do this. If your locale is set correctly, Mutt will take care of this for you automatically, and if it is not set correctly doing this can cause confusion about what is actually wrong. You should only ever set charset manually if you're doing something weird, AND you really know what you're doing. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgpdq310KdWXL.pgp Description: PGP signature
Re: built-in pager and utf-8
On 2012-07-01, Patrice Levesque wrote: Mutt's built-in pager does not seem to understand utf-8. Example copied from a gnu-emacs mailing list: [...] t₁ = 65536×h₁ + l₁ 0 ≤ h₁ 65536 0 ≤ l₁ 65536 Both display here just fine under mutt-1.5.21-r9 (gentoo). Make sure all the components of the displaying chain are set to UTF-8; in mutt, you'll need to configure 'set charset=UTF-8'; It should not be necessary to set charset as mutt determines that itself from the locale. Regards, Gary
Re: built-in pager and utf-8
On Mon, Jul 02, 2012 at 10:42:39AM -0500, Derek Martin wrote: On Mon, Jul 02, 2012 at 05:09:21PM +0200, Harald Weis wrote: I assume if you paste the equations with a mouse to emacs, the subscripts would be displayed correctly. What I can't guess is what *you* are seeing which causes you to query it ? I see character-size rectangles in place of the subscripts. Everything else is okay. In a terminal emulator window, this usually means that the font you're using has no glyph for the character (though, I believe as another poster mentioned, on the console you'll see question marks instead). Try using a more comlete font for whatever terminal you're using Mutt in. You're using xterm, I bet? There's a mostly-complete Unicode font maintained by the GNU people... Try that. Yes, mutt is running in xterm. The trouble is only in the pager where all subscripts are replaced by character-size rectangles. If I reply to the message I am in joe and everything is fine. That means I _have_got_ the glyphs, haven't I? Also, I'm using 1.5.21, you seem to be using 1.4 - no idea if that makes a difference. mutt-1.4.2.3_5 needs updating (port has 1.4.2.3_6) That's all I could do on this FreeBSD system. You should probably upgrade to 1.5.21, it has a lot of bug fixes and enhancements over the 1.4 series. You can always compile it from source... I do always compile. But the ports tree which is up-to-date has only 1.4.2.3_6 . Not sure whether 1.5.21 will compile for me. Thanks to all of you, Harald
Re: built-in pager and utf-8
I am running mutt 1.5.21 compiled from the mutt repo (no ports) on FreeBSD with no trouble. The parameters I passed to prepare are as follows: ./prepare --prefix=/usr/local --enable-locales-fix --disable-fcntl --enable-hcache --with-libiconv-prefix=/usr/local --with-slang=/usr/local (I am using slang over ncurses because of weird colour issues I was getting with curses; that part is optional) Having said that, I just tried to build mutt from ports using defaults options (version 1.4.2.3i, curses, not slang; locales fix enabled), and it displays unicode fine. If you did want to use 1.5.21, the mutt-devel port is usually up-to-date. Did you check your locale settings? Mine are as follows: LANG=ja_JP.UTF-8 LC_ALL=ja_JP.UTF-8 LC_CTYPE=ja_JP.UTF-8 You probably don't want Japanese, but some sort of locale setting with UTF-8 in the name ought to do you. I can confirm these settings are working on my FreeBSD machine with both the mutt and mutt-devel ports, and with my own version compiled from sources. Hope that helps. -Dani. Harald Weis (Mon, Jul 02, 2012 at 11:26:08PM +0200) On Mon, Jul 02, 2012 at 10:42:39AM -0500, Derek Martin wrote: On Mon, Jul 02, 2012 at 05:09:21PM +0200, Harald Weis wrote: I assume if you paste the equations with a mouse to emacs, the subscripts would be displayed correctly. What I can't guess is what *you* are seeing which causes you to query it ? I see character-size rectangles in place of the subscripts. Everything else is okay. In a terminal emulator window, this usually means that the font you're using has no glyph for the character (though, I believe as another poster mentioned, on the console you'll see question marks instead). Try using a more comlete font for whatever terminal you're using Mutt in. You're using xterm, I bet? There's a mostly-complete Unicode font maintained by the GNU people... Try that. Yes, mutt is running in xterm. The trouble is only in the pager where all subscripts are replaced by character-size rectangles. If I reply to the message I am in joe and everything is fine. That means I _have_got_ the glyphs, haven't I? Also, I'm using 1.5.21, you seem to be using 1.4 - no idea if that makes a difference. mutt-1.4.2.3_5 needs updating (port has 1.4.2.3_6) That's all I could do on this FreeBSD system. You should probably upgrade to 1.5.21, it has a lot of bug fixes and enhancements over the 1.4 series. You can always compile it from source... I do always compile. But the ports tree which is up-to-date has only 1.4.2.3_6 . Not sure whether 1.5.21 will compile for me. Thanks to all of you, Harald
Re: built-in pager and utf-8
On Sun, Jul 01, 2012 at 04:59:51PM +0200, Harald Weis wrote: My muttrc sets editor to joe. Mutt's built-in pager does not seem to understand utf-8. Yes, it does. I read UTF-8 mail in non-Latin charsets all the time. There is, however, a bug in how Mutt deals with whitespace, which on occasion causes it to misinterpret certain UTF-8 sequences. So you could be seeing that, but it's more likely the case that there's some other problem with your environment. Unfortunately right at this moment I'm reading mail from a non-unicode-enabled terminal, so I can't read your example even if it is right. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgplfDDa5Ritu.pgp Description: PGP signature
Re: built-in pager and utf-8
The example works fine in my mutt. You might find it's a result of your compile settings -- try mutt -v to see them. In particular, look for +HAVE_WC_FUNCS. If you don't have widechar funcs, try recompiling against (or finding a package which is compiled against) ncursesw or slang. Harald Weis (Sun, Jul 01, 2012 at 04:59:51PM +0200) My muttrc sets editor to joe. Mutt's built-in pager does not seem to understand utf-8. Example copied from a gnu-emacs mailing list: The first time is characterized with this system of equations: t₁ = 65536×h₁ + l₁ 0 ≤ h₁ 65536 0 ≤ l₁ 65536 The second time with this similar system: t₂ = 65536×h₂ + l₂ 0 ≤ h₂ 65536 0 ≤ l₂ 65536 Is this right? -- Harald Weis
