Re: [MlMt] text/plain?
Ok, here’s what’s going on: 1. The message is encoded in windows-1252, and contains non-breaking spaces (encoded as the byte 0xA0). 2. Your terminal is using a different character encoding (probably UTF-8), in which 0xA0 (as used) does not map to a character. (In UTF-8, 0xA0 is a “continuation character”, which is only valid as a non-first byte in a multi-byte sequence.) In the absence of a valid UTF-8 character code, the terminal gives up and displays the value of the unmappable byte in angle brackets. However, “” is only how the terminal displays the nbsp character. The .eml file itself does not contain the four-character string “”. Dealing with non-breaking spaces can be confusing, since they are difficult to differentiate from normal spaces in many editors, and they have different character codes in the common 8-bit encodings and UTF-8. But they are usually encoded either as the single byte 0xA0 (in the 8-bit encodings) or as the two-byte sequence 0xC2A0 (in UTF-8). As others have pointed out, a quick call to `tr '\240' ' '` to translate the nbsps to normal spaces will often do the trick. If you happen to be using perl, `use feature “unicode_strings”` will make pattern-matching behave properly with nbsps (yes, even with strings from windows-1252-encoded files!) — for example, it will make `\s` match nbsps, which it normally doesn’t. Best of luck with the scripting! Galen On 6 Apr 2019, at 4:27, Randy Bush wrote: i receive an email User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0) Gecko/20100101 PostboxApp/6.1.13 MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US the text has funny space characters that i see if i save the text to disk and look at it with less 0. flo: 2.31 2018.11.03 1. CLIMATE ACTION * (N)ew (M)odify (D)elete..: N 2. * NAME OF CLOUD: cumulus i presume the sender is thunderbird and they have created the text with some sort of windows encoding on a mac? how can i save the content as vanilla ascii text? randy ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] text/plain?
On 6 Apr 2019, at 7:27, Randy Bush wrote: i receive an email User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0) Gecko/20100101 PostboxApp/6.1.13 MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US the text has funny space characters that i see if i save the text to disk and look at it with less 0. flo: 2.31 2018.11.03 1. CLIMATE ACTION * (N)ew (M)odify (D)elete..: N 2. * NAME OF CLOUD: cumulus i presume the sender is thunderbird and they have created the text with some sort of windows encoding on a mac? Apparently. I've seen this in mail recently as well and suspect it may be a new default in Postbox or TBird or both to use windows-1252 instead of UTF-8. Windows-1252 is a close relative of ISO 8859-1 (Latin-1) and 0xA0 is a non-breaking space in both. how can i save the content as vanilla ascii text? Well, technically you can't represent a non-breaking space in ASCII because there's no such character defined in ASCII so there's no way to convert text with 0xA0 characters into "vanilla ascii" text. You can convert 0xA0 to 0x20 (ASCII space) by piping the text through: tr '\240' ' ' However, doing the reverse could create a mess for anything that understands the different between a regular space and a non-breaking space. Tools which support POSIX character classes and locales should treat 0xA0 as whitespace (class '[:space:]') if LC_ALL or LC_CTYPE is a Latin-1 locale, e.g. en_US.ISO8859-1. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Available For Hire: https://linkedin.com/in/billcole ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] text/plain?
I don’t think this is an encoding issue. I think it’s an issue with
the original source of the text in the email. (It looks like it was
possibly copied from a terminal interface of some sort… is that
right?)
In the terminal, you can use `tr` to fix this:
```
tr '\xA0' ' ' < mymail.txt > mymail.fixed.txt
```
If that’s not your bag, a sufficiently advanced text editor should be
able to take care of it. In BBEdit (my editor of choice) this will do
suffice:
Find: `\x{A0}`
Replace: ` ` <-a single space character
Grep: on
By the way, the BBEdit command “Zap Gremlins” can take care of all
sorts of weird characters for you. I use it habitually when receiving
“text” from questionable sources. Other text editors may have
similar functionality.
Hope this helps!
-sam
On 6 Apr 2019, at 7:27, Randy Bush wrote:
i receive an email
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0)
Gecko/20100101 PostboxApp/6.1.13
MIME-Version: 1.0
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
the text has funny space characters that i see if i save the text to
disk and look at it with less
0. flo: 2.31 2018.11.03
1. CLIMATE
ACTION
* (N)ew
(M)odify (D)elete..: N
2. * NAME OF CLOUD:
cumulus
i presume the sender is thunderbird and they have created the text
with
some sort of windows encoding on a mac?
how can i save the content as vanilla ascii text?
randy
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
Re: [MlMt] text/plain?
Would you be willing to send me a copy of the .eml file so I can take a look at it? On 6 Apr 2019, at 15:21, Randy Bush wrote: Looks like I confused windows-1252 with windows-1251 in some places in the message below my mind stops at the word 'windows' if it does not have an X in front. :) the question remains, at what point are those non-breaking spaces being translated into ``. if i save the messge body and look at it with less. the problem is that i need to run scripts against the body, and they want real white space. Do they occur in the email you received, or are they introduced by the way you are saving the text from the email? Examining the contents of the raw `.eml` file is the first step to figuring that out. if i drag the message from mailmate to the desktop and look at the .eml wth emacs or less, the windows artifacts are there. randy ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Tab stops and previews of multiple selected messages
thx for the hint. but for my feature request it is only the half way On 7 Apr 2019, at 1:28, David Shepherdson wrote: On 6 Apr 2019, at 07.00, Greg Earle wrote: In my job I get a lot of e-mails that are of a similar nature and of an informational level so I delete a lot of them. When I go to select a lot of messages (to either delete or forward) - or select a few, but they are really long messages (like log file outputs) - MailMate invariably takes a while and beach-balls. I’ve previously seen mention on the mailing list here that you can use the following command to prevent the display of multiple messages when selected: defaults write com.freron.MailMate MmMaximumMessagesDisplayed -integer 1 From a quick try, it seems that with this applied, you just get an empty preview pane if you select more than one message. Given it seems to be an integer setting, you could potentially experiment with larger values than 1 to see if there’s a good compromise between usefulness and performance. David ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Tab stops and previews of multiple selected messages
On 6 Apr 2019, at 07.00, Greg Earle wrote: In my job I get a lot of e-mails that are of a similar nature and of an informational level so I delete a lot of them. When I go to select a lot of messages (to either delete or forward) - or select a few, but they are really long messages (like log file outputs) - MailMate invariably takes a while and beach-balls. I’ve previously seen mention on the mailing list here that you can use the following command to prevent the display of multiple messages when selected: defaults write com.freron.MailMate MmMaximumMessagesDisplayed -integer 1 From a quick try, it seems that with this applied, you just get an empty preview pane if you select more than one message. Given it seems to be an integer setting, you could potentially experiment with larger values than 1 to see if there’s a good compromise between usefulness and performance. David___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] text/plain?
On 6 Apr 2019, at 18:21, Randy Bush wrote: >> Do they occur in the email you received, or are they introduced by the >> way you are saving the text from the email? Examining the contents of >> the raw `.eml` file is the first step to figuring that out. > > if i drag the message from mailmate to the desktop and look at the .eml > wth emacs or less, the windows artifacts are there. > You don't even have to drag it — ⌘-C gives the filename of the .eml. --Steve Bellovin, https://www.cs.columbia.edu/~smb ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] text/plain?
> Looks like I confused windows-1252 with windows-1251 in some places in > the message below my mind stops at the word 'windows' if it does not have an X in front. :) > the question remains, at what point are those non-breaking spaces > being translated into ``. if i save the messge body and look at it with less. the problem is that i need to run scripts against the body, and they want real white space. > Do they occur in the email you received, or are they introduced by the > way you are saving the text from the email? Examining the contents of > the raw `.eml` file is the first step to figuring that out. if i drag the message from mailmate to the desktop and look at the .eml wth emacs or less, the windows artifacts are there. randy ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] text/plain?
Looks like I confused windows-1252 with windows-1251 in some places in the message below, but nbsp is 0xa0 in both encodings, and the question remains, at what point are those non-breaking spaces being translated into ``. Do they occur in the email you received, or are they introduced by the way you are saving the text from the email? Examining the contents of the raw `.eml` file is the first step to figuring that out. Best, Galen On 6 Apr 2019, at 10:51, Galen Menzel wrote: They’re using [Postbox](https://www.postbox-inc.com/). [windows-1252](https://en.wikipedia.org/wiki/Windows-1251) is an 8-bit extended ascii encoding, which Postbox supports. The character with the `A0` character code in the windows-1251 encoding is the [non-breaking space](https://en.wikipedia.org/wiki/Non-breaking_space). So somewhere along the way the non-breaking spaces are being replaced by ``. Does this `` stuff show up when you view the raw message in MailMate (with ⌥⌘U)? What happens if you use `less` to view the message’s `.eml` file? Best, Galen On 6 Apr 2019, at 4:27, Randy Bush wrote: i receive an email User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0) Gecko/20100101 PostboxApp/6.1.13 MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US the text has funny space characters that i see if i save the text to disk and look at it with less 0. flo: 2.31 2018.11.03 1. CLIMATE ACTION * (N)ew (M)odify (D)elete..: N 2. * NAME OF CLOUD: cumulus i presume the sender is thunderbird and they have created the text with some sort of windows encoding on a mac? how can i save the content as vanilla ascii text? randy ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] text/plain?
They’re using [Postbox](https://www.postbox-inc.com/). [windows-1252](https://en.wikipedia.org/wiki/Windows-1251) is an 8-bit extended ascii encoding, which Postbox supports. The character with the `A0` character code in the windows-1251 encoding is the [non-breaking space](https://en.wikipedia.org/wiki/Non-breaking_space). So somewhere along the way the non-breaking spaces are being replaced by ``. Does this `` stuff show up when you view the raw message in MailMate (with ⌥⌘U)? What happens if you use `less` to view the message’s `.eml` file? Best, Galen On 6 Apr 2019, at 4:27, Randy Bush wrote: i receive an email User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0) Gecko/20100101 PostboxApp/6.1.13 MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US the text has funny space characters that i see if i save the text to disk and look at it with less 0. flo: 2.31 2018.11.03 1. CLIMATE ACTION * (N)ew (M)odify (D)elete..: N 2. * NAME OF CLOUD: cumulus i presume the sender is thunderbird and they have created the text with some sort of windows encoding on a mac? how can i save the content as vanilla ascii text? randy ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Tab stops and previews of multiple selected messages
On 5 Apr 2019, at 14:59, Robert Brenstein wrote: Have you tried opening each message in its own window? Shift-cmd-o instead of cmd-o after you select multiple messages. I am opening some times on the order of 30-40 messages this way at once. Robert, I'm not opening multiple messages. I am either selecting them to delete them or I am selecting them to forward them (in one message). - Greg ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
[MlMt] text/plain?
i receive an email User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0) Gecko/20100101 PostboxApp/6.1.13 MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US the text has funny space characters that i see if i save the text to disk and look at it with less 0. flo: 2.31 2018.11.03 1. CLIMATE ACTION * (N)ew (M)odify (D)elete..: N 2. * NAME OF CLOUD: cumulus i presume the sender is thunderbird and they have created the text with some sort of windows encoding on a mac? how can i save the content as vanilla ascii text? randy ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
