Re: highlighting "weird" characters...

[Spencer Collyer]

On Mon, 19 Mar 2007 16:38:23 -0400, Mitch Wiedemann wrote:
> Hi all,
> 
> I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007
> 22:17:23) to write mainly XHTML/PHP and I sometimes have to get
> content from word processed documents and paste it into Vim for HTML
> markup.  This usually results in having non-visually detectable
> characters (which I assume are high ASCII) which display strangely on
> the Web.
> 
> Is there a way I can have my Vim highlight these characters so I can
> see them and replace them with their HTML counterparts?
> 
> I've searched Google, the Vim e-mail archive, and I've helped Ugandan
> children :),  but I'm no closer to the answer.
> 
> Any hints?
> 
> Thanks,
> Mitch
> Freeville, NY

Not a VIM-based method, but there is a Perl script called Demoroniser
(http://www.fourmilab.ch/webtools/demoroniser/) which is designed to
handle a lot of these characters and convert them into ASCII characters.

Spencer

-- 
<<< Eagles may soar, but weasels don't get sucked into jet engines >>>
8:47am up 25 days 15:29, 18 users, load average: 0.08, 0.14, 0.18
Registered Linux User #232457 | LFS ID 11703

Re: highlighting "weird" characters...[solved]

[Mitch Wiedemann]

Tobia wrote:
> Mitch Wiedemann wrote:
>   
>> " maps to quickly find Unicode characters within the document
>> map ,uni :match Error /[\x7f-\xff]/
>> map ,uni2 /[^ -~]
>> 
>
> FYI [\x7f-\xff] doesn't find non-ASCII characters that are also outside
> Latin-1, such as em-dash —, typographic quotes “”, ellipsis …, and the
> hundred thousand others defined by Unicode :-)  At least on my system.
>
> [^ -~] on the other hand finds all non-ASCII characters.
>
> You can use either one in both :match Error and /
>   

Ah! Good to know.  The characters you mention are some of the most
frequent offenders.

~/.vimrc updated...

" maps to quickly find unicode characters within the document
map ,uni :match Error /[^ -~]/
map ,uni2 /[^ -~]

Re: highlighting "weird" characters...[solved]

[Tobia]

Mitch Wiedemann wrote:
> " maps to quickly find Unicode characters within the document
> map ,uni :match Error /[\x7f-\xff]/
> map ,uni2 /[^ -~]

FYI [\x7f-\xff] doesn't find non-ASCII characters that are also outside
Latin-1, such as em-dash —, typographic quotes “”, ellipsis …, and the
hundred thousand others defined by Unicode :-)  At least on my system.

[^ -~] on the other hand finds all non-ASCII characters.

You can use either one in both :match Error and /


Tobia

Re: highlighting "weird" characters...[solved]

[Mitch Wiedemann]

Mitch Wiedemann wrote:
> Hi all,
>
> I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
> to write mainly XHTML/PHP and I sometimes have to get content from word
> processed documents and paste it into Vim for HTML markup.  This usually
> results in having non-visually detectable characters (which I assume are
> high ASCII) which display strangely on the Web.
>
> Is there a way I can have my Vim highlight these characters so I can see
> them and replace them with their HTML counterparts?
>
> I've searched Google, the Vim e-mail archive, and I've helped Ugandan
> children :),  but I'm no closer to the answer.
>
> Any hints?
>   

I've added two new maps to my ~/.vimrc

" maps to quickly find Unicode characters within the document
map ,uni :match Error /[\x7f-\xff]/
map ,uni2 /[^ -~]

I've found both methods useful, one just highlights (allowing me to
/find other stuff) and one highlights via /find.

Neat. Thanks again for the help. 

Re: highlighting "weird" characters... Thanks everyone!

[Mitch Wiedemann]

Thanks everyone for your suggestions!  I'm now looking around for a
typical document that I can paste in to use a a guinea pig.

Viva la Vim!

Re: highlighting "weird" characters...

[Tobia]

Mitch Wiedemann wrote:
> non-visually detectable characters (which I assume are high ASCII)

If I may nitpick, "high ASCII" is not the right terminology here.  
The ASCII table only contains 128 characters, with codes 0…127 (where
only codes 32…126 have a visual representation.)

What you call "high ASCII" are Unicode characters that are autside
ASCII, or Latin-1 characters that are autside ASCII.  In both cases they
have codes greater than 127.


> Is there a way I can have my Vim highlight these characters so I can
> see them and replace them with their HTML counterparts?

I usually do a :set hls (highlight search results) followed by /[^ -~]
(search for any character that is not between space and tilde, ie. that
is not a printable character from the ASCII set.)  You can add the tab
character (with Ctrl-V Tab) inside the square brackets if you use it in
your documents.  It will show as /[^ -~^I] with a blue ^I.

This has the added benefit of hitting n to get to the next one.


Tobia

Re: highlighting "weird" characters...

[Yakov Lerner]

On 3/19/07, Mitch Wiedemann <[EMAIL PROTECTED]> wrote:

Hi all,

I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
to write mainly XHTML/PHP and I sometimes have to get content from word
processed documents and paste it into Vim for HTML markup.  This usually
results in having non-visually detectable characters (which I assume are
high ASCII) which display strangely on the Web.

Is there a way I can have my Vim highlight these characters so I can see
them and replace them with their HTML counterparts?


Try this:

:hi clear SpecialKey
:hi link SpecialKey Error

All uprintable chars will be highlighter red
(or whaterver your Error color is defined).

Yakov

Re: highlighting "weird" characters...

[Gary Johnson]

On 2007-03-19, Gary Johnson <[EMAIL PROTECTED]> wrote:
> On 2007-03-19, Mitch Wiedemann <[EMAIL PROTECTED]> wrote:
> > Hi all,
> > 
> > I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
> > to write mainly XHTML/PHP and I sometimes have to get content from word
> > processed documents and paste it into Vim for HTML markup.  This usually
> > results in having non-visually detectable characters (which I assume are
> > high ASCII) which display strangely on the Web.

P.S.

See

:help ga

HTH,
Gary

-- 
Gary Johnson | Agilent Technologies
[EMAIL PROTECTED] | Mobile Broadband Division
 | Spokane, Washington, USA

Re: highlighting "weird" characters...

[Tim Chase]

I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
to write mainly XHTML/PHP and I sometimes have to get content from word
processed documents and paste it into Vim for HTML markup.  This usually
results in having non-visually detectable characters (which I assume are
high ASCII) which display strangely on the Web.

Is there a way I can have my Vim highlight these characters so I can see
them and replace them with their HTML counterparts?

I've searched Google, the Vim e-mail archive, and I've helped Ugandan
children :),  but I'm no closer to the answer.


Well, if you want to highlight them, you can do something like

:match Error /[\x7f-\xff]/

which should catch most of the offenders.  Or you can search for 
them with the same regexp.  And if you have ":set hls", it will 
highlight them too. :)


It does require certain knobs to be set properly in your 
'cpoptions' setting, but if you run vim (rather than "vi"), they 
should automatically be set unless you had reason to bung with them.


Once you're on a particular one, you can use

ga

to determine the ASCII value of the character in question, and 
can use that value for search&replace.


Hope this helps,

-tim

Re: highlighting "weird" characters...

[Gary Johnson]

On 2007-03-19, Mitch Wiedemann <[EMAIL PROTECTED]> wrote:
> Hi all,
> 
> I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
> to write mainly XHTML/PHP and I sometimes have to get content from word
> processed documents and paste it into Vim for HTML markup.  This usually
> results in having non-visually detectable characters (which I assume are
> high ASCII) which display strangely on the Web.
> 
> Is there a way I can have my Vim highlight these characters so I can see
> them and replace them with their HTML counterparts?
> 
> I've searched Google, the Vim e-mail archive, and I've helped Ugandan
> children :),  but I'm no closer to the answer.
> 
> Any hints?

One way to do this would be to

:set isprint=

which will tell vim that only the characters in the range 32 - 126 
are "printable".  Vim will then highlight all the other characters 
as SpecialKey.  You can then search for these "non-printable" 
characters with

/[^[:print:]]

Another, probably better, way would be to simply search for

/[^Vx80-^Vxff]

where ^V means a literal Ctrl-V.  That will search for any character 
in the range 0x80 - 0xff and will highlight them all with the Search 
highlight if 'hlsearch' is set.  I think this way is better because 
it preserves vim's rendering of the non-ASCII characters, which may 
make it easier for you to choose their replacements.

HTH,
Gary

-- 
Gary Johnson | Agilent Technologies
[EMAIL PROTECTED] | Mobile Broadband Division
 | Spokane, Washington, USA

Re: highlighting "weird" characters...

[Cyril Slobin]

On 3/19/07, Mitch Wiedemann <[EMAIL PROTECTED]> wrote:


Is there a way I can have my Vim highlight these characters so I can see
them and replace them with their HTML counterparts?


No idea about highlighting, but exactly to replace strange characters
with HTML or LaTeX or anything else or to do it in backward direction
I've write this plugin:

http://www.vim.org/scripts/script.php?script_id=1761

Documentation inside. I hope this helps.

--
Cyril Slobin <[EMAIL PROTECTED]> `When I use a word,' Humpty Dumpty said,
 `it means just what I choose it to mean'

highlighting "weird" characters...

[Mitch Wiedemann]

Hi all,

I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
to write mainly XHTML/PHP and I sometimes have to get content from word
processed documents and paste it into Vim for HTML markup.  This usually
results in having non-visually detectable characters (which I assume are
high ASCII) which display strangely on the Web.

Is there a way I can have my Vim highlight these characters so I can see
them and replace them with their HTML counterparts?

I've searched Google, the Vim e-mail archive, and I've helped Ugandan
children :),  but I'm no closer to the answer.

Any hints?

Thanks,
Mitch
Freeville, NY