Allison Randal wrote:
> More importantly, whitespace skipping isn't a very significant option in
> grammars in general, so creating two keywords that distinguish between
> skipping and no skipping is linguistically infelicitous. It's like
> creating two different words for "shirts with horizontal stripes" and
> "shirts with vertical stripes". Sure, they're different, but the
> difference isn't particularly significant, so it's better expressed by a
> modifier on "shirt" than by a different word.

This is not only "space" skipping; as we discussed, <ws> skips over
comments as well as spaces, because a language (such as Perl 6) can
defined its own <ws> that serves as valid separator. To wit:

    void main () {}
    void/* this also works */main () {}

Or, in Perl 6:

    say time;
    say#( this also works )time;

> From a practical perspective, both the Perl 6 and Punie grammars have
> ended up using 'token' in many places (for things that aren't tokens),
> because :words isn't really the semantics you want for parsing computer
> languages. (Though it is quite useful for parsing natural language and
> other things.) What you want is comment skipping, which isn't the same
> as :words.

Currently it's defined, and used, the same as :words.

I think the confusion arises from <ws> being read as "whitespace"
instead of as "word separator".  Maybe an explicit <wordsep> can fix
that, or maybe rename it to something else, but the token/rule
distinction of :words is very useful, because it's more usual for
languages to behave like C and Perl 6, instead of:

    ex/* this calls exit */it();

which is rarer, and can be treated with separate "token" rules than <ws>.


Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to