Bryan C . Warnock <[EMAIL PROTECTED]> writes:
>On Thu, 10 Aug 2000, Dan Sugalski wrote:
>
>> With each box being replaceable, and the process being freezable between
>> boxes. The lexer and parser probably ought to be separated, thinking about
>> it.
>
>Not necessarily. It might be dangerous to change what the lexer think
>is a token. You might be able to get away with having the parser just
>interpret multiple tokens as needed. (Which means the lexer, if it
>doesn't do it already, needs to return some indicator of whitespace.
>It could be that extraneous whitespace is itself a token, with the
>standard resolution being ignoring it.
>
>@foo = @bar >|< @baz;
>
>The lexer would return something like.....
>
>@
>foo
>{space}
>=
>{space}
>@
>bar
>{space}
>>
>|
><
>{space}
>@
>baz
>;
>
>Which the parser could then interpret:
>
>@foo, okay.
>Assignment, got it.
>@bar, okay.
>Greater than, okay.
>Pipe? Hmmm, no space before it, back up one, and make it >|
>Less than. After a >|? No space before it, so let's make it all >|<.
>@baz. Okay, I see I have >|< in my user dictionary, so I'll allow this.
>EOS.
In order to make that work you have required the lexer to pass you the {space}
markers. That makes the parser do a lot more work throwing them away.
It can also easily stop grammar being LALR(1) and hence yacc/bison
parsable - you burn off your one "look ahead" token with the space.
--
Nick Ing-Simmons <[EMAIL PROTECTED]>
Via, but not speaking for: Texas Instruments Ltd.
