On 9/12/07, Thomas Mueller <[EMAIL PROTECTED]> wrote: > >... context-sensitive tokenizing > > I'm not sure what you refer to. Keywords versus identifiers? Example > token types are: 'integer value', 'decimal value', 'text value', > 'operator', 'quoted identifier', 'name'. The keywords are well defined > in Java, but for SQL, I wouldn't decide if it's a keyword or > identifier while tokenizing....
That's what I meant, being too strict in tokenizing can make things harder downstream. I agree about having a soft boundary between tokenizing and the actual parsing. -Bertrand
