Trying to make a lexer, stops if it hits an operator.

[axyz]

Hi, I have a quick question about `BaseLexer` `open` method in `std/lexbase`, 
that given your experience on the topic you may have an answer for.

It seems that the lexer is optimized for line based parsing? What is the best 
idiomatic way to parse a language that can ignore new lines? Currently for 
testing purpose my workaround is to do something like
    
    
    lexbase.open(self, input, 999999)
    
    
    Run

where `999999` is an arbitrarily large number for the `bufLen` in
    
    
    proc open(L: var BaseLexer; input: Stream; bufLen: int = 8192;
              refillChars: set[char] = NewLines) {.....}
    
    
    Run

but this is quite unreliable and probably not very efficient.