Thanks for the extensive example, I will have a look in detail. Maybe, though, my problem was not too clear. I have no problem in ignoring the new lines or white spaces. My issue is that when using the `open` method of `BaseLexer` with the default `BufLen` and I try to parse a file where all the code is in a single line longer than the `BufLen` the program crashes.
I can see in your example you are also using `result.lex.open(newStringStream(fileContents))` with default `BufLen`, will it work if your test file has more than 8192 characters in one line?
