Issue ID: 18498
Summary: File buffer overrun when using lexer-as-library
As reported on the internet:
Reminds me of a bug I ran into recently while using the dmd lexer in a project.
When I passed any valid file to the lexer it worked fine.
When I passed a batch of valid files to the lexer, after a few files were
processed, it started to spit out all kinds of errors about invalid Unicode
code-points and ghost tokens. All those locations were past EOF.
When I passed in that particular file individually it worked like a charm.
When I over-sized the buffer that held a file by one char and set that to 0 it
worked for the batch of files.