On Wed, Feb 29, 2012 at 07:28:48PM +0100, Martin Nowak wrote: > On Wed, 29 Feb 2012 17:41:19 +0100, Timon Gehr <[email protected]> wrote: > > >On 02/28/2012 07:46 PM, Martin Nowak wrote: > >> > >>https://gist.github.com/1255439 - lexer generator > >>https://gist.github.com/1262321 - complete and fast D lexer > >> > > > >Well, it is slower at lexing than DMD at parsing. What is the bottleneck? > > No, it's as fast as dmd's lexer. > > Writing the tokens to stdout takes a lot of time though. > Just disable the "writeln(tok);" in the main loop. > There is also an --echo option which recreates the complete > source from the tokens.
This sounds like something I ran into in my D lexer project. Lexing Phobos took upwards of 10-15 seconds, which is extremely slow considering that dmd can *compile* it in just a few seconds. So I did some profiling, and found out that most of the time was spent formatting tokens into strings and running writeln. After I commented that out, the running time dropped to just a few seconds. :-) T -- The fact that anyone still uses AOL shows that even the presence of options doesn't stop some people from picking the pessimal one. - Mike Ellis
