Quoting your post in another thread: On Monday, 14 May 2012 at 15:10:25 UTC, Roman D. Boiko wrote:
Making it a class would give several benefits:
* allow not to worry about allocating a big array of tokens. E.g., on 64-bit OS the largest module in Phobos (IIRC, the std.datetime) consumes 13.5MB in an array of almost 500K tokens. It would require 4 times smaller chunk of contiguous memory if it was an array of class objects, because each would consume only 8 bytes instead of 32.
You'll still have count the space the tokens claim on the heap. So it's basically the 500k tokens plus 500k references. I'm not sure, why you would need such a big array of tokens, though.
Aren't they produced by the lexer to be directly consumed and discarded by the parser?
* allow subclassing, for example, for storing strongly typed literal values; this flexibility could also facilitate future extensibility (but it's difficult to predict which kind of extension may be needed)
If performance matters, why would you subclass and risk a virtual method call for something as basic as tokens?
* there would be no need to copy data from tokens into AST, passing an object would be enough (again, copy 8 instead of 32 bytes); the same applies to passing into methods - no need to pass by ref to minimise overhead
I'm using string to store source content in tokens. Because of the way string in D works, there is no need for data copies.
These considerations are mostly about performance. I think there is also some impact on design, but couldn't find anything significant (given that currently I see a token as merely a datastructure without associated behavior).
IMO token are value types.
