On Friday, 11 May 2012 at 09:36:28 UTC, Jacob Carlborg wrote:
On 2012-05-11 11:22, Roman D. Boiko wrote:
Locations will possible to calculate
both taking into account special token sequences (e.g., #line 3
"ab/c.d"), or discarding them.
Aha, clever. As long as I can get out the information I'm happy
:) How about adding properties for this in the token struct?
There is a method for that in Lexer interface, for me it looks
like it belongth there and not to token. Version accepting token
and producing a pair of start/end Locations will be added.
* Does it convert numerical literals and similar to their
actual values
It is planned to add a post-processor for that as part of
parser,
please see README.md for some more details.
Isn't that a job for the lexer?
That might be done in lexer for efficiency reasons (to avoid
lexing
token value again). But separating this into a dedicated
post-processing
phase leads to a much cleaner design (IMO), also suitable for
uses when
such values are not needed.
That might be the case. But I don't think it belongs in the
parser.
I will provide example code and a dedicated post later to
illustrate my point.
Also I don't think that performance would be
improved given the ratio of number of literals to total number
of tokens
and the need to store additional information per token if it
is done in
lexer. I will elaborate on that later.
Ok, fair enough. Perhaps this could be a property in the Token
struct as well. In that case I would suggest renaming "value"
to lexeme/spelling/representation, or something like that, and
then name the new property "value".
I was going to rename value, but couldn't find a nice term.
Thanks for your suggestions!
As for the property with strongly typed literal value, currently
I plan to put it into AST.
