On 10/7/13 5:16 PM, Andrei Alexandrescu wrote:
On 10/4/13 5:24 PM, Andrei Alexandrescu wrote:
On 10/2/13 7:41 AM, Dicebot wrote:
After brief discussion with Brian and gathering data from the review
thread, I have decided to start voting for `std.d.lexer` inclusion into
Phobos.
Thanks all involved for the work, first of all Brian.
I have the proverbial good news and bad news. The only bad news is that
I'm voting "no" on this proposal.
But there's plenty of good news.
1. I am not attempting to veto this, so just consider it a normal vote
when tallying.
2. I do vote for inclusion in the /etc/ package for the time being.
3. The work is good and the code valuable, so even in the case my
suggestions (below) will be followed, a virtually all code pulp that
gets work done can be reused.
[snip]
To put my money where my mouth is, I have a proof-of-concept tokenizer
for C++ in working state.
http://dpaste.dzfl.pl/d07dd46d
I made an improvement to the way tokens are handled. In the paste above,
"tk" is a function. A CTFE-able function that just returns a
compile-time constant, but a function nevertheless.
To actually reduce "tk" to a compile-time constant in all cases, I
changed it as follows:
template tk(string symbol) {
import std.range;
static if (symbol == "") {
// Token ID 0 is reserved for "unrecognized token".
enum tk = TokenType2(0);
} else static if (symbol == "\0") {
// Token ID max is reserved for "end of input".
enum tk = TokenType2(
cast(TokenIDRep) (1 + tokens.length + reservedTokens.length));
} else {
//enum id = chain(tokens, reservedTokens).countUntil(symbol);
// Find the id within the regular tokens realm
enum idTokens = tokens.countUntil(symbol);
static if (idTokens >= 0) {
// Found, regular token. Add 1 because 0 is reserved.
enum id = idTokens + 1;
} else {
// not found, only chance is within the reserved tokens realm
enum idResTokens = reservedTokens.countUntil(symbol);
enum id = idResTokens >= 0 ? tokens.length + idResTokens + 1 : -1;
}
static assert(id >= 0 && id < TokenIDRep.max,
"Invalid token: " ~ symbol);
enum tk = TokenType2(id);
}
This is even better now because token types are simple static constants.
Andrei
