In the lexers for my C++ parsers I mostly use the make_FOO functions
to build tokens as usual. But in some cases I can't use them. So I
just want to check if what I do is officially supported.
Basically, I'm too lazy to write long sections of predefined words
as found in many lexers, e.g. Bison's scan-gram.l (sure, these are a
bit different with the leading "%", but many languages have
predefined/reserved words which are otherwise normal identifiers):
"%default-prec" return PERCENT_DEFAULT_PREC;
"%define" return PERCENT_DEFINE;
"%defines" return PERCENT_DEFINES;
"%destructor" return PERCENT_DESTRUCTOR;
"%dprec" return PERCENT_DPREC;
"%empty" return PERCENT_EMPTY;
// ...
To make things easier for me (and avoid one source of errors), I use
a small gawk script that generates a lookup-table such as this from
the %token declarations in my grammar:
const unordered_map <string, int> PredefinedStrings
{
{ "if", TParser::token::TOKEN_IF },
{ "else", TParser::token::TOKEN_ELSE },
// ...
};
Then my yylex function does this (slightly simplified from my actual
code; GetToken, TokenText and Loc are functions of my RE-based
lexer):
switch (GetToken ())
{
// ...
case lt_Identifier:
if (auto i = Find (PredefinedStrings, TokenText ()))
return TParser::symbol_type (TParser::token_type (i->second), Loc ());
return TParser::make_identifier (TokenText (), Loc ());
}
The only interesting thing here (as far as Bison is concerned) is
that I determine the token kind dynamically, so I can't use the
make_FOO functions, but call the TParser::symbol_type constructor
manually. (In my experimental changes, I had added a make_token
function for this purpose, but I hadn't mentioned it in my
announcements, so obviously it's not present in Bison currently;
I'm not sure if it was very useful anyway, since it was just a
wrapper around this constructor).
So I need to make sure that this is officially supported, i.e. this
constructor and enum yytokentype will remain in the public interface
in the generated header, or if other alternatives are recommended
instead.
Regards,
Frank
