I think 0001 with complete keyword lookup replacement is in decent
enough shape to post. Make check-world passes. A few notes and
caveats:
-I added an --extern option to the script for the core keyword
headers. This also capitalizes variables.
-ECPG keyword lookup is a bit different in that the ecpg and sql
lookup functions are wrapped in a single function rather than called
separately within pgc.l. It might be worth untangling that, but I have
not done so.
-Some variable names haven't changed even though now they're only
referring to token values, which might be confusing.
-I haven't checked if I need to install the generated headers.
-I haven't measured performance or binary size. If anyone is excited
enough to do that, great, otherwise I'll do that as time permits.
-There are probably makefile bugs.
Now, on to previous review points:
On 12/27/18, Tom Lane <t...@sss.pgh.pa.us> wrote:
> +/* Payload data for keywords */
> +typedef struct ScanKeywordAux
> +{
> + int16 value; /* grammar's token code */
> + char category; /* see codes above */
> +} ScanKeywordAux;
>
> There isn't really any point in changing category to "char", because
> alignment considerations will mandate that sizeof(ScanKeywordAux) be
> a multiple of 2 anyway. With some compilers we could get around that
> with a pragma to force non-aligned storage, but doing so would be a
> net loss on most non-Intel architectures.
Reverted, especially since we can skip the struct entirely for some
callers as you pointed out below.
> diff --git a/src/pl/plpgsql/src/.gitignore b/src/pl/plpgsql/src/.gitignore
> @@ -1,3 +1,4 @@
> +/*kwlist_d.h
>
> Not a fan of using wildcards in .gitignore files, at least not when
> there's just one or two files you intend to match.
Removed.
> # Force these dependencies to be known even without dependency info built:
> -pl_gram.o pl_handler.o pl_comp.o pl_exec.o pl_funcs.o pl_scanner.o:
> plpgsql.h pl_gram.h plerrcodes.h
> +pl_gram.o pl_handler.o pl_comp.o pl_exec.o pl_funcs.o pl_scanner.o:
> plpgsql.h pl_gram.h plerrcodes.h pl_unreserved_kwlist_d.h
>
> Hm, do we really need any more than pl_scanner.o to depend on that header?
I think you're right, so separated into a new rule.
> +# Parse keyword header for names.
> +my @keywords;
> +while (<$kif>)
> +{
> + if (/^PG_KEYWORD\("(\w+)",\s*\w+,\s*\w+\)/)
>
> This is assuming more than it should about the number of arguments for
> PG_KEYWORD, as well as what's in them. I think it'd be sufficient to
> match like this:
>
> if (/^PG_KEYWORD\("(\w+)",/)
...and...
> diff --git a/src/pl/plpgsql/src/pl_unreserved_kwlist.h
> b/src/pl/plpgsql/src/pl_unreserved_kwlist.h
> +/* name, value, category */
> +PG_KEYWORD("absolute", K_ABSOLUTE, UNRESERVED_KEYWORD)
>
> Likewise, I'd just have these be two-argument macros. There's no reason
> for the various kwlist.h headers to agree on the number of payload
> arguments for PG_KEYWORD.
Both done, however...
> +/* FIXME: Have to redefine this symbol for the WIP. */
> +#undef PG_KEYWORD
> +#define PG_KEYWORD(kwname, value, category) {value, category},
> +
> +static const ScanKeywordAux unreserved_keywords[] = {
> +#include "pl_unreserved_kwlist.h"
> };
>
> The category isn't useful for this keyword list, so couldn't you
> just make this an array of uint16 values?
Yes, this works for the unreserved keywords. The reserved ones still
need the aux struct to work with the core scanner, even though scan.l
doesn't reference category either. This has the consequence that we
can't dispense with category, e.g.:
PG_KEYWORD("all", K_ALL, RESERVED_KEYWORD)
...unless we do without the struct entirely, but that's not without
disadvantages as you mentioned.
I decided to export the struct (rather than just int16 for category)
to the frontend, even though we have to set the token values to zero,
since there might someday be another field of use to the frontend.
Also to avoid confusion.
> I don't mind allowing the prefix to default to empty. What I was
> concerned about was that base_filename could end up undefined.
> Probably the thing to do is to generate base_filename separately,
> say by stripping any initial ".*/" sequence and then substitute
> '_' for '.'.
I removed assumptions about the filename.
> +Options:
> + -o output path
> + -p optional prefix for generated data structures
>
> This usage message is pretty vague about how you write the options
> (cf gripe above).
I tried it like this:
Usage: gen_keywords.pl [--output/-o <path>] [--prefix/-p <prefix>] input_file
--output Output directory
--prefix String prepended to var names in the output file
On 12/27/18, Andrew Dunstan <andrew.duns...@2ndquadrant.com> wrote:
> I would rather we used the standard perl module Getopt::Long, as
> numerous programs we have already do.
Done. I'll also send a patch later to bring some other scripts in line.
-John Naylor
From 87e3e5e045823070a865933e969262d55e156137 Mon Sep 17 00:00:00 2001
From: John Naylor <jcnay...@gmail.com>
Date: Sat, 29 Dec 2018 16:34:14 -0500
Subject: [PATCH v5] Use offset-based keyword lookup.
Replace binary search over an array of ScanKeyword structs with a binary
search over an array of offsets into a keyword string. Access auxillary
data only after a keyword hit. This has better locality of reference and
a smaller memory footprint.
---
.../pg_stat_statements/pg_stat_statements.c | 2 +
src/backend/parser/parser.c | 4 +-
src/backend/parser/scan.l | 38 ++--
src/backend/utils/adt/misc.c | 2 +-
src/backend/utils/adt/ruleutils.c | 9 +-
src/common/.gitignore | 1 +
src/common/Makefile | 25 ++-
src/common/keywords.c | 42 ++--
src/fe_utils/string_utils.c | 9 +-
src/include/common/keywords.h | 19 +-
src/include/parser/kwlist.h | 2 +-
src/include/parser/scanner.h | 8 +-
src/interfaces/ecpg/preproc/.gitignore | 2 +
src/interfaces/ecpg/preproc/Makefile | 20 +-
src/interfaces/ecpg/preproc/c_keywords.c | 65 ++-----
src/interfaces/ecpg/preproc/c_kwlist.h | 53 ++++++
src/interfaces/ecpg/preproc/ecpg_keywords.c | 81 ++------
src/interfaces/ecpg/preproc/ecpg_kwlist.h | 68 +++++++
src/interfaces/ecpg/preproc/keywords.c | 7 +-
src/interfaces/ecpg/preproc/pgc.l | 22 +--
src/interfaces/ecpg/preproc/preproc_extern.h | 8 +-
src/pl/plpgsql/src/.gitignore | 2 +
src/pl/plpgsql/src/Makefile | 16 +-
src/pl/plpgsql/src/pl_reserved_kwlist.h | 55 ++++++
src/pl/plpgsql/src/pl_scanner.c | 179 ++++--------------
src/pl/plpgsql/src/pl_unreserved_kwlist.h | 111 +++++++++++
src/tools/gen_keywords.pl | 135 +++++++++++++
src/tools/msvc/Solution.pm | 44 +++++
src/tools/msvc/clean.bat | 6 +
29 files changed, 704 insertions(+), 331 deletions(-)
create mode 100644 src/common/.gitignore
create mode 100644 src/interfaces/ecpg/preproc/c_kwlist.h
create mode 100644 src/interfaces/ecpg/preproc/ecpg_kwlist.h
create mode 100644 src/pl/plpgsql/src/pl_reserved_kwlist.h
create mode 100644 src/pl/plpgsql/src/pl_unreserved_kwlist.h
create mode 100644 src/tools/gen_keywords.pl
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index 33f9a79f54..6a3b3f5576 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -3076,6 +3076,8 @@ fill_in_constant_lengths(pgssJumbleState *jstate, const char *query,
yyscanner = scanner_init(query,
&yyextra,
ScanKeywords,
+ KeywordString,
+ KeywordOffsets,
NumScanKeywords);
/* we don't want to re-emit any escape string warnings */
diff --git a/src/backend/parser/parser.c b/src/backend/parser/parser.c
index db30483459..0de6f60897 100644
--- a/src/backend/parser/parser.c
+++ b/src/backend/parser/parser.c
@@ -40,8 +40,8 @@ raw_parser(const char *str)
int yyresult;
/* initialize the flex scanner */
- yyscanner = scanner_init(str, &yyextra.core_yy_extra,
- ScanKeywords, NumScanKeywords);
+ yyscanner = scanner_init(str, &yyextra.core_yy_extra, ScanKeywords,
+ KeywordString, KeywordOffsets, NumScanKeywords);
/* base_yylex() only needs this much initialization */
yyextra.have_lookahead = false;
diff --git a/src/backend/parser/scan.l b/src/backend/parser/scan.l
index 74e34df71f..222840e25b 100644
--- a/src/backend/parser/scan.l
+++ b/src/backend/parser/scan.l
@@ -504,18 +504,20 @@ other .
* We will pass this along as a normal character string,
* but preceded with an internally-generated "NCHAR".
*/
- const ScanKeyword *keyword;
+ int kwnum;
SET_YYLLOC();
yyless(1); /* eat only 'n' this time */
- keyword = ScanKeywordLookup("nchar",
- yyextra->keywords,
- yyextra->num_keywords);
- if (keyword != NULL)
+ kwnum = ScanKeywordLookup("nchar",
+ yyextra->kw_string,
+ yyextra->kw_offsets,
+ yyextra->num_keywords);
+ if (kwnum >= 0)
{
- yylval->keyword = keyword->name;
- return keyword->value;
+ yylval->keyword = yyextra->kw_string
+ + yyextra->kw_offsets[kwnum];
+ return yyextra->keywords[kwnum].value;
}
else
{
@@ -1021,19 +1023,21 @@ other .
{identifier} {
- const ScanKeyword *keyword;
+ int kwnum;
char *ident;
SET_YYLLOC();
/* Is it a keyword? */
- keyword = ScanKeywordLookup(yytext,
- yyextra->keywords,
- yyextra->num_keywords);
- if (keyword != NULL)
+ kwnum = ScanKeywordLookup(yytext,
+ yyextra->kw_string,
+ yyextra->kw_offsets,
+ yyextra->num_keywords);
+ if (kwnum >= 0)
{
- yylval->keyword = keyword->name;
- return keyword->value;
+ yylval->keyword = yyextra->kw_string
+ + yyextra->kw_offsets[kwnum];
+ return yyextra->keywords[kwnum].value;
}
/*
@@ -1142,7 +1146,9 @@ scanner_yyerror(const char *message, core_yyscan_t yyscanner)
core_yyscan_t
scanner_init(const char *str,
core_yy_extra_type *yyext,
- const ScanKeyword *keywords,
+ const ScanKeywordAux *keywords,
+ const char *kw_string,
+ const uint16 *kw_offsets,
int num_keywords)
{
Size slen = strlen(str);
@@ -1154,6 +1160,8 @@ scanner_init(const char *str,
core_yyset_extra(yyext, scanner);
yyext->keywords = keywords;
+ yyext->kw_string = kw_string;
+ yyext->kw_offsets = kw_offsets;
yyext->num_keywords = num_keywords;
yyext->backslash_quote = backslash_quote;
diff --git a/src/backend/utils/adt/misc.c b/src/backend/utils/adt/misc.c
index b8f86973dc..d4029ac8ae 100644
--- a/src/backend/utils/adt/misc.c
+++ b/src/backend/utils/adt/misc.c
@@ -423,7 +423,7 @@ pg_get_keywords(PG_FUNCTION_ARGS)
HeapTuple tuple;
/* cast-away-const is ugly but alternatives aren't much better */
- values[0] = unconstify(char *, ScanKeywords[funcctx->call_cntr].name);
+ values[0] = unconstify(char *, KeywordString + KeywordOffsets[funcctx->call_cntr]);
switch (ScanKeywords[funcctx->call_cntr].category)
{
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 4857caecaa..bd4061cbfb 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -10601,11 +10601,12 @@ quote_identifier(const char *ident)
* Note: ScanKeywordLookup() does case-insensitive comparison, but
* that's fine, since we already know we have all-lower-case.
*/
- const ScanKeyword *keyword = ScanKeywordLookup(ident,
- ScanKeywords,
- NumScanKeywords);
+ int kwnum = ScanKeywordLookup(ident,
+ KeywordString,
+ KeywordOffsets,
+ NumScanKeywords);
- if (keyword != NULL && keyword->category != UNRESERVED_KEYWORD)
+ if (kwnum >= 0 && ScanKeywords[kwnum].category != UNRESERVED_KEYWORD)
safe = false;
}
diff --git a/src/common/.gitignore b/src/common/.gitignore
new file mode 100644
index 0000000000..ffa3284fbf
--- /dev/null
+++ b/src/common/.gitignore
@@ -0,0 +1 @@
+/kwlist_d.h
diff --git a/src/common/Makefile b/src/common/Makefile
index ec8139f014..fbd4933554 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -63,7 +63,18 @@ OBJS_FRONTEND = $(OBJS_COMMON) fe_memutils.o file_utils.o restricted_token.o
OBJS_SHLIB = $(OBJS_FRONTEND:%.o=%_shlib.o)
OBJS_SRV = $(OBJS_COMMON:%.o=%_srv.o)
-all: libpgcommon.a libpgcommon_shlib.a libpgcommon_srv.a
+all: libpgcommon.a libpgcommon_shlib.a libpgcommon_srv.a kwlist_d.h
+
+distprep: kwlist_d.h
+
+# generate keyword header for the core and frontend scanners
+kwlist_d.h: $(top_srcdir)/src/include/parser/kwlist.h $(top_srcdir)/src/tools/gen_keywords.pl
+ $(PERL) $(top_srcdir)/src/tools/gen_keywords.pl --extern $<
+
+$(top_builddir)/src/include/common/kwlist_d.h: kwlist_d.h
+ prereqdir=`cd '$(dir $<)' >/dev/null && pwd` && \
+ cd '$(dir $@)' && rm -f $(notdir $@) && \
+ $(LN_S) "$$prereqdir/$(notdir $<)" .
# libpgcommon is needed by some contrib
install: all installdirs
@@ -121,10 +132,14 @@ libpgcommon_srv.a: $(OBJS_SRV)
$(top_builddir)/src/include/parser/gram.h: $(top_srcdir)/src/backend/parser/gram.y
$(MAKE) -C $(top_builddir)/src/backend $(top_builddir)/src/include/parser/gram.h
-keywords.o: $(top_srcdir)/src/include/parser/kwlist.h
-keywords_shlib.o: $(top_srcdir)/src/include/parser/kwlist.h
-keywords_srv.o: $(top_builddir)/src/include/parser/gram.h $(top_srcdir)/src/include/parser/kwlist.h
+keywords.o: $(top_builddir)/src/include/common/kwlist_d.h $(top_srcdir)/src/include/parser/kwlist.h
+keywords_shlib.o: $(top_builddir)/src/include/common/kwlist_d.h $(top_srcdir)/src/include/parser/kwlist.h
+keywords_srv.o: $(top_builddir)/src/include/common/kwlist_d.h $(top_builddir)/src/include/parser/gram.h $(top_srcdir)/src/include/parser/kwlist.h
-clean distclean maintainer-clean:
+# kwlist_d.h is in the distribution tarball, so it is not cleaned here.
+clean distclean:
rm -f libpgcommon.a libpgcommon_shlib.a libpgcommon_srv.a
rm -f $(OBJS_FRONTEND) $(OBJS_SHLIB) $(OBJS_SRV)
+
+maintainer-clean: distclean
+ rm -f kwlist_d.h
diff --git a/src/common/keywords.c b/src/common/keywords.c
index 0c0c794c68..ef712d6633 100644
--- a/src/common/keywords.c
+++ b/src/common/keywords.c
@@ -23,7 +23,10 @@
#include "parser/gramparse.h"
-#define PG_KEYWORD(a,b,c) {a,b,c},
+/* String lookup table for keywords */
+#include "common/kwlist_d.h"
+
+#define PG_KEYWORD(kwname, value, category) {value, category},
#else
@@ -33,12 +36,11 @@
* We don't need the token number for frontend uses, so leave it out to avoid
* requiring backend headers that won't compile cleanly here.
*/
-#define PG_KEYWORD(a,b,c) {a,0,c},
+#define PG_KEYWORD(kwname, value, category) {0, category},
#endif /* FRONTEND */
-
-const ScanKeyword ScanKeywords[] = {
+const ScanKeywordAux ScanKeywords[] = {
#include "parser/kwlist.h"
};
@@ -48,10 +50,13 @@ const int NumScanKeywords = lengthof(ScanKeywords);
/*
* ScanKeywordLookup - see if a given word is a keyword
*
- * The table to be searched is passed explicitly, so that this can be used
- * to search keyword lists other than the standard list appearing above.
+ * The keyword string along with an array of offsets into it are passed
+ * explicitly, so that callers can use different keyword lists. These are
+ * generated from the appropriate "kwlist" header at compile time.
*
- * Returns a pointer to the ScanKeyword table entry, or NULL if no match.
+ * Returns an array index, or -1 if no match. The caller can use this array
+ * index to retrieve the keyword string as well as any needed auxiliary data
+ * such as token value or category.
*
* The match is done case-insensitively. Note that we deliberately use a
* dumbed-down case conversion that will only translate 'A'-'Z' into 'a'-'z',
@@ -60,21 +65,22 @@ const int NumScanKeywords = lengthof(ScanKeywords);
* keywords are to be matched in this way even though non-keyword identifiers
* receive a different case-normalization mapping.
*/
-const ScanKeyword *
+int
ScanKeywordLookup(const char *text,
- const ScanKeyword *keywords,
+ const char *kw_string,
+ const uint16 *kw_offsets,
int num_keywords)
{
int len,
i;
char word[NAMEDATALEN];
- const ScanKeyword *low;
- const ScanKeyword *high;
+ const uint16 *low;
+ const uint16 *high;
len = strlen(text);
/* We assume all keywords are shorter than NAMEDATALEN. */
if (len >= NAMEDATALEN)
- return NULL;
+ return -1;
/*
* Apply an ASCII-only downcasing. We must not use tolower() since it may
@@ -93,22 +99,22 @@ ScanKeywordLookup(const char *text,
/*
* Now do a binary search using plain strcmp() comparison.
*/
- low = keywords;
- high = keywords + (num_keywords - 1);
+ low = kw_offsets;
+ high = kw_offsets + (num_keywords - 1);
while (low <= high)
{
- const ScanKeyword *middle;
+ const uint16 *middle;
int difference;
middle = low + (high - low) / 2;
- difference = strcmp(middle->name, word);
+ difference = strcmp(kw_string + *middle, word);
if (difference == 0)
- return middle;
+ return middle - kw_offsets;
else if (difference < 0)
low = middle + 1;
else
high = middle - 1;
}
- return NULL;
+ return -1;
}
diff --git a/src/fe_utils/string_utils.c b/src/fe_utils/string_utils.c
index af0d9d5173..0a138c138d 100644
--- a/src/fe_utils/string_utils.c
+++ b/src/fe_utils/string_utils.c
@@ -104,11 +104,12 @@ fmtId(const char *rawid)
* Note: ScanKeywordLookup() does case-insensitive comparison, but
* that's fine, since we already know we have all-lower-case.
*/
- const ScanKeyword *keyword = ScanKeywordLookup(rawid,
- ScanKeywords,
- NumScanKeywords);
+ int kwnum = ScanKeywordLookup(rawid,
+ KeywordString,
+ KeywordOffsets,
+ NumScanKeywords);
- if (keyword != NULL && keyword->category != UNRESERVED_KEYWORD)
+ if (kwnum >= 0 && ScanKeywords[kwnum].category != UNRESERVED_KEYWORD)
need_quotes = true;
}
diff --git a/src/include/common/keywords.h b/src/include/common/keywords.h
index 0b31505b66..a06899c222 100644
--- a/src/include/common/keywords.h
+++ b/src/include/common/keywords.h
@@ -21,24 +21,29 @@
#define RESERVED_KEYWORD 3
-typedef struct ScanKeyword
+/* Auxiliary data for keywords */
+typedef struct ScanKeywordAux
{
- const char *name; /* in lower case */
int16 value; /* grammar's token code */
int16 category; /* see codes above */
-} ScanKeyword;
+} ScanKeywordAux;
#ifndef FRONTEND
-extern PGDLLIMPORT const ScanKeyword ScanKeywords[];
+extern PGDLLIMPORT const ScanKeywordAux ScanKeywords[];
+extern PGDLLIMPORT const char *KeywordString;
+extern PGDLLIMPORT const uint16 KeywordOffsets[];
extern PGDLLIMPORT const int NumScanKeywords;
#else
-extern const ScanKeyword ScanKeywords[];
+extern const ScanKeywordAux ScanKeywords[];
+extern const char *KeywordString;
+extern const uint16 KeywordOffsets[];
extern const int NumScanKeywords;
#endif
-extern const ScanKeyword *ScanKeywordLookup(const char *text,
- const ScanKeyword *keywords,
+extern int ScanKeywordLookup(const char *text,
+ const char *kw_string,
+ const uint16 *kw_offsets,
int num_keywords);
#endif /* KEYWORDS_H */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 23db40147b..5bea89b7e1 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -2,7 +2,7 @@
*
* kwlist.h
*
- * The keyword list is kept in its own source file for possible use by
+ * The keyword lists are kept in their own source files for use by
* automatic tools. The exact representation of a keyword is determined
* by the PG_KEYWORD macro, which is not defined in this file; it can
* be defined by the caller for special purposes.
diff --git a/src/include/parser/scanner.h b/src/include/parser/scanner.h
index 20bf1e6672..d04419c55b 100644
--- a/src/include/parser/scanner.h
+++ b/src/include/parser/scanner.h
@@ -75,7 +75,9 @@ typedef struct core_yy_extra_type
/*
* The keyword list to use.
*/
- const ScanKeyword *keywords;
+ const ScanKeywordAux *keywords;
+ const char *kw_string;
+ const uint16 *kw_offsets;
int num_keywords;
/*
@@ -119,7 +121,9 @@ typedef void *core_yyscan_t;
/* Entry points in parser/scan.l */
extern core_yyscan_t scanner_init(const char *str,
core_yy_extra_type *yyext,
- const ScanKeyword *keywords,
+ const ScanKeywordAux *keywords,
+ const char *kw_string,
+ const uint16 *kw_offsets,
int num_keywords);
extern void scanner_finish(core_yyscan_t yyscanner);
extern int core_yylex(core_YYSTYPE *lvalp, YYLTYPE *llocp,
diff --git a/src/interfaces/ecpg/preproc/.gitignore b/src/interfaces/ecpg/preproc/.gitignore
index 38ae2fe4d9..685be1c4c7 100644
--- a/src/interfaces/ecpg/preproc/.gitignore
+++ b/src/interfaces/ecpg/preproc/.gitignore
@@ -1,3 +1,5 @@
+/c_kwlist_d.h
+/ecpg_kwlist_d.h
/preproc.y
/preproc.c
/preproc.h
diff --git a/src/interfaces/ecpg/preproc/Makefile b/src/interfaces/ecpg/preproc/Makefile
index 8ceadd112b..97c4ddfd93 100644
--- a/src/interfaces/ecpg/preproc/Makefile
+++ b/src/interfaces/ecpg/preproc/Makefile
@@ -53,9 +53,20 @@ preproc.y: ../../../backend/parser/gram.y parse.pl ecpg.addons ecpg.header ecpg.
$(PERL) $(srcdir)/parse.pl $(srcdir) < $< > $@
$(PERL) $(srcdir)/check_rules.pl $(srcdir) $<
+# generate keyword headers
+c_kwlist_d.h: c_kwlist.h $(top_srcdir)/src/tools/gen_keywords.pl
+ $(PERL) $(top_srcdir)/src/tools/gen_keywords.pl --prefix c_ $<
+
+ecpg_kwlist_d.h: ecpg_kwlist.h $(top_srcdir)/src/tools/gen_keywords.pl
+ $(PERL) $(top_srcdir)/src/tools/gen_keywords.pl --prefix ecpg_ $<
+
+# Force these dependencies to be known even without dependency info built:
ecpg_keywords.o c_keywords.o keywords.o preproc.o pgc.o parser.o: preproc.h
+ecpg_keywords.o: ecpg_kwlist_d.h
+c_keywords.o: c_kwlist_d.h
+keywords.o: $(top_builddir)/src/include/common/kwlist_d.h
-distprep: preproc.y preproc.c preproc.h pgc.c
+distprep: preproc.y preproc.c preproc.h pgc.c c_kwlist_d.h ecpg_kwlist_d.h
install: all installdirs
$(INSTALL_PROGRAM) ecpg$(X) '$(DESTDIR)$(bindir)'
@@ -66,12 +77,11 @@ installdirs:
uninstall:
rm -f '$(DESTDIR)$(bindir)/ecpg$(X)'
+# preproc.y, preproc.c, preproc.h, pgc.c, c_kwlist_d.h, and ecpg_kwlist_d.h
+# are in the distribution tarball, so they are not cleaned here.
clean distclean:
rm -f *.o ecpg$(X)
rm -f typename.c
-# `make distclean' must not remove preproc.y, preproc.c, preproc.h, or pgc.c
-# since we want to ship those files in the distribution for people with
-# inadequate tools. Instead, `make maintainer-clean' will remove them.
maintainer-clean: distclean
- rm -f preproc.y preproc.c preproc.h pgc.c
+ rm -f preproc.y preproc.c preproc.h pgc.c c_kwlist_d.h ecpg_kwlist_d.h
diff --git a/src/interfaces/ecpg/preproc/c_keywords.c b/src/interfaces/ecpg/preproc/c_keywords.c
index c367dbfc20..989db3a5a5 100644
--- a/src/interfaces/ecpg/preproc/c_keywords.c
+++ b/src/interfaces/ecpg/preproc/c_keywords.c
@@ -14,72 +14,45 @@
#include "preproc_extern.h"
#include "preproc.h"
-/*
- * List of (keyword-name, keyword-token-value) pairs.
- *
- * !!WARNING!!: This list must be sorted, because binary
- * search is used to locate entries.
- */
-static const ScanKeyword ScanCKeywords[] = {
- /* name, value, category */
+/* String lookup table for C keywords */
+#include "c_kwlist_d.h"
- /*
- * category is not needed in ecpg, it is only here so we can share the
- * data structure with the backend
- */
- {"VARCHAR", VARCHAR, 0},
- {"auto", S_AUTO, 0},
- {"bool", SQL_BOOL, 0},
- {"char", CHAR_P, 0},
- {"const", S_CONST, 0},
- {"enum", ENUM_P, 0},
- {"extern", S_EXTERN, 0},
- {"float", FLOAT_P, 0},
- {"hour", HOUR_P, 0},
- {"int", INT_P, 0},
- {"long", SQL_LONG, 0},
- {"minute", MINUTE_P, 0},
- {"month", MONTH_P, 0},
- {"register", S_REGISTER, 0},
- {"second", SECOND_P, 0},
- {"short", SQL_SHORT, 0},
- {"signed", SQL_SIGNED, 0},
- {"static", S_STATIC, 0},
- {"struct", SQL_STRUCT, 0},
- {"to", TO, 0},
- {"typedef", S_TYPEDEF, 0},
- {"union", UNION, 0},
- {"unsigned", SQL_UNSIGNED, 0},
- {"varchar", VARCHAR, 0},
- {"volatile", S_VOLATILE, 0},
- {"year", YEAR_P, 0},
-};
+#define PG_KEYWORD(kwname, value) value,
+static const int16 ScanCKeywords[] = {
+#include "c_kwlist.h"
+};
/*
+ * ScanCKeywordLookup - see if a given word is a keyword
+ *
+ * Returns the token value of the keyword or -1 if no match.
+ *
* Do a binary search using plain strcmp() comparison. This is much like
* ScanKeywordLookup(), except we want case-sensitive matching.
*/
-const ScanKeyword *
+int16
ScanCKeywordLookup(const char *text)
{
- const ScanKeyword *low = &ScanCKeywords[0];
- const ScanKeyword *high = &ScanCKeywords[lengthof(ScanCKeywords) - 1];
+ const uint16 *low = &c_kw_offsets[0];
+ const uint16 *high = &c_kw_offsets[lengthof(c_kw_offsets) - 1];
while (low <= high)
{
- const ScanKeyword *middle;
+ const uint16 *middle;
int difference;
middle = low + (high - low) / 2;
- difference = strcmp(middle->name, text);
+ difference = strcmp(c_kw_string + *middle, text);
if (difference == 0)
- return middle;
+ {
+ return ScanCKeywords[middle - c_kw_offsets];
+ }
else if (difference < 0)
low = middle + 1;
else
high = middle - 1;
}
- return NULL;
+ return -1;
}
diff --git a/src/interfaces/ecpg/preproc/c_kwlist.h b/src/interfaces/ecpg/preproc/c_kwlist.h
new file mode 100644
index 0000000000..137cac515c
--- /dev/null
+++ b/src/interfaces/ecpg/preproc/c_kwlist.h
@@ -0,0 +1,53 @@
+/*-------------------------------------------------------------------------
+ *
+ * c_kwlist.h
+ *
+ * The keyword lists are kept in their own source files for use by
+ * automatic tools. The exact representation of a keyword is determined
+ * by the PG_KEYWORD macro, which is not defined in this file; it can
+ * be defined by the caller for special purposes.
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/interfaces/ecpg/preproc/c_kwlist.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/* There is deliberately not an #ifndef C_KWLIST_H here. */
+
+/*
+ * List of (keyword-name, keyword-token-value) pairs.
+ *
+ * !!WARNING!!: This list must be sorted, because binary
+ * search is used to locate entries.
+ */
+
+/* name, value */
+PG_KEYWORD("VARCHAR", VARCHAR)
+PG_KEYWORD("auto", S_AUTO)
+PG_KEYWORD("bool", SQL_BOOL)
+PG_KEYWORD("char", CHAR_P)
+PG_KEYWORD("const", S_CONST)
+PG_KEYWORD("enum", ENUM_P)
+PG_KEYWORD("extern", S_EXTERN)
+PG_KEYWORD("float", FLOAT_P)
+PG_KEYWORD("hour", HOUR_P)
+PG_KEYWORD("int", INT_P)
+PG_KEYWORD("long", SQL_LONG)
+PG_KEYWORD("minute", MINUTE_P)
+PG_KEYWORD("month", MONTH_P)
+PG_KEYWORD("register", S_REGISTER)
+PG_KEYWORD("second", SECOND_P)
+PG_KEYWORD("short", SQL_SHORT)
+PG_KEYWORD("signed", SQL_SIGNED)
+PG_KEYWORD("static", S_STATIC)
+PG_KEYWORD("struct", SQL_STRUCT)
+PG_KEYWORD("to", TO)
+PG_KEYWORD("typedef", S_TYPEDEF)
+PG_KEYWORD("union", UNION)
+PG_KEYWORD("unsigned", SQL_UNSIGNED)
+PG_KEYWORD("varchar", VARCHAR)
+PG_KEYWORD("volatile", S_VOLATILE)
+PG_KEYWORD("year", YEAR_P)
diff --git a/src/interfaces/ecpg/preproc/ecpg_keywords.c b/src/interfaces/ecpg/preproc/ecpg_keywords.c
index 37c97e162d..3255060976 100644
--- a/src/interfaces/ecpg/preproc/ecpg_keywords.c
+++ b/src/interfaces/ecpg/preproc/ecpg_keywords.c
@@ -16,82 +16,37 @@
#include "preproc_extern.h"
#include "preproc.h"
-/*
- * List of (keyword-name, keyword-token-value) pairs.
- *
- * !!WARNING!!: This list must be sorted, because binary
- * search is used to locate entries.
- */
-static const ScanKeyword ECPGScanKeywords[] = {
- /* name, value, category */
+/* String lookup table for ECPG keywords */
+#include "ecpg_kwlist_d.h"
+
+#define PG_KEYWORD(kwname, value) value,
- /*
- * category is not needed in ecpg, it is only here so we can share the
- * data structure with the backend
- */
- {"allocate", SQL_ALLOCATE, 0},
- {"autocommit", SQL_AUTOCOMMIT, 0},
- {"bool", SQL_BOOL, 0},
- {"break", SQL_BREAK, 0},
- {"cardinality", SQL_CARDINALITY, 0},
- {"connect", SQL_CONNECT, 0},
- {"count", SQL_COUNT, 0},
- {"datetime_interval_code", SQL_DATETIME_INTERVAL_CODE, 0},
- {"datetime_interval_precision", SQL_DATETIME_INTERVAL_PRECISION, 0},
- {"describe", SQL_DESCRIBE, 0},
- {"descriptor", SQL_DESCRIPTOR, 0},
- {"disconnect", SQL_DISCONNECT, 0},
- {"found", SQL_FOUND, 0},
- {"free", SQL_FREE, 0},
- {"get", SQL_GET, 0},
- {"go", SQL_GO, 0},
- {"goto", SQL_GOTO, 0},
- {"identified", SQL_IDENTIFIED, 0},
- {"indicator", SQL_INDICATOR, 0},
- {"key_member", SQL_KEY_MEMBER, 0},
- {"length", SQL_LENGTH, 0},
- {"long", SQL_LONG, 0},
- {"nullable", SQL_NULLABLE, 0},
- {"octet_length", SQL_OCTET_LENGTH, 0},
- {"open", SQL_OPEN, 0},
- {"output", SQL_OUTPUT, 0},
- {"reference", SQL_REFERENCE, 0},
- {"returned_length", SQL_RETURNED_LENGTH, 0},
- {"returned_octet_length", SQL_RETURNED_OCTET_LENGTH, 0},
- {"scale", SQL_SCALE, 0},
- {"section", SQL_SECTION, 0},
- {"short", SQL_SHORT, 0},
- {"signed", SQL_SIGNED, 0},
- {"sqlerror", SQL_SQLERROR, 0},
- {"sqlprint", SQL_SQLPRINT, 0},
- {"sqlwarning", SQL_SQLWARNING, 0},
- {"stop", SQL_STOP, 0},
- {"struct", SQL_STRUCT, 0},
- {"unsigned", SQL_UNSIGNED, 0},
- {"var", SQL_VAR, 0},
- {"whenever", SQL_WHENEVER, 0},
+static const int16 ECPGScanKeywords[] = {
+#include "ecpg_kwlist.h"
};
/*
* ScanECPGKeywordLookup - see if a given word is a keyword
*
- * Returns a pointer to the ScanKeyword table entry, or NULL if no match.
+ * Returns the token value of the keyword or -1 if no match.
* Keywords are matched using the same case-folding rules as in the backend.
*/
-const ScanKeyword *
+int16
ScanECPGKeywordLookup(const char *text)
{
- const ScanKeyword *res;
+ int kwnum;
/* First check SQL symbols defined by the backend. */
- res = ScanKeywordLookup(text, SQLScanKeywords, NumSQLScanKeywords);
- if (res)
- return res;
+ kwnum = ScanKeywordLookup(text, KeywordString, KeywordOffsets,
+ NumSQLScanKeywords);
+ if (kwnum >= 0)
+ return SQLScanKeywords[kwnum];
/* Try ECPG-specific keywords. */
- res = ScanKeywordLookup(text, ECPGScanKeywords, lengthof(ECPGScanKeywords));
- if (res)
- return res;
+ kwnum = ScanKeywordLookup(text, ecpg_kw_string, ecpg_kw_offsets,
+ lengthof(ECPGScanKeywords));
+ if (kwnum >= 0)
+ return ECPGScanKeywords[kwnum];
- return NULL;
+ return -1;
}
diff --git a/src/interfaces/ecpg/preproc/ecpg_kwlist.h b/src/interfaces/ecpg/preproc/ecpg_kwlist.h
new file mode 100644
index 0000000000..07df5be7b9
--- /dev/null
+++ b/src/interfaces/ecpg/preproc/ecpg_kwlist.h
@@ -0,0 +1,68 @@
+/*-------------------------------------------------------------------------
+ *
+ * ecpg_kwlist.h
+ *
+ * The keyword lists are kept in their own source files for use by
+ * automatic tools. The exact representation of a keyword is determined
+ * by the PG_KEYWORD macro, which is not defined in this file; it can
+ * be defined by the caller for special purposes.
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/interfaces/ecpg/preproc/ecpg_kwlist.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/* There is deliberately not an #ifndef ECPG_KWLIST_H here. */
+
+/*
+ * List of (keyword-name, keyword-token-value) pairs.
+ *
+ * !!WARNING!!: This list must be sorted, because binary
+ * search is used to locate entries.
+ */
+
+/* name, value */
+PG_KEYWORD("allocate", SQL_ALLOCATE)
+PG_KEYWORD("autocommit", SQL_AUTOCOMMIT)
+PG_KEYWORD("bool", SQL_BOOL)
+PG_KEYWORD("break", SQL_BREAK)
+PG_KEYWORD("cardinality", SQL_CARDINALITY)
+PG_KEYWORD("connect", SQL_CONNECT)
+PG_KEYWORD("count", SQL_COUNT)
+PG_KEYWORD("datetime_interval_code", SQL_DATETIME_INTERVAL_CODE)
+PG_KEYWORD("datetime_interval_precision", SQL_DATETIME_INTERVAL_PRECISION)
+PG_KEYWORD("describe", SQL_DESCRIBE)
+PG_KEYWORD("descriptor", SQL_DESCRIPTOR)
+PG_KEYWORD("disconnect", SQL_DISCONNECT)
+PG_KEYWORD("found", SQL_FOUND)
+PG_KEYWORD("free", SQL_FREE)
+PG_KEYWORD("get", SQL_GET)
+PG_KEYWORD("go", SQL_GO)
+PG_KEYWORD("goto", SQL_GOTO)
+PG_KEYWORD("identified", SQL_IDENTIFIED)
+PG_KEYWORD("indicator", SQL_INDICATOR)
+PG_KEYWORD("key_member", SQL_KEY_MEMBER)
+PG_KEYWORD("length", SQL_LENGTH)
+PG_KEYWORD("long", SQL_LONG)
+PG_KEYWORD("nullable", SQL_NULLABLE)
+PG_KEYWORD("octet_length", SQL_OCTET_LENGTH)
+PG_KEYWORD("open", SQL_OPEN)
+PG_KEYWORD("output", SQL_OUTPUT)
+PG_KEYWORD("reference", SQL_REFERENCE)
+PG_KEYWORD("returned_length", SQL_RETURNED_LENGTH)
+PG_KEYWORD("returned_octet_length", SQL_RETURNED_OCTET_LENGTH)
+PG_KEYWORD("scale", SQL_SCALE)
+PG_KEYWORD("section", SQL_SECTION)
+PG_KEYWORD("short", SQL_SHORT)
+PG_KEYWORD("signed", SQL_SIGNED)
+PG_KEYWORD("sqlerror", SQL_SQLERROR)
+PG_KEYWORD("sqlprint", SQL_SQLPRINT)
+PG_KEYWORD("sqlwarning", SQL_SQLWARNING)
+PG_KEYWORD("stop", SQL_STOP)
+PG_KEYWORD("struct", SQL_STRUCT)
+PG_KEYWORD("unsigned", SQL_UNSIGNED)
+PG_KEYWORD("var", SQL_VAR)
+PG_KEYWORD("whenever", SQL_WHENEVER)
diff --git a/src/interfaces/ecpg/preproc/keywords.c b/src/interfaces/ecpg/preproc/keywords.c
index c0ed492d41..bf655d6068 100644
--- a/src/interfaces/ecpg/preproc/keywords.c
+++ b/src/interfaces/ecpg/preproc/keywords.c
@@ -13,6 +13,7 @@
*
*-------------------------------------------------------------------------
*/
+
#include "postgres_fe.h"
/*
@@ -30,10 +31,12 @@
#include "preproc_extern.h"
#include "preproc.h"
+/* String lookup table for SQL keywords */
+#include "common/kwlist_d.h"
-#define PG_KEYWORD(a,b,c) {a,b,c},
+#define PG_KEYWORD(kwname, value, category) value,
-const ScanKeyword SQLScanKeywords[] = {
+const int16 SQLScanKeywords[] = {
#include "parser/kwlist.h"
};
diff --git a/src/interfaces/ecpg/preproc/pgc.l b/src/interfaces/ecpg/preproc/pgc.l
index 468ccbe2b4..3883e9e5d0 100644
--- a/src/interfaces/ecpg/preproc/pgc.l
+++ b/src/interfaces/ecpg/preproc/pgc.l
@@ -920,19 +920,19 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
{identifier} {
- const ScanKeyword *keyword;
+ int16 kwvalue;
if (!isdefine())
{
/* Is it an SQL/ECPG keyword? */
- keyword = ScanECPGKeywordLookup(yytext);
- if (keyword != NULL)
- return keyword->value;
+ kwvalue = ScanECPGKeywordLookup(yytext);
+ if (kwvalue >= 0)
+ return kwvalue;
/* Is it a C keyword? */
- keyword = ScanCKeywordLookup(yytext);
- if (keyword != NULL)
- return keyword->value;
+ kwvalue = ScanCKeywordLookup(yytext);
+ if (kwvalue >= 0)
+ return kwvalue;
/*
* None of the above. Return it as an identifier.
@@ -1010,7 +1010,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return CPP_LINE;
}
<C>{identifier} {
- const ScanKeyword *keyword;
+ int16 kwvalue;
/*
* Try to detect a function name:
@@ -1026,9 +1026,9 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
/* however, some defines have to be taken care of for compatibility */
if ((!INFORMIX_MODE || !isinformixdefine()) && !isdefine())
{
- keyword = ScanCKeywordLookup(yytext);
- if (keyword != NULL)
- return keyword->value;
+ kwvalue = ScanCKeywordLookup(yytext);
+ if (kwvalue >= 0)
+ return kwvalue;
else
{
base_yylval.str = mm_strdup(yytext);
diff --git a/src/interfaces/ecpg/preproc/preproc_extern.h b/src/interfaces/ecpg/preproc/preproc_extern.h
index 13eda670ff..0d7c63cc6c 100644
--- a/src/interfaces/ecpg/preproc/preproc_extern.h
+++ b/src/interfaces/ecpg/preproc/preproc_extern.h
@@ -59,7 +59,9 @@ extern struct when when_error,
extern struct ECPGstruct_member *struct_member_list[STRUCT_DEPTH];
/* Globals from keywords.c */
-extern const ScanKeyword SQLScanKeywords[];
+extern const int16 SQLScanKeywords[];
+extern const char *KeywordString;
+extern const uint16 KeywordOffsets[];
extern const int NumSQLScanKeywords;
/* functions */
@@ -102,8 +104,8 @@ extern void check_indicator(struct ECPGtype *);
extern void remove_typedefs(int);
extern void remove_variables(int);
extern struct variable *new_variable(const char *, struct ECPGtype *, int);
-extern const ScanKeyword *ScanCKeywordLookup(const char *);
-extern const ScanKeyword *ScanECPGKeywordLookup(const char *text);
+extern int16 ScanCKeywordLookup(const char *text);
+extern int16 ScanECPGKeywordLookup(const char *text);
extern void parser_init(void);
extern int filtered_base_yylex(void);
diff --git a/src/pl/plpgsql/src/.gitignore b/src/pl/plpgsql/src/.gitignore
index ff6ac965fd..3ab9a2243c 100644
--- a/src/pl/plpgsql/src/.gitignore
+++ b/src/pl/plpgsql/src/.gitignore
@@ -1,5 +1,7 @@
/pl_gram.c
/pl_gram.h
+/pl_reserved_kwlist_d.h
+/pl_unreserved_kwlist_d.h
/plerrcodes.h
/log/
/results/
diff --git a/src/pl/plpgsql/src/Makefile b/src/pl/plpgsql/src/Makefile
index 25a5a9d448..bdc28c5f01 100644
--- a/src/pl/plpgsql/src/Makefile
+++ b/src/pl/plpgsql/src/Makefile
@@ -61,6 +61,7 @@ uninstall-headers:
# Force these dependencies to be known even without dependency info built:
pl_gram.o pl_handler.o pl_comp.o pl_exec.o pl_funcs.o pl_scanner.o: plpgsql.h pl_gram.h plerrcodes.h
+pl_scanner.o: pl_reserved_kwlist_d.h pl_unreserved_kwlist_d.h
# See notes in src/backend/parser/Makefile about the following two rules
pl_gram.h: pl_gram.c
@@ -72,6 +73,12 @@ pl_gram.c: BISONFLAGS += -d
plerrcodes.h: $(top_srcdir)/src/backend/utils/errcodes.txt generate-plerrcodes.pl
$(PERL) $(srcdir)/generate-plerrcodes.pl $< > $@
+# generate keyword headers for the scanner
+pl_reserved_kwlist_d.h: pl_reserved_kwlist.h $(top_srcdir)/src/tools/gen_keywords.pl
+ $(PERL) $(top_srcdir)/src/tools/gen_keywords.pl --prefix pl_reserved_ $<
+
+pl_unreserved_kwlist_d.h: pl_unreserved_kwlist.h $(top_srcdir)/src/tools/gen_keywords.pl
+ $(PERL) $(top_srcdir)/src/tools/gen_keywords.pl --prefix pl_unreserved_ $<
check: submake
$(pg_regress_check) $(REGRESS_OPTS) $(REGRESS)
@@ -84,13 +91,14 @@ submake:
$(MAKE) -C $(top_builddir)/src/test/regress pg_regress$(X)
-distprep: pl_gram.h pl_gram.c plerrcodes.h
+distprep: pl_gram.h pl_gram.c plerrcodes.h pl_reserved_kwlist_d.h pl_unreserved_kwlist_d.h
-# pl_gram.c, pl_gram.h and plerrcodes.h are in the distribution tarball,
-# so they are not cleaned here.
+# pl_gram.c, pl_gram.h, plerrcodes.h, pl_reserved_kwlist_d.h, and
+# pl_unreserved_kwlist_d.h are in the distribution tarball, so they
+# are not cleaned here.
clean distclean: clean-lib
rm -f $(OBJS)
rm -rf $(pg_regress_clean_files)
maintainer-clean: distclean
- rm -f pl_gram.c pl_gram.h plerrcodes.h
+ rm -f pl_gram.c pl_gram.h plerrcodes.h pl_reserved_kwlist_d.h pl_unreserved_kwlist_d.h
diff --git a/src/pl/plpgsql/src/pl_reserved_kwlist.h b/src/pl/plpgsql/src/pl_reserved_kwlist.h
new file mode 100644
index 0000000000..87c154266b
--- /dev/null
+++ b/src/pl/plpgsql/src/pl_reserved_kwlist.h
@@ -0,0 +1,55 @@
+/*-------------------------------------------------------------------------
+ *
+ * pl_reserved_kwlist.h
+ *
+ * The keyword lists are kept in their own source files for use by
+ * automatic tools. The exact representation of a keyword is determined
+ * by the PG_KEYWORD macro, which is not defined in this file; it can
+ * be defined by the caller for special purposes.
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/pl/plpgsql/src/pl_reserved_kwlist.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/* There is deliberately not an #ifndef PL_RESERVED_KWLIST_H here. */
+
+/*
+ * List of (keyword-name, keyword-token-value, category) pairs.
+ * Category is only here to share the same auxiliary data structure
+ * with the backend.
+ *
+ * Be careful not to put the same word in both lists.
+ *
+ * !!WARNING!!: This list must be sorted by ASCII name, because binary
+ * search is used to locate entries.
+ */
+
+/* name, value, category */
+PG_KEYWORD("all", K_ALL, RESERVED_KEYWORD)
+PG_KEYWORD("begin", K_BEGIN, RESERVED_KEYWORD)
+PG_KEYWORD("by", K_BY, RESERVED_KEYWORD)
+PG_KEYWORD("case", K_CASE, RESERVED_KEYWORD)
+PG_KEYWORD("declare", K_DECLARE, RESERVED_KEYWORD)
+PG_KEYWORD("else", K_ELSE, RESERVED_KEYWORD)
+PG_KEYWORD("end", K_END, RESERVED_KEYWORD)
+PG_KEYWORD("execute", K_EXECUTE, RESERVED_KEYWORD)
+PG_KEYWORD("for", K_FOR, RESERVED_KEYWORD)
+PG_KEYWORD("foreach", K_FOREACH, RESERVED_KEYWORD)
+PG_KEYWORD("from", K_FROM, RESERVED_KEYWORD)
+PG_KEYWORD("if", K_IF, RESERVED_KEYWORD)
+PG_KEYWORD("in", K_IN, RESERVED_KEYWORD)
+PG_KEYWORD("into", K_INTO, RESERVED_KEYWORD)
+PG_KEYWORD("loop", K_LOOP, RESERVED_KEYWORD)
+PG_KEYWORD("not", K_NOT, RESERVED_KEYWORD)
+PG_KEYWORD("null", K_NULL, RESERVED_KEYWORD)
+PG_KEYWORD("or", K_OR, RESERVED_KEYWORD)
+PG_KEYWORD("strict", K_STRICT, RESERVED_KEYWORD)
+PG_KEYWORD("then", K_THEN, RESERVED_KEYWORD)
+PG_KEYWORD("to", K_TO, RESERVED_KEYWORD)
+PG_KEYWORD("using", K_USING, RESERVED_KEYWORD)
+PG_KEYWORD("when", K_WHEN, RESERVED_KEYWORD)
+PG_KEYWORD("while", K_WHILE, RESERVED_KEYWORD)
diff --git a/src/pl/plpgsql/src/pl_scanner.c b/src/pl/plpgsql/src/pl_scanner.c
index ab18946847..b54ed696a7 100644
--- a/src/pl/plpgsql/src/pl_scanner.c
+++ b/src/pl/plpgsql/src/pl_scanner.c
@@ -21,8 +21,9 @@
#include "plpgsql.h"
#include "pl_gram.h" /* must be after parser/scanner.h */
-
-#define PG_KEYWORD(a,b,c) {a,b,c},
+/* String lookup tables for keywords */
+#include "pl_reserved_kwlist_d.h"
+#include "pl_unreserved_kwlist_d.h"
/* Klugy flag to tell scanner how to look up identifiers */
@@ -31,7 +32,9 @@ IdentifierLookup plpgsql_IdentifierLookup = IDENTIFIER_LOOKUP_NORMAL;
/*
* A word about keywords:
*
- * We keep reserved and unreserved keywords in separate arrays. The
+ * We keep reserved and unreserved keywords in separate headers. Be careful
+ * not to put the same word in both headers. Also be sure that pl_gram.y's
+ * unreserved_keyword production agrees with the unreserved header. The
* reserved keywords are passed to the core scanner, so they will be
* recognized before (and instead of) any variable name. Unreserved words
* are checked for separately, usually after determining that the identifier
@@ -57,127 +60,19 @@ IdentifierLookup plpgsql_IdentifierLookup = IDENTIFIER_LOOKUP_NORMAL;
* BEGIN BY DECLARE EXECUTE FOREACH IF LOOP STRICT WHILE
*/
-/*
- * Lists of keyword (name, token-value, category) entries.
- *
- * !!WARNING!!: These lists must be sorted by ASCII name, because binary
- * search is used to locate entries.
- *
- * Be careful not to put the same word in both lists. Also be sure that
- * pl_gram.y's unreserved_keyword production agrees with the second list.
- */
+#define PG_KEYWORD(kwname, value, category) {value, category},
-static const ScanKeyword reserved_keywords[] = {
- PG_KEYWORD("all", K_ALL, RESERVED_KEYWORD)
- PG_KEYWORD("begin", K_BEGIN, RESERVED_KEYWORD)
- PG_KEYWORD("by", K_BY, RESERVED_KEYWORD)
- PG_KEYWORD("case", K_CASE, RESERVED_KEYWORD)
- PG_KEYWORD("declare", K_DECLARE, RESERVED_KEYWORD)
- PG_KEYWORD("else", K_ELSE, RESERVED_KEYWORD)
- PG_KEYWORD("end", K_END, RESERVED_KEYWORD)
- PG_KEYWORD("execute", K_EXECUTE, RESERVED_KEYWORD)
- PG_KEYWORD("for", K_FOR, RESERVED_KEYWORD)
- PG_KEYWORD("foreach", K_FOREACH, RESERVED_KEYWORD)
- PG_KEYWORD("from", K_FROM, RESERVED_KEYWORD)
- PG_KEYWORD("if", K_IF, RESERVED_KEYWORD)
- PG_KEYWORD("in", K_IN, RESERVED_KEYWORD)
- PG_KEYWORD("into", K_INTO, RESERVED_KEYWORD)
- PG_KEYWORD("loop", K_LOOP, RESERVED_KEYWORD)
- PG_KEYWORD("not", K_NOT, RESERVED_KEYWORD)
- PG_KEYWORD("null", K_NULL, RESERVED_KEYWORD)
- PG_KEYWORD("or", K_OR, RESERVED_KEYWORD)
- PG_KEYWORD("strict", K_STRICT, RESERVED_KEYWORD)
- PG_KEYWORD("then", K_THEN, RESERVED_KEYWORD)
- PG_KEYWORD("to", K_TO, RESERVED_KEYWORD)
- PG_KEYWORD("using", K_USING, RESERVED_KEYWORD)
- PG_KEYWORD("when", K_WHEN, RESERVED_KEYWORD)
- PG_KEYWORD("while", K_WHILE, RESERVED_KEYWORD)
+static const ScanKeywordAux reserved_keywords[] = {
+#include "pl_reserved_kwlist.h"
};
static const int num_reserved_keywords = lengthof(reserved_keywords);
-static const ScanKeyword unreserved_keywords[] = {
- PG_KEYWORD("absolute", K_ABSOLUTE, UNRESERVED_KEYWORD)
- PG_KEYWORD("alias", K_ALIAS, UNRESERVED_KEYWORD)
- PG_KEYWORD("array", K_ARRAY, UNRESERVED_KEYWORD)
- PG_KEYWORD("assert", K_ASSERT, UNRESERVED_KEYWORD)
- PG_KEYWORD("backward", K_BACKWARD, UNRESERVED_KEYWORD)
- PG_KEYWORD("call", K_CALL, UNRESERVED_KEYWORD)
- PG_KEYWORD("close", K_CLOSE, UNRESERVED_KEYWORD)
- PG_KEYWORD("collate", K_COLLATE, UNRESERVED_KEYWORD)
- PG_KEYWORD("column", K_COLUMN, UNRESERVED_KEYWORD)
- PG_KEYWORD("column_name", K_COLUMN_NAME, UNRESERVED_KEYWORD)
- PG_KEYWORD("commit", K_COMMIT, UNRESERVED_KEYWORD)
- PG_KEYWORD("constant", K_CONSTANT, UNRESERVED_KEYWORD)
- PG_KEYWORD("constraint", K_CONSTRAINT, UNRESERVED_KEYWORD)
- PG_KEYWORD("constraint_name", K_CONSTRAINT_NAME, UNRESERVED_KEYWORD)
- PG_KEYWORD("continue", K_CONTINUE, UNRESERVED_KEYWORD)
- PG_KEYWORD("current", K_CURRENT, UNRESERVED_KEYWORD)
- PG_KEYWORD("cursor", K_CURSOR, UNRESERVED_KEYWORD)
- PG_KEYWORD("datatype", K_DATATYPE, UNRESERVED_KEYWORD)
- PG_KEYWORD("debug", K_DEBUG, UNRESERVED_KEYWORD)
- PG_KEYWORD("default", K_DEFAULT, UNRESERVED_KEYWORD)
- PG_KEYWORD("detail", K_DETAIL, UNRESERVED_KEYWORD)
- PG_KEYWORD("diagnostics", K_DIAGNOSTICS, UNRESERVED_KEYWORD)
- PG_KEYWORD("do", K_DO, UNRESERVED_KEYWORD)
- PG_KEYWORD("dump", K_DUMP, UNRESERVED_KEYWORD)
- PG_KEYWORD("elseif", K_ELSIF, UNRESERVED_KEYWORD)
- PG_KEYWORD("elsif", K_ELSIF, UNRESERVED_KEYWORD)
- PG_KEYWORD("errcode", K_ERRCODE, UNRESERVED_KEYWORD)
- PG_KEYWORD("error", K_ERROR, UNRESERVED_KEYWORD)
- PG_KEYWORD("exception", K_EXCEPTION, UNRESERVED_KEYWORD)
- PG_KEYWORD("exit", K_EXIT, UNRESERVED_KEYWORD)
- PG_KEYWORD("fetch", K_FETCH, UNRESERVED_KEYWORD)
- PG_KEYWORD("first", K_FIRST, UNRESERVED_KEYWORD)
- PG_KEYWORD("forward", K_FORWARD, UNRESERVED_KEYWORD)
- PG_KEYWORD("get", K_GET, UNRESERVED_KEYWORD)
- PG_KEYWORD("hint", K_HINT, UNRESERVED_KEYWORD)
- PG_KEYWORD("import", K_IMPORT, UNRESERVED_KEYWORD)
- PG_KEYWORD("info", K_INFO, UNRESERVED_KEYWORD)
- PG_KEYWORD("insert", K_INSERT, UNRESERVED_KEYWORD)
- PG_KEYWORD("is", K_IS, UNRESERVED_KEYWORD)
- PG_KEYWORD("last", K_LAST, UNRESERVED_KEYWORD)
- PG_KEYWORD("log", K_LOG, UNRESERVED_KEYWORD)
- PG_KEYWORD("message", K_MESSAGE, UNRESERVED_KEYWORD)
- PG_KEYWORD("message_text", K_MESSAGE_TEXT, UNRESERVED_KEYWORD)
- PG_KEYWORD("move", K_MOVE, UNRESERVED_KEYWORD)
- PG_KEYWORD("next", K_NEXT, UNRESERVED_KEYWORD)
- PG_KEYWORD("no", K_NO, UNRESERVED_KEYWORD)
- PG_KEYWORD("notice", K_NOTICE, UNRESERVED_KEYWORD)
- PG_KEYWORD("open", K_OPEN, UNRESERVED_KEYWORD)
- PG_KEYWORD("option", K_OPTION, UNRESERVED_KEYWORD)
- PG_KEYWORD("perform", K_PERFORM, UNRESERVED_KEYWORD)
- PG_KEYWORD("pg_context", K_PG_CONTEXT, UNRESERVED_KEYWORD)
- PG_KEYWORD("pg_datatype_name", K_PG_DATATYPE_NAME, UNRESERVED_KEYWORD)
- PG_KEYWORD("pg_exception_context", K_PG_EXCEPTION_CONTEXT, UNRESERVED_KEYWORD)
- PG_KEYWORD("pg_exception_detail", K_PG_EXCEPTION_DETAIL, UNRESERVED_KEYWORD)
- PG_KEYWORD("pg_exception_hint", K_PG_EXCEPTION_HINT, UNRESERVED_KEYWORD)
- PG_KEYWORD("print_strict_params", K_PRINT_STRICT_PARAMS, UNRESERVED_KEYWORD)
- PG_KEYWORD("prior", K_PRIOR, UNRESERVED_KEYWORD)
- PG_KEYWORD("query", K_QUERY, UNRESERVED_KEYWORD)
- PG_KEYWORD("raise", K_RAISE, UNRESERVED_KEYWORD)
- PG_KEYWORD("relative", K_RELATIVE, UNRESERVED_KEYWORD)
- PG_KEYWORD("reset", K_RESET, UNRESERVED_KEYWORD)
- PG_KEYWORD("return", K_RETURN, UNRESERVED_KEYWORD)
- PG_KEYWORD("returned_sqlstate", K_RETURNED_SQLSTATE, UNRESERVED_KEYWORD)
- PG_KEYWORD("reverse", K_REVERSE, UNRESERVED_KEYWORD)
- PG_KEYWORD("rollback", K_ROLLBACK, UNRESERVED_KEYWORD)
- PG_KEYWORD("row_count", K_ROW_COUNT, UNRESERVED_KEYWORD)
- PG_KEYWORD("rowtype", K_ROWTYPE, UNRESERVED_KEYWORD)
- PG_KEYWORD("schema", K_SCHEMA, UNRESERVED_KEYWORD)
- PG_KEYWORD("schema_name", K_SCHEMA_NAME, UNRESERVED_KEYWORD)
- PG_KEYWORD("scroll", K_SCROLL, UNRESERVED_KEYWORD)
- PG_KEYWORD("set", K_SET, UNRESERVED_KEYWORD)
- PG_KEYWORD("slice", K_SLICE, UNRESERVED_KEYWORD)
- PG_KEYWORD("sqlstate", K_SQLSTATE, UNRESERVED_KEYWORD)
- PG_KEYWORD("stacked", K_STACKED, UNRESERVED_KEYWORD)
- PG_KEYWORD("table", K_TABLE, UNRESERVED_KEYWORD)
- PG_KEYWORD("table_name", K_TABLE_NAME, UNRESERVED_KEYWORD)
- PG_KEYWORD("type", K_TYPE, UNRESERVED_KEYWORD)
- PG_KEYWORD("use_column", K_USE_COLUMN, UNRESERVED_KEYWORD)
- PG_KEYWORD("use_variable", K_USE_VARIABLE, UNRESERVED_KEYWORD)
- PG_KEYWORD("variable_conflict", K_VARIABLE_CONFLICT, UNRESERVED_KEYWORD)
- PG_KEYWORD("warning", K_WARNING, UNRESERVED_KEYWORD)
+#undef PG_KEYWORD
+#define PG_KEYWORD(kwname, value) value,
+
+static const int16 unreserved_keywords[] = {
+#include "pl_unreserved_kwlist.h"
};
static const int num_unreserved_keywords = lengthof(unreserved_keywords);
@@ -256,7 +151,7 @@ plpgsql_yylex(void)
{
int tok1;
TokenAuxData aux1;
- const ScanKeyword *kw;
+ int kwnum;
tok1 = internal_yylex(&aux1);
if (tok1 == IDENT || tok1 == PARAM)
@@ -332,12 +227,14 @@ plpgsql_yylex(void)
&aux1.lval.word))
tok1 = T_DATUM;
else if (!aux1.lval.word.quoted &&
- (kw = ScanKeywordLookup(aux1.lval.word.ident,
- unreserved_keywords,
- num_unreserved_keywords)))
+ (kwnum = ScanKeywordLookup(aux1.lval.word.ident,
+ pl_unreserved_kw_string,
+ pl_unreserved_kw_offsets,
+ num_unreserved_keywords)) >= 0)
{
- aux1.lval.keyword = kw->name;
- tok1 = kw->value;
+ aux1.lval.keyword = pl_unreserved_kw_string
+ + pl_unreserved_kw_offsets[kwnum];
+ tok1 = unreserved_keywords[kwnum];
}
else
tok1 = T_WORD;
@@ -362,12 +259,14 @@ plpgsql_yylex(void)
{
/* try for unreserved keyword, then for variable name */
if (core_yy.scanbuf[aux1.lloc] != '"' &&
- (kw = ScanKeywordLookup(aux1.lval.str,
- unreserved_keywords,
- num_unreserved_keywords)))
+ (kwnum = ScanKeywordLookup(aux1.lval.str,
+ pl_unreserved_kw_string,
+ pl_unreserved_kw_offsets,
+ num_unreserved_keywords)) >= 0)
{
- aux1.lval.keyword = kw->name;
- tok1 = kw->value;
+ aux1.lval.keyword = pl_unreserved_kw_string
+ + pl_unreserved_kw_offsets[kwnum];
+ tok1 = unreserved_keywords[kwnum];
}
else if (plpgsql_parse_word(aux1.lval.str,
core_yy.scanbuf + aux1.lloc,
@@ -386,12 +285,14 @@ plpgsql_yylex(void)
&aux1.lval.word))
tok1 = T_DATUM;
else if (!aux1.lval.word.quoted &&
- (kw = ScanKeywordLookup(aux1.lval.word.ident,
- unreserved_keywords,
- num_unreserved_keywords)))
+ (kwnum = ScanKeywordLookup(aux1.lval.word.ident,
+ pl_unreserved_kw_string,
+ pl_unreserved_kw_offsets,
+ num_unreserved_keywords)) >= 0)
{
- aux1.lval.keyword = kw->name;
- tok1 = kw->value;
+ aux1.lval.keyword = pl_unreserved_kw_string
+ + pl_unreserved_kw_offsets[kwnum];
+ tok1 = unreserved_keywords[kwnum];
}
else
tok1 = T_WORD;
@@ -511,7 +412,7 @@ plpgsql_token_is_unreserved_keyword(int token)
for (i = 0; i < num_unreserved_keywords; i++)
{
- if (unreserved_keywords[i].value == token)
+ if (unreserved_keywords[i] == token)
return true;
}
return false;
@@ -707,8 +608,10 @@ void
plpgsql_scanner_init(const char *str)
{
/* Start up the core scanner */
- yyscanner = scanner_init(str, &core_yy,
- reserved_keywords, num_reserved_keywords);
+ yyscanner = scanner_init(str, &core_yy, reserved_keywords,
+ pl_reserved_kw_string,
+ pl_reserved_kw_offsets,
+ num_reserved_keywords);
/*
* scanorig points to the original string, which unlike the scanner's
diff --git a/src/pl/plpgsql/src/pl_unreserved_kwlist.h b/src/pl/plpgsql/src/pl_unreserved_kwlist.h
new file mode 100644
index 0000000000..494f476576
--- /dev/null
+++ b/src/pl/plpgsql/src/pl_unreserved_kwlist.h
@@ -0,0 +1,111 @@
+/*-------------------------------------------------------------------------
+ *
+ * pl_unreserved_kwlist.h
+ *
+ * The keyword lists are kept in their own source files for use by
+ * automatic tools. The exact representation of a keyword is determined
+ * by the PG_KEYWORD macro, which is not defined in this file; it can
+ * be defined by the caller for special purposes.
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/pl/plpgsql/src/pl_unreserved_kwlist.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/* There is deliberately not an #ifndef PL_UNRESERVED_KWLIST_H here. */
+
+/*
+ * List of (keyword-name, keyword-token-value) pairs.
+ *
+ * Be careful not to put the same word in both lists. Also be sure that
+ * pl_gram.y's unreserved_keyword production agrees with this list.
+ *
+ * !!WARNING!!: This list must be sorted by ASCII name, because binary
+ * search is used to locate entries.
+ */
+
+/* name, value */
+PG_KEYWORD("absolute", K_ABSOLUTE)
+PG_KEYWORD("alias", K_ALIAS)
+PG_KEYWORD("array", K_ARRAY)
+PG_KEYWORD("assert", K_ASSERT)
+PG_KEYWORD("backward", K_BACKWARD)
+PG_KEYWORD("call", K_CALL)
+PG_KEYWORD("close", K_CLOSE)
+PG_KEYWORD("collate", K_COLLATE)
+PG_KEYWORD("column", K_COLUMN)
+PG_KEYWORD("column_name", K_COLUMN_NAME)
+PG_KEYWORD("commit", K_COMMIT)
+PG_KEYWORD("constant", K_CONSTANT)
+PG_KEYWORD("constraint", K_CONSTRAINT)
+PG_KEYWORD("constraint_name", K_CONSTRAINT_NAME)
+PG_KEYWORD("continue", K_CONTINUE)
+PG_KEYWORD("current", K_CURRENT)
+PG_KEYWORD("cursor", K_CURSOR)
+PG_KEYWORD("datatype", K_DATATYPE)
+PG_KEYWORD("debug", K_DEBUG)
+PG_KEYWORD("default", K_DEFAULT)
+PG_KEYWORD("detail", K_DETAIL)
+PG_KEYWORD("diagnostics", K_DIAGNOSTICS)
+PG_KEYWORD("do", K_DO)
+PG_KEYWORD("dump", K_DUMP)
+PG_KEYWORD("elseif", K_ELSIF)
+PG_KEYWORD("elsif", K_ELSIF)
+PG_KEYWORD("errcode", K_ERRCODE)
+PG_KEYWORD("error", K_ERROR)
+PG_KEYWORD("exception", K_EXCEPTION)
+PG_KEYWORD("exit", K_EXIT)
+PG_KEYWORD("fetch", K_FETCH)
+PG_KEYWORD("first", K_FIRST)
+PG_KEYWORD("forward", K_FORWARD)
+PG_KEYWORD("get", K_GET)
+PG_KEYWORD("hint", K_HINT)
+PG_KEYWORD("import", K_IMPORT)
+PG_KEYWORD("info", K_INFO)
+PG_KEYWORD("insert", K_INSERT)
+PG_KEYWORD("is", K_IS)
+PG_KEYWORD("last", K_LAST)
+PG_KEYWORD("log", K_LOG)
+PG_KEYWORD("message", K_MESSAGE)
+PG_KEYWORD("message_text", K_MESSAGE_TEXT)
+PG_KEYWORD("move", K_MOVE)
+PG_KEYWORD("next", K_NEXT)
+PG_KEYWORD("no", K_NO)
+PG_KEYWORD("notice", K_NOTICE)
+PG_KEYWORD("open", K_OPEN)
+PG_KEYWORD("option", K_OPTION)
+PG_KEYWORD("perform", K_PERFORM)
+PG_KEYWORD("pg_context", K_PG_CONTEXT)
+PG_KEYWORD("pg_datatype_name", K_PG_DATATYPE_NAME)
+PG_KEYWORD("pg_exception_context", K_PG_EXCEPTION_CONTEXT)
+PG_KEYWORD("pg_exception_detail", K_PG_EXCEPTION_DETAIL)
+PG_KEYWORD("pg_exception_hint", K_PG_EXCEPTION_HINT)
+PG_KEYWORD("print_strict_params", K_PRINT_STRICT_PARAMS)
+PG_KEYWORD("prior", K_PRIOR)
+PG_KEYWORD("query", K_QUERY)
+PG_KEYWORD("raise", K_RAISE)
+PG_KEYWORD("relative", K_RELATIVE)
+PG_KEYWORD("reset", K_RESET)
+PG_KEYWORD("return", K_RETURN)
+PG_KEYWORD("returned_sqlstate", K_RETURNED_SQLSTATE)
+PG_KEYWORD("reverse", K_REVERSE)
+PG_KEYWORD("rollback", K_ROLLBACK)
+PG_KEYWORD("row_count", K_ROW_COUNT)
+PG_KEYWORD("rowtype", K_ROWTYPE)
+PG_KEYWORD("schema", K_SCHEMA)
+PG_KEYWORD("schema_name", K_SCHEMA_NAME)
+PG_KEYWORD("scroll", K_SCROLL)
+PG_KEYWORD("set", K_SET)
+PG_KEYWORD("slice", K_SLICE)
+PG_KEYWORD("sqlstate", K_SQLSTATE)
+PG_KEYWORD("stacked", K_STACKED)
+PG_KEYWORD("table", K_TABLE)
+PG_KEYWORD("table_name", K_TABLE_NAME)
+PG_KEYWORD("type", K_TYPE)
+PG_KEYWORD("use_column", K_USE_COLUMN)
+PG_KEYWORD("use_variable", K_USE_VARIABLE)
+PG_KEYWORD("variable_conflict", K_VARIABLE_CONFLICT)
+PG_KEYWORD("warning", K_WARNING)
diff --git a/src/tools/gen_keywords.pl b/src/tools/gen_keywords.pl
new file mode 100644
index 0000000000..45b1bcfa55
--- /dev/null
+++ b/src/tools/gen_keywords.pl
@@ -0,0 +1,135 @@
+#----------------------------------------------------------------------
+#
+# gen_keywords.pl
+# Perl script that transforms a list of keywords into a single string
+# and an array of offsets into it. These are emitted into a header so
+# they can be passed to ScanKeywordLookup().
+#
+# Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/tools/gen_keywords.pl
+#
+#----------------------------------------------------------------------
+
+use strict;
+use warnings;
+use Getopt::Long;
+
+my $output_path = '';
+my $extern = 0;
+my $prefix = '';
+
+GetOptions(
+ 'output:s' => \$output_path,
+ 'extern' => \$extern,
+ 'prefix:s' => \$prefix) || usage();
+
+my $kw_input_file = shift @ARGV || die "No input file.\n";
+
+# Make sure output_path ends in a slash.
+if ($output_path ne '' && substr($output_path, -1) ne '/')
+{
+ $output_path .= '/';
+}
+
+$kw_input_file =~ /(\w+)\.h$/;
+my $base_filename = $1 . '_d';
+my $kw_def_file = $output_path . $base_filename . '.h';
+
+open(my $kif, '<', $kw_input_file) || die "$kw_input_file: $!";
+open(my $kwdef, '>', $kw_def_file) || die "$kw_def_file: $!";
+
+# Opening boilerplate for keyword definition header.
+printf $kwdef <<EOM, $base_filename, uc $base_filename, uc $base_filename;
+/*-------------------------------------------------------------------------
+ *
+ * %s.h
+ * List of keywords represented as a keyword string and offsets into it.
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * NOTES
+ * ******************************
+ * *** DO NOT EDIT THIS FILE! ***
+ * ******************************
+ *
+ * It has been GENERATED by src/tools/gen_keywords.pl
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef %s_H
+#define %s_H
+
+EOM
+
+# Parse keyword header for names.
+my @keywords;
+while (<$kif>)
+{
+ if (/^PG_KEYWORD\("(\w+)",/)
+ {
+ push @keywords, $1;
+ }
+}
+
+# Error out if the keyword names are not in ASCII order.
+for my $i (0..$#keywords - 1)
+{
+ die qq|The keyword "$keywords[$i + 1]" is out of order in $kw_input_file|
+ if ($keywords[$i] cmp $keywords[$i + 1]) >= 0;
+}
+
+# Emit the keyword string.
+
+if ($extern)
+{
+ printf $kwdef qq|const char *%sKeywordString =\n\t"|, $prefix;
+}
+else
+{
+ printf $kwdef qq|static const char *%skw_string =\n\t"|, $prefix;
+}
+
+print $kwdef join qq|\\0"\n\t"|, @keywords;
+print $kwdef qq|";\n\n|;
+printf $kwdef "#endif\t\t\t\t\t\t\t/* %s_H */\n\n", uc $base_filename;
+
+# Emit an array of numerical offsets which will be used to index into the
+# keyword string.
+if ($extern)
+{
+ printf $kwdef "const uint16 %sKeywordOffsets[] = {\n\t", $prefix;
+}
+else
+{
+ printf $kwdef "static const uint16 %skw_offsets[] = {\n\t", $prefix;
+}
+
+my $offset = 0;
+foreach my $name (@keywords)
+{
+ print $kwdef "$offset,\n\t";
+
+ # Calculate the cumulative offset of the next keyword,
+ # taking into account the null terminator.
+ $offset += length($name) + 1;
+}
+
+print $kwdef "};\n";
+
+
+sub usage
+{
+ die <<EOM;
+Usage: gen_keywords.pl [--output/-o <path>] [--prefix/-p <prefix>] input_file
+ --output Output directory
+ --prefix String prepended to var names in the output file
+
+gen_keywords.pl transforms a list of keywords into a single string and
+an array of offsets into it.
+
+EOM
+}
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 0b7cdf8dd5..a2284dad8c 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -268,6 +268,22 @@ sub GenerateFiles
"src/interfaces/ecpg/pgtypeslib/exports.txt",
"LIBPGTYPES");
+ if (IsNewer(
+ 'src/common/kwlist_d.h',
+ 'src/include/parser/kwlist.h'))
+ {
+ print "Generating kwlist_d.h...\n";
+ system('perl src/tools/gen_keywords.pl --extern -o src/common src/include/parser/kwlist.h');
+ }
+
+ if (IsNewer(
+ 'src/include/common/kwlist_d.h',
+ 'src/common/kwlist_d.h'))
+ {
+ copyFile('src/common/kwlist_d.h',
+ 'src/include/common/kwlist_d.h');
+ }
+
chdir('src/backend/utils');
my $pg_language_dat = '../../../src/include/catalog/pg_language.dat';
my $pg_proc_dat = '../../../src/include/catalog/pg_proc.dat';
@@ -411,6 +427,34 @@ sub GenerateFiles
chdir('../../..');
}
+ if (IsNewer(
+ 'src/pl/plpgsql/src/pl_reserved_kwlist_d.h',
+ 'src/pl/plpgsql/src/pl_reserved_kwlist.h')
+ || IsNewer(
+ 'src/pl/plpgsql/src/pl_unreserved_kwlist_d.h',
+ 'src/pl/plpgsql/src/pl_unreserved_kwlist.h'))
+ {
+ print "Generating pl_reserved_kwlist_d.h and pl_unreserved_kwlist_d.h...\n";
+ chdir('src/pl/plpgsql/src');
+ system('perl ../../../tools/gen_keywords.pl --prefix pl_reserved_ pl_reserved_kwlist.h');
+ system('perl ../../../tools/gen_keywords.pl --prefix pl_unreserved_ pl_unreserved_kwlist.h');
+ chdir('../../../..');
+ }
+
+ if (IsNewer(
+ 'src/interfaces/ecpg/preproc/c_kwlist_d.h',
+ 'src/interfaces/ecpg/preproc/c_kwlist.h')
+ || IsNewer(
+ 'src/interfaces/ecpg/preproc/ecpg_kwlist_d.h',
+ 'src/interfaces/ecpg/preproc/ecpg_kwlist.h')
+ {
+ print "Generating c_kwlist_d.h and ecpg_kwlist_d.h...\n";
+ chdir('src/interfaces/ecpg/preproc');
+ system('perl ../../../tools/gen_keywords.pl --prefix c_ c_kwlist.h');
+ system('perl ../../../tools/gen_keywords.pl --prefix ecpg_ ecpg_kwlist.h');
+ chdir('../../../..');
+ }
+
if (IsNewer(
'src/interfaces/ecpg/preproc/preproc.y',
'src/backend/parser/gram.y'))
diff --git a/src/tools/msvc/clean.bat b/src/tools/msvc/clean.bat
index 7a23a2b55f..82619ce3d0 100755
--- a/src/tools/msvc/clean.bat
+++ b/src/tools/msvc/clean.bat
@@ -41,6 +41,7 @@ if exist src\include\pg_config.h del /q src\include\pg_config.h
if exist src\include\pg_config_ext.h del /q src\include\pg_config_ext.h
if exist src\include\pg_config_os.h del /q src\include\pg_config_os.h
if %DIST%==1 if exist src\backend\parser\gram.h del /q src\backend\parser\gram.h
+if exist src\include\common/kwlist_d.h del /q src\include\common/kwlist_d.h
if exist src\include\utils\errcodes.h del /q src\include\utils\errcodes.h
if exist src\include\utils\fmgroids.h del /q src\include\utils\fmgroids.h
if exist src\include\utils\fmgrprotos.h del /q src\include\utils\fmgrprotos.h
@@ -51,6 +52,7 @@ if exist src\include\catalog\pg_*_d.h del /q src\include\catalog\pg_*_d.h
if exist src\include\catalog\header-stamp del /q src\include\catalog\header-stamp
if exist doc\src\sgml\version.sgml del /q doc\src\sgml\version.sgml
+if %DIST%==1 if exist src\common\kwlist_d.h del /q src\common\kwlist_d.h
if %DIST%==1 if exist src\backend\utils\fmgroids.h del /q src\backend\utils\fmgroids.h
if %DIST%==1 if exist src\backend\utils\fmgrprotos.h del /q src\backend\utils\fmgrprotos.h
if %DIST%==1 if exist src\backend\utils\fmgrtab.c del /q src\backend\utils\fmgrtab.c
@@ -59,11 +61,15 @@ if %DIST%==1 if exist src\backend\utils\errcodes.h del /q src\backend\utils\errc
if %DIST%==1 if exist src\backend\storage\lmgr\lwlocknames.c del /q src\backend\storage\lmgr\lwlocknames.c
if %DIST%==1 if exist src\backend\storage\lmgr\lwlocknames.h del /q src\backend\storage\lmgr\lwlocknames.h
if %DIST%==1 if exist src\pl\plpython\spiexceptions.h del /q src\pl\plpython\spiexceptions.h
+if %DIST%==1 if exist src\pl\plpgsql\src\pl_reserved_kwlist_d.h del /q src\pl\plpgsql\src\pl_reserved_kwlist_d.h
+if %DIST%==1 if exist src\pl\plpgsql\src\pl_unreserved_kwlist_d.h del /q src\pl\plpgsql\src\pl_unreserved_kwlist_d.h
if %DIST%==1 if exist src\pl\plpgsql\src\plerrcodes.h del /q src\pl\plpgsql\src\plerrcodes.h
if %DIST%==1 if exist src\pl\tcl\pltclerrcodes.h del /q src\pl\tcl\pltclerrcodes.h
if %DIST%==1 if exist src\backend\utils\sort\qsort_tuple.c del /q src\backend\utils\sort\qsort_tuple.c
if %DIST%==1 if exist src\bin\psql\sql_help.c del /q src\bin\psql\sql_help.c
if %DIST%==1 if exist src\bin\psql\sql_help.h del /q src\bin\psql\sql_help.h
+if %DIST%==1 if exist src\interfaces\ecpg\preproc\c_kwlist_d.h del /q src\interfaces\ecpg\preproc\c_kwlist_d.h
+if %DIST%==1 if exist src\interfaces\ecpg\preproc\ecpg_kwlist_d.h del /q src\interfaces\ecpg\preproc\ecpg_kwlist_d.h
if %DIST%==1 if exist src\interfaces\ecpg\preproc\preproc.y del /q src\interfaces\ecpg\preproc\preproc.y
if %DIST%==1 if exist src\backend\catalog\postgres.bki del /q src\backend\catalog\postgres.bki
if %DIST%==1 if exist src\backend\catalog\postgres.description del /q src\backend\catalog\postgres.description
--
2.17.1
