Hello,
src/tools/gen_keywordlist.pl opens and writes its output header before
checking keyword case and ordering. An uppercase, duplicate, or out-of-order
keyword therefore causes the command to fail after truncating a previously
valid header. If no output exists, the command leaves a partial new header.
The attached patch moves output creation and the opening boilerplate after
the existing validation. It does not change keyword parsing, validation
rules, or the generated format. It covers these input validation errors;
it does not make output replacement atomic for later generation or I/O errors.
The patch is based on master at
f52d405113c83f691fffac261bfec297209c21b4.
A standalone regression script is attached separately. Run it against
either the original or patched tree with:
perl check-keyword-output.pl /path/to/postgres/src/tools/gen_keywordlist.pl
Validation:
- On Windows (Git for Windows Perl 5.36.1) and Linux/WSL (Perl 5.34.0),
the original fails 6 of 16 assertions; the patch passes all 16. The
failing assertions cover preserving an existing header and avoiding a
partial new header for each of the three invalid inputs. Valid input
and uppercase keywords with --no-case-fold also remain supported.
- On Windows, outputs from the SQL, ECPG C, ECPG, and both PL/pgSQL keyword
lists are byte-identical before and after, using their build options.
- Perl syntax checks on both platforms and patch application checks pass.
The standalone regression is not wired into PostgreSQL's test suite.
No PostgreSQL server build or full regression run is claimed for this patch.
This was first reported downstream in IvorySQL; the maintainer requested
an upstream submission:
https://github.com/IvorySQL/IvorySQL/pull/2839#issuecomment-6354402538
The patch and validation were prepared with assistance from OpenAI Codex.
Would this small change be appropriate?
# Standalone regression for gen_keywordlist.pl; requires only core Perl modules.
use strict;
use warnings FATAL => 'all';
use File::Temp qw(tempdir);
use FindBin;
use Test::More;
my $generator = shift @ARGV
|| "$FindBin::RealBin/repo/src/tools/gen_keywordlist.pl";
my $tempdir = tempdir(CLEANUP => 1);
my $input = "$tempdir/keywords.h";
my $output = "$tempdir/keywords_d.h";
sub write_keywords
{
open(my $fh, '>', $input) or die "$input: $!";
print $fh map { qq{PG_KEYWORD("$_", TOKEN)\n} } @_;
close($fh) or die "$input: $!";
}
sub read_output
{
open(my $fh, '<', $output) or die "$output: $!";
local $/;
return <$fh>;
}
sub generate
{
return system($^X, $generator, '--output', $tempdir, @_, $input);
}
write_keywords('alpha', 'beta');
is(generate(), 0, 'valid keyword list generates a header');
my $valid_header = read_output();
like($valid_header, qr/#define SCANKEYWORDS_NUM_KEYWORDS 2/,
'generated header contains both keywords');
for my $case (
['uppercase', 'Alpha', 'beta'],
['out of order', 'beta', 'alpha'],
['duplicate', 'alpha', 'alpha'])
{
my ($name, @keywords) = @$case;
open(my $fh, '>', $output) or die "$output: $!";
print $fh $valid_header;
close($fh) or die "$output: $!";
write_keywords(@keywords);
isnt(generate(), 0, "$name keywords are rejected");
is(read_output(), $valid_header,
"$name keywords leave the existing header intact");
unlink($output) or die "$output: $!";
isnt(generate(), 0, "$name keywords without a header are rejected");
ok(!-e $output, "$name keywords do not create a partial header");
}
write_keywords('Alpha', 'beta');
is(generate('--no-case-fold'), 0,
'uppercase remains valid with --no-case-fold');
like(read_output(), qr/"Alpha\\0"/,
'case-sensitive output preserves the uppercase keyword');
done_testing();
diff --git a/src/tools/gen_keywordlist.pl b/src/tools/gen_keywordlist.pl
index bb03dbb..0d55697 100644
--- a/src/tools/gen_keywordlist.pl
+++ b/src/tools/gen_keywordlist.pl
@@ -62,34 +62,6 @@ my $base_filename = $1 . '_d';
my $kw_def_file = $output_path . $base_filename . '.h';
open(my $kif, '<', $kw_input_file) || die "$kw_input_file: $!\n";
-open(my $kwdef, '>', $kw_def_file) || die "$kw_def_file: $!\n";
-
-# Opening boilerplate for keyword definition header.
-printf $kwdef <<EOM, $base_filename, uc $base_filename, uc $base_filename;
-/*-------------------------------------------------------------------------
- *
- * %s.h
- * List of keywords represented as a ScanKeywordList.
- *
- * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
- * Portions Copyright (c) 1994, Regents of the University of California
- *
- * NOTES
- * ******************************
- * *** DO NOT EDIT THIS FILE! ***
- * ******************************
- *
- * It has been GENERATED by src/tools/gen_keywordlist.pl
- *
- *-------------------------------------------------------------------------
- */
-
-#ifndef %s_H
-#define %s_H
-
-#include "common/kwlookup.h"
-
-EOM
# Parse input file for keyword names.
my @keywords;
@@ -124,6 +96,37 @@ for my $i (0 .. $#keywords - 1)
if ($keywords[$i] cmp $keywords[ $i + 1 ]) >= 0;
}
+# Validate the input before opening the output, so invalid keywords cannot
+# truncate an existing header or leave a partial new one.
+open(my $kwdef, '>', $kw_def_file) || die "$kw_def_file: $!\n";
+
+# Opening boilerplate for keyword definition header.
+printf $kwdef <<EOM, $base_filename, uc $base_filename, uc $base_filename;
+/*-------------------------------------------------------------------------
+ *
+ * %s.h
+ * List of keywords represented as a ScanKeywordList.
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * NOTES
+ * ******************************
+ * *** DO NOT EDIT THIS FILE! ***
+ * ******************************
+ *
+ * It has been GENERATED by src/tools/gen_keywordlist.pl
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef %s_H
+#define %s_H
+
+#include "common/kwlookup.h"
+
+EOM
+
# Emit the string containing all the keywords.
printf $kwdef qq|static const char %s_kw_string[] =\n\t"|, $varname;
