Hi all,
I am a member of the Japanese translation team for the PostgreSQL
documentation. When reviewing translations, we check for blank lines that
may indicate a translation mistake. In practice, however, many of the blank
lines we find are already present in the English source.
I am referring here to blank lines within text flow, rather than blank lines
that separate block elements. For example:
<para>
First sentence.
Second sentence.
</para>
Assuming that an automatic formatter for the PostgreSQL documentation is not
currently practical, I wrote a script, check-blank-lines.py, to find these
blank lines. The script is not limited to <para> elements; it reports blank
lines that appear to be unnecessary.
On the current master branch, it reports 35 candidate blank lines in 436
files. That is about 0.1% of the 35,625 blank lines in those files. In my
view, the reported cases are unusual and worth reviewing. The count has also
increased over recent releases:
version Candidate blank lines
15.0 24
16.0 28
17.0 29
18.0 31
master 35
In the rendered HTML, these blank lines appear as whitespace, so removing
them would not change the rendered output. In many cases, though, splitting
the content into separate <para> elements would better express its
structure.
I would like to propose reviewing and correcting the cases reported by the
script, and adding an equivalent check to the documentation build tools.
The check need not use this script as-is; it could provide the same
functionality as part of the existing tooling.
Would this be a useful check to add?
I have attached check-blank-lines.py and its output, blank.log, for
reference.
Regards,
Noboru Saito
#!/usr/bin/env python3
"""Report blank lines in SGML files not adjacent to a block element."""
import argparse
import re
import sys
from pathlib import Path
BLOCK_TAGS = {
"abstract",
"appendix",
"bibliography",
"biblioentry",
"blockquote",
"bridgehead",
"calloutlist",
"chapter",
"caution",
"cmdsynopsis",
"entry",
"epigraph",
"example",
"figure",
"formalpara",
"funcsynopsis",
"glossary",
"glossdiv",
"glossentry",
"glosslist",
"highlights",
"important",
"indexterm",
"informalexample",
"informalfigure",
"informaltable",
"itemizedlist",
"listitem",
"literallayout",
"mediaobject",
"note",
"orderedlist",
"para",
"part",
"preface",
"procedure",
"programlisting",
"refentry",
"refsect1",
"refsect2",
"refsect3",
"refsection",
"revhistory",
"row",
"segmentedlist",
"sect1",
"sect2",
"sect3",
"sect4",
"sect5",
"section",
"sidebar",
"simplelist",
"simpara",
"screen",
"step",
"synopsis",
"table",
"task",
"tip",
"toc",
"title",
"variablelist",
"varlistentry",
"warning",
}
VERBATIM_TAGS = {"literallayout", "programlisting", "screen", "synopsis"}
TAG_RE = re.compile(r"<\s*(/?)\s*([A-Za-z][\w.-]*)\b")
COMMENT_RE = re.compile(r"<!--|-->")
TAG_ONLY_RE = re.compile(r"(?:\s*<[^<>]+>)+\s*")
ENTITY_ONLY_RE = re.compile(r"\s*&[a-z]+;\s*")
def contains_block_tag(line):
return any(
tag.lower() in BLOCK_TAGS
for _, tag in TAG_RE.findall(line)
)
def contains_only_tags(line):
return TAG_ONLY_RE.fullmatch(line) is not None
def is_ignored_adjacent_line(line):
return (
contains_block_tag(line)
or contains_only_tags(line)
or ENTITY_ONLY_RE.fullmatch(line) is not None
or COMMENT_RE.search(line) is not None
)
def find_candidates(path):
lines = path.read_text(encoding="utf-8").splitlines()
in_verbatim = False
in_comment = False
for index, line in enumerate(lines):
comment_markers = list(COMMENT_RE.finditer(line))
was_in_comment = in_comment
for marker in comment_markers:
if marker.group() == "<!--":
in_comment = True
else:
in_comment = False
if was_in_comment or comment_markers:
continue
if in_verbatim:
if any(f"</{tag}" in line.lower() for tag in VERBATIM_TAGS):
in_verbatim = False
continue
if any(f"<{tag}" in line.lower() for tag in VERBATIM_TAGS):
if not any(f"</{tag}" in line.lower() for tag in VERBATIM_TAGS):
in_verbatim = True
continue
if line.strip():
continue
previous_is_block = (
index > 0 and is_ignored_adjacent_line(lines[index - 1])
)
following_is_block = (
index + 1 < len(lines)
and is_ignored_adjacent_line(lines[index + 1])
)
if not previous_is_block and not following_is_block:
yield index + 1, lines[index - 1].strip() if index else "", (
lines[index + 1].strip() if index + 1 < len(lines) else ""
)
def main():
sgml_dir = Path(__file__).resolve().parent
parser = argparse.ArgumentParser(
description=(
"Find blank lines in SGML files that have no block tag on either "
"adjacent line."
)
)
parser.add_argument(
"paths",
nargs="*",
type=Path,
help="SGML files or directories to check (default: this directory)",
)
args = parser.parse_args()
paths = args.paths or [sgml_dir]
files = []
for path in paths:
if path.is_dir():
files.extend(sorted(path.rglob("*.sgml")))
elif path.is_file() and path.suffix == ".sgml":
files.append(path)
else:
parser.error(f"not an SGML file or directory: {path}")
findings = 0
for path in sorted(set(files)):
try:
candidates = find_candidates(path)
for line_number, previous, following in candidates:
try:
display_path = path.relative_to(sgml_dir)
except ValueError:
display_path = path
print(f"{display_path}:{line_number}: ")
print(f" before: {previous}")
print(f" after: {following}")
findings += 1
except (OSError, UnicodeError) as error:
print(f"{path}: {error}", file=sys.stderr)
return 2
print(f"{findings} candidate blank line(s) in {len(set(files))} file(s)")
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
catalogs.sgml:2086:
before: behavior</link>.
after: It is updated by
charset.sgml:2347:
before: Using <command>SET client_encoding TO</command>.
after: Setting the client encoding can be done with this SQL command:
client-auth.sgml:2232:
before: resource.
after: OAuth client support has to be enabled when <productname>PostgreSQL</productname>
config.sgml:5667:
before: except crash recovery.
after: <varname>hot_standby_feedback</varname> will be delayed by use of this feature
config.sgml:5734:
before: can only be set at server start.
after: <literal>max_active_replication_origins</literal> must be set to at least the
datatype.sgml:4012:
before: 4th and 5th bytes set to FF and FE, respectively.
after: Note that IPv6 uses a modified EUI-64 format where the 7th bit
datatype.sgml:4017:
before: change.
after: Generally speaking, any input which is comprised of pairs of hex
datatype.sgml:4023:
before: 12 (6 bytes). Leading and trailing whitespace is ignored.
after: The following are examples of input formats that are accepted:
datetime.sgml:847:
before: 2000, and 2400 are leap years.
after: By contrast, in the older Julian calendar all years divisible by 4 are leap
datetime.sgml:860:
before: until the start of the 20th century.
after: The reform was observed by Great Britain and its dominions (including what
datetime.sgml:864:
before: Thus 2 September 1752 was followed by 14 September 1752.
after: This is why Unix systems that have the <command>cal</command> program
datetime.sgml:889:
before: world, many predating the Gregorian system.
after: For example,
datetime.sgml:894:
before: calendar in 2637 BC.
after: The People's Republic of China uses the Gregorian calendar
docguide.sgml:421:
before: <xref linkend="targets-meson-documentation"/>.
after: The output appears in the
func/func.sgml:51:
before:
after: &func-logical;
indexam.sgml:629:
before: in <filename>amapi.h</filename>.
after: Tests done by this function will typically be a subset of those
indexam.sgml:637:
before: reported by throwing an error.
after: The dependency-related fields of
installation.sgml:1708:
before: option wins.
after: This option is only for developers, to test segment related code.
installation.sgml:3254:
before: wins.
after: This option is only for developers, to test segment related code.
installation.sgml:3269:
before: Individual build targets can be built using <command>ninja</command> <replaceable>target</replaceable>.
after: When no target is specified, everything except documentation is
monitoring.sgml:258:
before: worrying that the numbers are changing underneath you.
after: When analyzing statistics interactively, or with expensive queries, the
monitoring.sgml:268:
before: <varname>stats_fetch_consistency</varname> to <literal>none</literal>.
after: You can invoke <function>pg_stat_clear_snapshot()</function> to discard the
monitoring.sgml:3478:
before:
after:
mvcc.sgml:1132:
before:
after:
pgbuffercache.sgml:595:
before: evict all buffers for all forks in that relation.
after: It returns the number of evicted buffers, flushed buffers and the number of
ref/create_table.sgml:1207:
before: <term><literal>REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH <replaceable class="parameter">matchtype</replaceable> ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ]</literal> (column constraint)</term>
after: <term><literal>FOREIGN KEY ( <replaceable class="parameter">column_name</replaceable> [, ... ] [, PERIOD <replaceable class="parameter">column_name</replaceable> ] )
ref/pg_dump.sgml:1589:
before: <term><envar>PGPORT</envar></term>
after: <term><envar>PGUSER</envar></term>
ref/pg_waldump.sgml:296:
before: <literal><replaceable>TIMELINE</replaceable>-<replaceable>LSN</replaceable>.<replaceable>RELTABLESPACE</replaceable>.<replaceable>DATOID</replaceable>.<replaceable>RELNODE</replaceable>.<replaceable>BLKNO</replaceable>_<replaceable>FORK</replaceable></literal>
after: The file names are composed of the following parts:
ref/psql-ref.sgml:1124:
before: [ where <replaceable class="parameter">condition</replaceable> ]</literal></term>
after: <term><literal>\copy { <replaceable class="parameter">table</replaceable> [ ( <replaceable class="parameter">column_list</replaceable> ) ] | ( <replaceable class="parameter">query</replaceable> ) }
runtime.sgml:715:
before: (see <xref linkend="guc-shared-memory-type"/>).
after: In addition a significant number of semaphores, which can be either
runtime.sgml:1475:
before: <varname>shared_memory_size_in_huge_pages</varname>.
after: While we need at least <literal>3170</literal> huge pages in this example,
sepgsql.sgml:95:
before: linkend="install-meson">meson</link>).
after: Be sure that the <filename>libselinux-devel</filename> RPM is installed at
textsearch.sgml:3152:
before: The available values of the <literal>language</literal> parameter are:
after: <literal>arabic</literal>,
textsearch.sgml:3187:
before: <literal>yiddish</literal>.
after: The <literal>porter</literal> algorithm is an old stemmer for English,
textsearch.sgml:3194:
before: respective languages.
after: All these algorithms except <literal>porter</literal>
35 candidate blank line(s) in 436 file(s)
