https://bugs.documentfoundation.org/show_bug.cgi?id=173798
Bug ID: 173798
Summary: Writer: legacy bidi control character in a REF field
result makes --headless --convert-to allocate without
bound (regression between 26.2 and 26.8)
Product: LibreOffice
Version: 26.8.0.3 release
Hardware: All
OS: Linux (All)
Status: UNCONFIRMED
Keywords: bibisectRequest, perf, regression
Severity: normal
Priority: medium
Component: Writer
Assignee: [email protected]
Reporter: [email protected]
Description: We convert .docx documents to PDF server-side with soffice
--headless --convert-to. After upgrading from 24.8.4.2 to 26.8.0.3, those
conversion processes started being killed by the OOM killer several times a
day. The previous version had run for weeks without a single occurrence. Same
documents, same load, same machines — the LibreOffice version was the only
thing that changed.
Investigating, we found that certain documents make soffice allocate without
bound: memory climbs past 2.5 GB and keeps climbing for as long as the process
is left alive, and the conversion never produces a PDF. The same files convert
in about 2 seconds using roughly 150 MB on 24.8.4.2, and also on 26.2.6.3, so
this is a regression somewhere between 26.2 and 26.8. 26.8.1.1, the latest
release of the branch, behaves exactly like 26.8.0.3.
The trigger is a legacy Unicode bidirectional control character sitting in the
cached result of a REF cross-reference field. Neither part is enough on its own
— taking U+200E (LEFT-TO-RIGHT MARK) as the example:
U+200E in ordinary text, no field converts, ~171 MB
REF field whose result has no U+200E converts, ~509 MB
U+200E in the result of a PAGE field converts, ~492 MB
U+200E in the result of a REF field never completes, 2.6 - 2.9 GB
Seven characters behave this way, listed further down; U+200E is simply the one
we hit in production. That combination is not exotic: Word inserts this
character by itself when it writes cross-references, which is why it turns up
in ordinary documents produced from templates.
The attached documents are built from scratch by a small script, not derived
from any existing file. Three parts, 1.1 KB, no metadata, no images, no styles,
no revision ids. The two differ by nine bytes: the single U+200E character.
Steps to Reproduce:
1. Take the attached ref-field-lrm.docx.
2. Run: soffice --headless --convert-to pdf --outdir /tmp ref-field-lrm.docx
3. Watch the resident memory of the process tree.
4. For comparison, do the same with ref-field-lrm-control.docx, which is the
same document with the U+200E removed from the field result.
Actual Results: The conversion does not complete. Memory climbs continuously —
measured peaks of 2.6 to 2.9 GB when the process is killed after 30 seconds,
and the figure tracks how long the process is allowed to run, so it does not
appear to plateau. No PDF is produced.
In roughly one run out of three, SIGTERM was not enough to stop it and only the
following SIGKILL did; the process seems too busy allocating to service the
signal.
Expected Results: The conversion completes and produces a PDF, as it does on
24.8.4.2 and 26.2.6.3, in about 2 seconds using roughly 150 MB.
Reproducible: Always
Additional Info:
VERSIONS TESTED
Official TDF .deb packages on Linux x86-64, default user profile, no
extensions, conversions run one at a time, no network access. Memory is the
peak of the process tree sampled every 50 ms. Each conversion is given 30 s
before being killed. Three runs per version.
version ref-field-lrm.docx ref-field-lrm-control.docx
--------- ------------------------- --------------------------
24.8.4.2 converts, ~143 MB converts, ~153 MB
26.2.6.3 converts, ~131 MB converts, ~132 MB
26.8.1.1 never completes, ~2.66 GB converts, ~137 MB
On the larger document of tdf#173432 (its attachment 208431), for reference:
24.8.4.2 converts in 2.3 s, ~209 MB
26.2.6.3 converts in 2.3 s, ~195 MB
26.8.0.3 never completes, 2.7 - 3.4 GB
26.8.1.1 never completes, 2.7 - 3.3 GB
26.2.6.3 is in fact slightly cheaper than 24.8.4.2 on the same input.
HOW THE TRIGGER WAS FOUND
Starting from attachment 208431 of tdf#173432, package parts were replaced with
empty or stub content one at a time. None of these made any difference — all
still blew up: the only WMF image, both charts (each carrying an external
Target="Book1" that cannot be resolved), the only OLE object, the only JPEG,
numbering.xml, styles.xml, stylesWithEffects.xml, settings.xml, fontTable.xml,
theme1.xml.
Two plausible suspects are therefore ruled out: the unresolvable external chart
links, and the WMF/EMF import path.
The body was then bisected structurally. It has 361 top-level children; keeping
only the first N of them plus the trailing sectPr:
N = 210 converts normally
N = 212 converts normally, ~167 MB
N = 213 never completes, ~3.3 GB
N = 361 never completes
The 213th block is a paragraph carrying a REF field. Rebuilding an equivalent
paragraph from scratch, however, did NOT reproduce — which is how the U+200E
came to light. The original cached result contains one, and adding it back to
the generated document makes it blow up. The four-line table at the top of this
report is the result of testing each element separately, each in its own
generated document.
Two further negatives worth recording: the character is enough on its own, it
does not need the complex-script markup (w:cs, w:rFonts w:hint="cs") it carries
in the original; and an empty run between instrText and separate, also present
in the original, changes nothing.
WHICH CHARACTERS
Same document each time, generated separately, only the character inside the
field result changing. On 26.8.1.1:
U+200E LEFT-TO-RIGHT MARK never completes, ~3.09 GB
U+200F RIGHT-TO-LEFT MARK never completes, ~2.91 GB
U+202A LEFT-TO-RIGHT EMBEDDING never completes, ~2.52 GB
U+202B RIGHT-TO-LEFT EMBEDDING never completes, ~3.42 GB
U+202C POP DIRECTIONAL FORMATTING never completes, ~3.73 GB
U+202D LEFT-TO-RIGHT OVERRIDE never completes, ~3.62 GB
U+202E RIGHT-TO-LEFT OVERRIDE never completes, ~2.94 GB
U+2066 LEFT-TO-RIGHT ISOLATE converts, ~512 MB
U+2069 POP DIRECTIONAL ISOLATE converts, ~132 MB
U+200B ZERO WIDTH SPACE converts, ~482 MB
U+200D ZERO WIDTH JOINER converts, ~133 MB
U+2060 WORD JOINER converts, ~508 MB
U+00AD SOFT HYPHEN converts, ~133 MB
(no character at all) converts, ~411 MB
The split is not "invisible characters". It is the seven legacy bidi controls,
all of them, and nothing else we tried. The isolates introduced in Unicode 6.3
are fine, and so are invisible characters with no directional meaning. That
points at whatever handles legacy bidi controls in the layout of a field
result.
WHAT WE HAVE NOT TESTED
This list is what we happened to try, not a systematic sweep, and we stopped
once the pattern was clear. Worth checking before assuming the scope:
- the remaining isolates, U+2067 RLI and U+2068 FSI, which we did not run
- other field types than REF and PAGE — we only compared those two
- the same characters in a document opened interactively rather than
converted headless
- whether more than one such character, or one in a RTL paragraph, changes
the picture
We would not be surprised if the real condition is broader than REF fields
specifically. It is simply the case we could reach from a real document.
SETTING THAT DOES NOT HELP
Writer / Content / Update / Field = false (do not update fields on load) makes
no difference: peaks are the same with and without it. Whatever is expensive
here does not appear to be the field update on load.
RELATION TO tdf#173432
That report describes very high memory on a larger document, filed against
Windows and describing an interactive open. This one is Linux, the headless
conversion path, and narrows the cause to one character in one field. The
document behind it is the same one, so this is likely the same underlying
defect — please mark as a duplicate if a bibisect shows one commit behind both.
ATTACHMENTS
ref-field-lrm.docx reproduces
ref-field-lrm-control.docx identical but for the U+200E, converts normally
Both are generated, contain nothing but [Content_Types].xml, _rels/.rels and
word/document.xml, and carry no third-party content of any kind.
--
You are receiving this mail because:
You are the assignee for the bug.
