Gustavo A. created PDFBOX-6252:
----------------------------------
Summary: PDDocument.saveIncremental() writes an update section
that contradicts the sections it extends, on a document whose cross-reference
marks an object free
Key: PDFBOX-6252
URL: https://issues.apache.org/jira/browse/PDFBOX-6252
Project: PDFBox
Issue Type: Bug
Affects Versions: 3.0.8 PDFBox, 4.0.0
Reporter: Gustavo A.
h2. Summary
{{PDDocument#saveIncremental}} derives the update section's trailer and
cross-reference entries
from the objects it holds in memory, rather than from the combination of its
own section with the
ones already in the file. On a document whose cross-reference marks an object
free, the section it
appends contradicts the sections it extends in three ways: {{/Size}} shrinks,
the free list is
discarded, and a number the previous section maps into that free list is handed
out to a new
object.
No signing is involved, and neither is a cross-reference stream nor a file
produced by an older
PDFBox: the input built by the attached reproducers is a five-object PDF
written by hand. Measured
identically on 3.0.6, 3.0.7, 3.0.8, and on the 3.0.9-SNAPSHOT and
4.0.0-SNAPSHOT builds published
on 2026-09-03 (JDK 25). None of the three is a regression, and none is fixed in
trunk.
h2. Steps to reproduce
All the save does is this, on the attached {{{}free-object-input.pdf{}}}:
{code:java}
try (PDDocument doc = Loader.loadPDF(input))
{
doc.getDocumentInformation().setTitle("a change, so the save has something
to write");
// doc.addPage(new PDPage()); // uncomment for reading 3 below
doc.saveIncremental(out);
}
{code}
{{IncrementalXrefRepro.java}} is attached and does exactly that. It needs
nothing but PDFBox and
its own runtime dependencies, and builds the same input in memory, so it can be
run on its own:
{noformat}
java -cp
"pdfbox-3.0.8.jar;pdfbox-io-3.0.8.jar;fontbox-3.0.8.jar;commons-logging-1.4.0.jar"
IncrementalXrefRepro.java
{noformat}
The input is a five-object PDF whose cross-reference table covers 0..6 in a
single subsection,
declares {{{}/Size 7{}}}, marks object 6 free with the free list linked 0 -> 6
-> end, and whose
catalog references it as {{{}/Outlines 6 0 R{}}}. That is the shape a document
takes when an exporter
drops an outline tree but leaves the reference behind; it is common enough in
office-suite output
that we ran into it in production. The program changes only the
document-information title, so the
update section is as small as an update section gets.
The reproducer runs the same input a second time with nothing referencing
object 6 at all. The update section comes out identical, so the dangling
reference is not what is at fault.
There is no exception and so no stack trace. With logging active, nothing is
written at INFO level or above during either the load or the save; that silence
is part of what is being reported, and the PDFBOX-5382 section below covers why
{{SigUtils.checkCrossReferenceTable}} does not catch it either.
h2. Expected result
The appended section, combined with the one it extends, still describes the
same address space:
{{/Size}} stays at least 7, object 6 stays free and reachable from the head of
the free list, and
any object the save adds gets a number no previous section has already spoken
for.
h2. Actual result
{noformat}
original cross-reference section
/Size : 7
free-list head : 0000000006 65535 f (object 0 -> object 6)
object 6 : free, end of the list
catalog : /Outlines 6 0 R
update section written by saveIncremental
/Size : 6
free-list head : 0000000000 65535 f
{noformat}
h3. 1. /Size shrinks
The input declares {{{}/Size 7{}}}; the update section declares {{{}/Size
6{}}}. Table 15 defines it as
"the total number of entries in the file's cross-reference table, as defined by
the combination of
the original section and all update sections" – seven entries here, numbered 0
to 6 – and goes on
to say that "any object in a cross-reference section whose number is greater
than this value shall
be ignored and defined to be missing by a conforming reader". So this is not a
cosmetic
discrepancy: the spec instructs readers to treat object 6 as gone, while the
catalog still
references it as {{{}/Outlines 6 0 R{}}}.
7.5.6 says the same thing from the other side: "The added trailer shall contain
all the entries
except the Prev entry (if present) from the previous trailer, whether modified
or not". And 7.5.4
requires the combined table to "contain one entry for each object number from 0
to the maximum
object number defined in the file, even if one or more of the object numbers in
this range do not
actually occur in the file", which a lowered {{/Size}} silently undoes.
h3. 2. The free list is broken
The update section writes {{{}0 1 / 0000000000 65535 f{}}}, declaring the free
list empty, while the
previous section still marks object 6 free with generation 0. 7.5.4 gives free
entries exactly two
lawful shapes: members of the linked list headed by object 0, or entries that
"link back to object
number 0 and have a generation number of 65,535, even though these entries are
not in the linked
list itself". After the update, object 6 is neither: it is not reachable from
the head, and its
generation is 0, not 65,535.
Annex H.7 shows what that entry is {_}for{_}. In H.7.3, an update section that
frees two objects writes
{{{}0 1 / 0000000008 65535 f{}}}, moving the head onto the object it just
freed; in H.7.4, which reuses
both of them, it writes {{{}0000000000 65535 f{}}}, because by then the list
really is empty. That
entry is how the head is carried across revisions. PDFBox writes the second
form unconditionally.
The same example never lowers {{/Size}} either: 12, 12, 12, then 13.
h3. 3. A number the previous section maps into its free list is handed out to a
new object
The two readings above come from a save that adds nothing. If the same save
adds any object – a
{{PDPage}} in the attached {{{}XrefReuseRepro.java{}}}, though the type is
irrelevant – that object is
written as {{{}6 0 obj{}}}, the number the previous section marks free and the
catalog still
references. Read back, {{/Outlines 6 0 R}} from the earlier revision resolves
to the new page, in
PDFBox itself and in pypdf 6.16.2:
{noformat}
BEFORE /Outlines -> IndirectObject(6, 0) "Object 6 0 not defined."
(dangling)
AFTER /Outlines -> IndirectObject(6, 0) {'/Type': '/Page', '/MediaBox':
[...]}
{noformat}
Reusing a free number is legal on its own – 7.5.4 makes the free entry's
generation the one to use
on reuse – so the objection is not the reuse. It is that PDFBox is not choosing
to reuse anything:
the number looks unallocated, because the free entry is not in the model. When
the earlier revision
is signed, what it says changes underneath it.
h2. What decides it is the seed, not the kind of section
{{/Size}} is written as _highest object number written_ + 1, and the numbering
starts from
{{{}COSDocument#getHighestXRefObjectNumber(){}}}, which counts only numbers
that have an in-use entry.
So all three readings follow from one thing: numbers the previous sections put
in the free list are
invisible, and everything the writer decides is measured from a ceiling that is
therefore too low.
Two further reproducers make that concrete rather than asserted:
* {{XrefStreamRepro.java}} – a cross-reference _stream_ input with object 6
free but the xref
stream object (7) above it. {{/Size}} comes out right (8 -> 9) and the one new
object the save
writes, its own {{/XRef}} stream, is numbered 8, because the visible ceiling
happens to be above
the free number. Only reading 2 shows.
* {{XrefStreamTopFree.java}} – the same, with the free objects at the _top_ of
the address space
(1..5 in use, 6 the xref stream, 7 and 8 free, {{{}/Size 9{}}}, catalog
{{{}/Outlines 7 0 R{}}}). Here
{{/Size}} drops to 8 on every build tested, and the update's own
cross-reference stream is written
as object {*}7{*}, so {{/Outlines 7 0 R}} resolves to the {{/XRef}} dictionary
itself – the same
reading PDFBOX-6236 describes, reached from a free entry instead of a missing
one. For a
cross-reference stream the requirement is stated even more directly than in
table 15: table 17
defines {{/Size}} as "the number one greater than the highest object number
used in this section
{_}or in any section for which this shall be an update{_}" – 9 here, not 8.
So reading 1 is not specific to cross-reference tables; it appears wherever
free numbers sit above
the highest in-use one.
Incidentally, {{XrefStreamRepro.java}} also shows PDFBOX-6176 taking effect.
The update's own xref
stream is object 8, and 3.0.6 and 3.0.7 omit its entry:
{noformat}
3.0.6, 3.0.7 /Index [0 1 5 1]
3.0.8, 3.0.9-SNAPSHOT, 4.0.0-SNAP /Index [0 1 5 1 8 1]
{noformat}
h2. Suggested area to investigate
Reading trunk as of 2026-09-03:
* {{COSWriter#write(PDDocument, SignatureInterface)}} seeds the allocator with
{{{}number = pdDocument.getDocument().getHighestXRefObjectNumber(){}}}.
* {{COSWriter#doWriteTrailer(COSDocument)}} writes
{{trailer.setLong(COSName.SIZE, number + 1)}}
for the non-xref-stream case; the stream path does the equivalent in
{{doWriteXRefInc}} with
{{{}pdfxRefStream.setSize(number + 1){}}}.
* {{COSWriter#doWriteXRefTable()}} writes the free-list entry unconditionally
on the incremental
branch: {{{}addXRefEntry(FreeXReference.NULL_ENTRY){}}}. The
{{fillGapsWithFreeEntries()}} next to it
runs only when the save is _not_ incremental.
* The cross-reference stream path arrives at the same entry by a different
route:
{{PDFXRefStream#getIndexEntry()}} always adds object 0 to the index, and
{{PDFXRefStream#writeStreamData()}} always writes {{FreeXReference.NULL_ENTRY}}
as the first entry
of the stream. So reading 2 is not specific to tables either.
h2. Suggested fix
All three come from the same place – the update section is computed from
PDFBox's object model
rather than from the file it extends – and each has a narrow fix:
* {*}Number allocation{*}: seed it with {{max(getHighestXRefObjectNumber(),
previous trailer /Size - 1)}}
rather than with {{getHighestXRefObjectNumber()}} alone. The previous trailer
already declares the
address space the file claims, whether or not every number in it has an entry,
so object 6 stops
looking unallocated without the writer having to learn what the free list
holds. On a well-formed
file the two values agree and nothing changes. The same rule also covers
PDFBOX-6236, where the
invisible number is a cross-reference stream object rather than a free one.
* {*}The Size entry{*}: take {{max(previous trailer /Size, highest object
number written + 1)}}
instead of a count of the in-memory objects. An incremental update can never
lower the address space of the file
it extends.
* {*}Free list{*}: carry the previous head forward instead of writing
{{0000000000 65535 f}}
unconditionally, which is what annex H.7's own update sections do. Omitting the
entry altogether is
equally correct and even smaller: 7.5.6 asks an update section for "entries
only for objects that
have been changed, replaced, or deleted", and object 0 has not changed.
h2. What is not being suggested
Since it is the natural next question: please do not repair the document before
an incremental
save. Rewriting anything in the existing bytes would break every signature
already in the file,
which is exactly what an incremental save exists to avoid. Any repair has to be
appended as its own
revision, by the caller, before the first signature. That is what we ended up
doing on our side,
and it is out of scope for PDFBox.
Likewise, a strict mode that refuses malformed input would run against PDFBox's
own design;
tolerating broken documents is a large part of why people reach for it. The ask
here is only that
the section PDFBox itself writes not contradict the ones it is extending.
h2. Relation to PDFBOX-6236
That issue is the closest sibling, and the mechanism looks like the same one.
There, a new field
object is given the number of the increment's own {{/XRef}} stream, because
that object had no
entry in its own revision's cross-reference, so to the writer the number looked
unallocated.
PDFBOX-6176 fixed the missing entry in 3.0.8.
What is reported here is the same "the number looks unallocated" condition
arriving from a
different direction – a free entry rather than a missing one – and that
direction is _not_ fixed
in 3.0.8.
Measured on that issue's own attachment, for what it is worth: loading
{{already-signed.pdf}} with
3.0.8 reports {{getHighestXRefObjectNumber() = 42}} while the newest trailer of
the file says
{{{}/Size 44{}}}, so {{addPage}} plus {{saveIncremental}} writes the new page
as object 43 – the number
revision 4's cross-reference stream already occupies at offset 76040. PDFBox
reads the result as
two pages, pypdf 6.16.2 as one. Identical on all five builds tested. Both cases
share the seed, and
the first suggestion above fixes both.
h2. Relation to PDFBOX-5382
All three readings look like the write-side consequence of the limitation
discussed there. In that
thread, {{SigUtils.checkCrossReferenceTable}} was added in r1898484/r1898485
and wired into the
three signing examples in r1898487/r1898488; r1899085/r1899086 then reverted
the call sites only
("revert, but keep a comment for now"), after a discussion in which Michael
Klink pointed out that
{{{}COSDocument{}}}'s cross-reference {{Map}} holds no entries for free
objects, so an object number
mapped into the free list cannot be told apart from one that is absent
altogether. The method
itself was never removed; it is in trunk today, and {{CreateSignature}} still
carries the line
{{{}// call SigUtils.checkCrossReferenceTable(document) if Adobe complains{}}}.
That is the same missing information, one step later: because the free entries
are gone by the time
the update section is written, the writer has no free list to carry forward and
no object 6 to
count towards {{{}/Size{}}}.
The difference worth drawing out is the direction. PDFBOX-5382 is about
_noticing_ a gap in a
document that arrives with one. What is reported here is on the other side:
given a document where
object 6 is mapped into the free list, which is a legitimate mapping, an
incremental save hands
back one where it is not, because the newest {{/Size}} no longer covers it and
the newest free-list
head no longer leads to it.
Worth stating plainly, since it is the obvious question:
{{SigUtils.checkCrossReferenceTable}} does
*not* warn on either file, before or after the save.
{{COSDocument#getXrefTable()}} comes back with
keys 1..5 and {{getHighestXRefObjectNumber()}} with 5, so object 6 is absent
from the model and
nothing looks missing to the check. A control with an actual hole (object 4
defined in no section)
does produce the warning, so this is not a logging artefact. The blind spot is
the same one Michael
Klink described in 2022; the point here is only that the writer sits behind it
too.
h2. Impact
A conforming reader is instructed by table 15 to treat object 6 as missing
after the save, and the
free list the file used to have is gone. Where it bites hardest is a signed
document: references
made by an earlier, signed revision can be made to resolve to different objects
by a later
incremental save, without any of the existing bytes being touched.
On the reader side we can offer a measurement rather than the "may fail
validation in Acrobat
Reader" of PDFBOX-5382, though it needs stating precisely, because it is a
necessary condition and
not a sufficient one. Bisecting roughly thirty signed variants of one document,
Acrobat reports an
earlier signature as "altered or corrupted" as soon as any later revision is
appended – even one
whose entire content is a single object nothing references – when the signed
document has *both* a
gap in the object numbering *and* annotations written inline in a page's
{{/Annots}} (where ISO
32000-1 table 30 requires indirect references). Repairing either one alone left
the signature
invalid; repairing both made it valid. So the gap is half of a reproducible
Acrobat failure, not
the whole of it, and the other half is nothing to do with PDFBox. What is
squarely PDFBox's is that
an incremental save _introduces_ the condition: given a document whose
{{/Size}} is 7, it hands
back one whose newest section says 6.
A last note for completeness, since it bears on how much of this is worth
chasing: the document
that started our investigation broke identically under a signing stack that
uses no PDFBox at all.
The defect that caused it was in the source PDF, and we fixed it on our side by
appending a repair
revision before the first signature. Nothing in this report is offered as the
cause of that.
h2. Attachments
The PDFs are the inputs and outputs of the two table-based readings, in case it
is easier to look
at the bytes than to run anything. They are synthetic, a few hundred bytes
each, and contain no
data of any kind.
* {{free-object-input.pdf}} – the five-object input: {{{}/Size 7{}}}, object 6
free, list 0 -> 6 ->
end, catalog {{{}/Outlines 6 0 R{}}}.
* {{free-object-after-addpage.pdf}} – the same file after {{addPage}} plus
{{saveIncremental}} on
3.0.8, where {{/Outlines 6 0 R}} now resolves to the new page.
* {{topfree-input.pdf}} and {{topfree-after-incremental.pdf}} – the
cross-reference stream pair,
where {{/Size}} drops from 9 to 8 and {{/Outlines 7 0 R}} ends up resolving to
the update's own
{{/XRef}} dictionary.
* {{IncrementalXrefRepro.java}} – readings 1 and 2, cross-reference table
input, run twice (with
and without a reference to the free object).
* {{XrefReuseRepro.java}} – reading 3, same input, the save adds a
{{{}PDPage{}}}.
* {{XrefStreamRepro.java}} – cross-reference stream input whose free number is
below the highest
in-use one; only reading 2 shows.
* {{XrefStreamTopFree.java}} – cross-reference stream input whose free numbers
are above it; the
update's own {{/XRef}} stream takes a referenced free number.
All four run the same way as the first, needing only PDFBox and its runtime
dependencies – on
4.0.0-SNAPSHOT, {{log4j-api}} as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
