[
https://issues.apache.org/jira/browse/PDFBOX-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo A. updated PDFBOX-6252:
-------------------------------
Component/s: Writing
Affects Version/s: 2.0.37
> PDDocument.saveIncremental() writes an update section that contradicts the
> sections it extends, on a document whose cross-reference marks an object free
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: PDFBOX-6252
> URL: https://issues.apache.org/jira/browse/PDFBOX-6252
> Project: PDFBox
> Issue Type: Bug
> Components: Writing
> Affects Versions: 2.0.37, 3.0.8 PDFBox, 4.0.0
> Reporter: Gustavo A.
> Priority: Major
> Attachments: IncrementalXrefRepro.java, IncrementalXrefRepro20.java,
> XrefReuseRepro.java, XrefStreamRepro.java, XrefStreamTopFree.java,
> free-object-after-addpage.pdf, free-object-input.pdf,
> topfree-after-incremental.pdf, topfree-input.pdf
>
>
> h2. Summary
> {{PDDocument#saveIncremental}} derives the update section's trailer and
> cross-reference entries
> from the objects it holds in memory, rather than from the combination of its
> own section with the
> ones already in the file. On a document whose cross-reference marks an object
> free, the section it
> appends contradicts the sections it extends in three ways: {{/Size}} shrinks,
> the free list is
> discarded, and a number the previous section maps into that free list is
> handed out to a new
> object.
> No signing is involved, and neither is a cross-reference stream nor a file
> produced by an older
> PDFBox: the input built by the attached reproducers is a five-object PDF
> written by hand. Measured
> identically on 3.0.6, 3.0.7, 3.0.8, and on the 3.0.9-SNAPSHOT and
> 4.0.0-SNAPSHOT builds published
> on 2026-09-03 (JDK 25). None of the three is a regression, and none is fixed
> in trunk.
> h2. Steps to reproduce
> All the save does is this, on the attached {{{}free-object-input.pdf{}}}:
> {code:java}
> try (PDDocument doc = Loader.loadPDF(input))
> {
> doc.getDocumentInformation().setTitle("a change, so the save has
> something to write");
> // doc.addPage(new PDPage()); // uncomment for reading 3 below
> doc.saveIncremental(out);
> }
> {code}
> {{IncrementalXrefRepro.java}} is attached and does exactly that. It needs
> nothing but PDFBox and
> its own runtime dependencies, and builds the same input in memory, so it can
> be run on its own:
> {noformat}
> java -cp
> "pdfbox-3.0.8.jar;pdfbox-io-3.0.8.jar;fontbox-3.0.8.jar;commons-logging-1.4.0.jar"
> IncrementalXrefRepro.java
> {noformat}
> The input is a five-object PDF whose cross-reference table covers 0..6 in a
> single subsection,
> declares {{{}/Size 7{}}}, marks object 6 free with the free list linked 0 ->
> 6 -> end, and whose
> catalog references it as {{{}/Outlines 6 0 R{}}}. That is the shape a
> document takes when an exporter
> drops an outline tree but leaves the reference behind; it is common enough in
> office-suite output
> that we ran into it in production. The program changes only the
> document-information title, so the
> update section is as small as an update section gets.
> The reproducer runs the same input a second time with nothing referencing
> object 6 at all. The update section comes out identical, so the dangling
> reference is not what is at fault.
> There is no exception and so no stack trace. With logging active, nothing is
> written at INFO level or above during either the load or the save; that
> silence is part of what is being reported, and the PDFBOX-5382 section below
> covers why {{SigUtils.checkCrossReferenceTable}} does not catch it either.
> h2. Expected result
> The appended section, combined with the one it extends, still describes the
> same address space:
> {{/Size}} stays at least 7, object 6 stays free and reachable from the head
> of the free list, and
> any object the save adds gets a number no previous section has already spoken
> for.
> h2. Actual result
> {noformat}
> original cross-reference section
> /Size : 7
> free-list head : 0000000006 65535 f (object 0 -> object 6)
> object 6 : free, end of the list
> catalog : /Outlines 6 0 R
> update section written by saveIncremental
> /Size : 6
> free-list head : 0000000000 65535 f
> {noformat}
> h3. 1. /Size shrinks
> The input declares {{{}/Size 7{}}}; the update section declares {{{}/Size
> 6{}}}. Table 15 defines it as
> "the total number of entries in the file's cross-reference table, as defined
> by the combination of
> the original section and all update sections" – seven entries here, numbered
> 0 to 6 – and goes on
> to say that "any object in a cross-reference section whose number is greater
> than this value shall
> be ignored and defined to be missing by a conforming reader". So this is not
> a cosmetic
> discrepancy: the spec instructs readers to treat object 6 as gone, while the
> catalog still
> references it as {{{}/Outlines 6 0 R{}}}.
> 7.5.6 says the same thing from the other side: "The added trailer shall
> contain all the entries
> except the Prev entry (if present) from the previous trailer, whether
> modified or not". And 7.5.4
> requires the combined table to "contain one entry for each object number from
> 0 to the maximum
> object number defined in the file, even if one or more of the object numbers
> in this range do not
> actually occur in the file", which a lowered {{/Size}} silently undoes.
> h3. 2. The free list is broken
> The update section writes {{{}0 1 / 0000000000 65535 f{}}}, declaring the
> free list empty, while the
> previous section still marks object 6 free with generation 0. 7.5.4 gives
> free entries exactly two
> lawful shapes: members of the linked list headed by object 0, or entries that
> "link back to object
> number 0 and have a generation number of 65,535, even though these entries
> are not in the linked
> list itself". After the update, object 6 is neither: it is not reachable from
> the head, and its
> generation is 0, not 65,535.
> Annex H.7 shows what that entry is {_}for{_}. In H.7.3, an update section
> that frees two objects writes
> {{{}0 1 / 0000000008 65535 f{}}}, moving the head onto the object it just
> freed; in H.7.4, which reuses
> both of them, it writes {{{}0000000000 65535 f{}}}, because by then the list
> really is empty. That
> entry is how the head is carried across revisions. PDFBox writes the second
> form unconditionally.
> The same example never lowers {{/Size}} either: 12, 12, 12, then 13.
> h3. 3. A number the previous section maps into its free list is handed out to
> a new object
> The two readings above come from a save that adds nothing. If the same save
> adds any object – a
> {{PDPage}} in the attached {{{}XrefReuseRepro.java{}}}, though the type is
> irrelevant – that object is
> written as {{{}6 0 obj{}}}, the number the previous section marks free and
> the catalog still
> references. Read back, {{/Outlines 6 0 R}} from the earlier revision resolves
> to the new page, in
> PDFBox itself and in pypdf 6.16.2:
> {noformat}
> BEFORE /Outlines -> IndirectObject(6, 0) "Object 6 0 not defined."
> (dangling)
> AFTER /Outlines -> IndirectObject(6, 0) {'/Type': '/Page', '/MediaBox':
> [...]}
> {noformat}
> Reusing a free number is legal on its own – 7.5.4 makes the free entry's
> generation the one to use
> on reuse – so the objection is not the reuse. It is that PDFBox is not
> choosing to reuse anything:
> the number looks unallocated, because the free entry is not in the model.
> When the earlier revision
> is signed, what it says changes underneath it.
> h2. What decides it is the seed, not the kind of section
> {{/Size}} is written as _highest object number written_ + 1, and the
> numbering starts from
> {{{}COSDocument#getHighestXRefObjectNumber(){}}}, which counts only numbers
> that have an in-use entry.
> So all three readings follow from one thing: numbers the previous sections
> put in the free list are
> invisible, and everything the writer decides is measured from a ceiling that
> is therefore too low.
> Two further reproducers make that concrete rather than asserted:
> * {{XrefStreamRepro.java}} – a cross-reference _stream_ input with object 6
> free but the xref
> stream object (7) above it. {{/Size}} comes out right (8 -> 9) and the one
> new object the save
> writes, its own {{/XRef}} stream, is numbered 8, because the visible ceiling
> happens to be above
> the free number. Only reading 2 shows.
> * {{XrefStreamTopFree.java}} – the same, with the free objects at the _top_
> of the address space
> (1..5 in use, 6 the xref stream, 7 and 8 free, {{{}/Size 9{}}}, catalog
> {{{}/Outlines 7 0 R{}}}). Here
> {{/Size}} drops to 8 on every build tested, and the update's own
> cross-reference stream is written
> as object {*}7{*}, so {{/Outlines 7 0 R}} resolves to the {{/XRef}}
> dictionary itself – the same
> reading PDFBOX-6236 describes, reached from a free entry instead of a missing
> one. For a
> cross-reference stream the requirement is stated even more directly than in
> table 15: table 17
> defines {{/Size}} as "the number one greater than the highest object number
> used in this section
> {_}or in any section for which this shall be an update{_}" – 9 here, not 8.
> So reading 1 is not specific to cross-reference tables; it appears wherever
> free numbers sit above
> the highest in-use one.
> Incidentally, {{XrefStreamRepro.java}} also shows PDFBOX-6176 taking effect.
> The update's own xref
> stream is object 8, and 3.0.6 and 3.0.7 omit its entry:
> {noformat}
> 3.0.6, 3.0.7 /Index [0 1 5 1]
> 3.0.8, 3.0.9-SNAPSHOT, 4.0.0-SNAP /Index [0 1 5 1 8 1]
> {noformat}
> h2. Suggested area to investigate
> Reading trunk as of 2026-09-03:
> * {{COSWriter#write(PDDocument, SignatureInterface)}} seeds the allocator
> with
> {{{}number = pdDocument.getDocument().getHighestXRefObjectNumber(){}}}.
> * {{COSWriter#doWriteTrailer(COSDocument)}} writes
> {{trailer.setLong(COSName.SIZE, number + 1)}}
> for the non-xref-stream case; the stream path does the equivalent in
> {{doWriteXRefInc}} with
> {{{}pdfxRefStream.setSize(number + 1){}}}.
> * {{COSWriter#doWriteXRefTable()}} writes the free-list entry
> unconditionally on the incremental
> branch: {{{}addXRefEntry(FreeXReference.NULL_ENTRY){}}}. The
> {{fillGapsWithFreeEntries()}} next to it
> runs only when the save is _not_ incremental.
> * The cross-reference stream path arrives at the same entry by a different
> route:
> {{PDFXRefStream#getIndexEntry()}} always adds object 0 to the index, and
> {{PDFXRefStream#writeStreamData()}} always writes
> {{FreeXReference.NULL_ENTRY}} as the first entry
> of the stream. So reading 2 is not specific to tables either.
> h2. Suggested fix
> All three come from the same place – the update section is computed from
> PDFBox's object model
> rather than from the file it extends – and each has a narrow fix:
> * {*}Number allocation{*}: seed it with {{max(getHighestXRefObjectNumber(),
> previous trailer /Size - 1)}}
> rather than with {{getHighestXRefObjectNumber()}} alone. The previous trailer
> already declares the
> address space the file claims, whether or not every number in it has an
> entry, so object 6 stops
> looking unallocated without the writer having to learn what the free list
> holds. On a well-formed
> file the two values agree and nothing changes. The same rule also covers
> PDFBOX-6236, where the
> invisible number is a cross-reference stream object rather than a free one.
> * {*}The Size entry{*}: take {{max(previous trailer /Size, highest object
> number written + 1)}}
> instead of a count of the in-memory objects. An incremental update can never
> lower the address space of the file
> it extends.
> * {*}Free list{*}: carry the previous head forward instead of writing
> {{0000000000 65535 f}}
> unconditionally, which is what annex H.7's own update sections do. Omitting
> the entry altogether is
> equally correct and even smaller: 7.5.6 asks an update section for "entries
> only for objects that
> have been changed, replaced, or deleted", and object 0 has not changed.
> h2. What is not being suggested
> Since it is the natural next question: please do not repair the document
> before an incremental
> save. Rewriting anything in the existing bytes would break every signature
> already in the file,
> which is exactly what an incremental save exists to avoid. Any repair has to
> be appended as its own
> revision, by the caller, before the first signature. That is what we ended up
> doing on our side,
> and it is out of scope for PDFBox.
> Likewise, a strict mode that refuses malformed input would run against
> PDFBox's own design;
> tolerating broken documents is a large part of why people reach for it. The
> ask here is only that
> the section PDFBox itself writes not contradict the ones it is extending.
> h2. Relation to PDFBOX-6236
> That issue is the closest sibling, and the mechanism looks like the same one.
> There, a new field
> object is given the number of the increment's own {{/XRef}} stream, because
> that object had no
> entry in its own revision's cross-reference, so to the writer the number
> looked unallocated.
> PDFBOX-6176 fixed the missing entry in 3.0.8.
> What is reported here is the same "the number looks unallocated" condition
> arriving from a
> different direction – a free entry rather than a missing one – and that
> direction is _not_ fixed
> in 3.0.8.
> Measured on that issue's own attachment, for what it is worth: loading
> {{already-signed.pdf}} with
> 3.0.8 reports {{getHighestXRefObjectNumber() = 42}} while the newest trailer
> of the file says
> {{{}/Size 44{}}}, so {{addPage}} plus {{saveIncremental}} writes the new page
> as object 43 – the number
> revision 4's cross-reference stream already occupies at offset 76040. PDFBox
> reads the result as
> two pages, pypdf 6.16.2 as one. Identical on all five builds tested. Both
> cases share the seed, and
> the first suggestion above fixes both.
> h2. Relation to PDFBOX-5382
> All three readings look like the write-side consequence of the limitation
> discussed there. In that
> thread, {{SigUtils.checkCrossReferenceTable}} was added in r1898484/r1898485
> and wired into the
> three signing examples in r1898487/r1898488; r1899085/r1899086 then reverted
> the call sites only
> ("revert, but keep a comment for now"), after a discussion in which Michael
> Klink pointed out that
> {{{}COSDocument{}}}'s cross-reference {{Map}} holds no entries for free
> objects, so an object number
> mapped into the free list cannot be told apart from one that is absent
> altogether. The method
> itself was never removed; it is in trunk today, and {{CreateSignature}} still
> carries the line
> {{{}// call SigUtils.checkCrossReferenceTable(document) if Adobe
> complains{}}}.
> That is the same missing information, one step later: because the free
> entries are gone by the time
> the update section is written, the writer has no free list to carry forward
> and no object 6 to
> count towards {{{}/Size{}}}.
> The difference worth drawing out is the direction. PDFBOX-5382 is about
> _noticing_ a gap in a
> document that arrives with one. What is reported here is on the other side:
> given a document where
> object 6 is mapped into the free list, which is a legitimate mapping, an
> incremental save hands
> back one where it is not, because the newest {{/Size}} no longer covers it
> and the newest free-list
> head no longer leads to it.
> Worth stating plainly, since it is the obvious question:
> {{SigUtils.checkCrossReferenceTable}} does
> *not* warn on either file, before or after the save.
> {{COSDocument#getXrefTable()}} comes back with
> keys 1..5 and {{getHighestXRefObjectNumber()}} with 5, so object 6 is absent
> from the model and
> nothing looks missing to the check. A control with an actual hole (object 4
> defined in no section)
> does produce the warning, so this is not a logging artefact. The blind spot
> is the same one Michael
> Klink described in 2022; the point here is only that the writer sits behind
> it too.
> h2. Impact
> A conforming reader is instructed by table 15 to treat object 6 as missing
> after the save, and the
> free list the file used to have is gone. Where it bites hardest is a signed
> document: references
> made by an earlier, signed revision can be made to resolve to different
> objects by a later
> incremental save, without any of the existing bytes being touched.
> On the reader side we can offer a measurement rather than the "may fail
> validation in Acrobat
> Reader" of PDFBOX-5382, though it needs stating precisely, because it is a
> necessary condition and
> not a sufficient one. Bisecting roughly thirty signed variants of one
> document, Acrobat reports an
> earlier signature as "altered or corrupted" as soon as any later revision is
> appended – even one
> whose entire content is a single object nothing references – when the signed
> document has *both* a
> gap in the object numbering *and* annotations written inline in a page's
> {{/Annots}} (where ISO
> 32000-1 table 30 requires indirect references). Repairing either one alone
> left the signature
> invalid; repairing both made it valid. So the gap is half of a reproducible
> Acrobat failure, not
> the whole of it, and the other half is nothing to do with PDFBox. What is
> squarely PDFBox's is that
> an incremental save _introduces_ the condition: given a document whose
> {{/Size}} is 7, it hands
> back one whose newest section says 6.
> A last note for completeness, since it bears on how much of this is worth
> chasing: the document
> that started our investigation broke identically under a signing stack that
> uses no PDFBox at all.
> The defect that caused it was in the source PDF, and we fixed it on our side
> by appending a repair
> revision before the first signature. Nothing in this report is offered as the
> cause of that.
> h2. Attachments
> The PDFs are the inputs and outputs of the two table-based readings, in case
> it is easier to look
> at the bytes than to run anything. They are synthetic, a few hundred bytes
> each, and contain no
> data of any kind.
> * {{free-object-input.pdf}} – the five-object input: {{{}/Size 7{}}}, object
> 6 free, list 0 -> 6 ->
> end, catalog {{{}/Outlines 6 0 R{}}}.
> * {{free-object-after-addpage.pdf}} – the same file after {{addPage}} plus
> {{saveIncremental}} on
> 3.0.8, where {{/Outlines 6 0 R}} now resolves to the new page.
> * {{topfree-input.pdf}} and {{topfree-after-incremental.pdf}} – the
> cross-reference stream pair,
> where {{/Size}} drops from 9 to 8 and {{/Outlines 7 0 R}} ends up resolving
> to the update's own
> {{/XRef}} dictionary.
> * {{IncrementalXrefRepro.java}} – readings 1 and 2, cross-reference table
> input, run twice (with
> and without a reference to the free object).
> * {{XrefReuseRepro.java}} – reading 3, same input, the save adds a
> {{{}PDPage{}}}.
> * {{XrefStreamRepro.java}} – cross-reference stream input whose free number
> is below the highest
> in-use one; only reading 2 shows.
> * {{XrefStreamTopFree.java}} – cross-reference stream input whose free
> numbers are above it; the
> update's own {{/XRef}} stream takes a referenced free number.
> All four run the same way as the first, needing only PDFBox and its runtime
> dependencies – on
> 4.0.0-SNAPSHOT, {{log4j-api}} as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
