My reply is interleaved below, but there's something important to cover before reading on.

There's clearly a difference in what I mean by de-duplication vs what you're thinking I mean by de-duplication. As far as I can tell you're looking at font substitution and un/re-embedding, where (eg) Helvetica LT Std is replaced with Helvetica Neue Sans, a different version of Helvetica LT Std, the built-in Helvetica derived from Adobe's multi-master fonts, or whatever. The replacement font might not have matching metrics and certainly wouldn't be identical.

That's *not* what I'm talking about. I'm talking about the case where multiple embedded subsets derived from the *exact* *same* *font* exist, each containing partially overlapping sets of glyphs where each glyph is *identical* to those in the other subsets.

This is best illustrated by example. Take three input PDFs that are being placed as images (say, engineering diagrams, advertisments or breakouts in a layout, or whatever), named "1.pdf", "2.pdf" and "3.pdf" that will be written into "out.pdf". For the sake of this example, presume that content in "out.pdf" uses "Arial Regular" for its own text so that font must also be embedded.

       Helvetica Neue Sans subset [a cde  h]
       Utopia Black               [abcd]
       Helvetica Neue Sans subset [abcde   ]
       Helvetica LT Std           [ab def  ijk]
       Helvetica Neue Sans subset [  c efgh]

Desired output is:

       Helvetica Neue Sans subset [abcdefgh]
       Utopia Black               [abcd]
Helvetica LT Std           [ab def  ijk]
       Arial Regular              (whatever the text in out.pdf requires)

Fop and fop-pdf-image currently produce:

       Helvetica Neue Sans subset [a cde  h]
       Helvetica Neue Sans subset [abcde   ]
       Helvetica Neue Sans subset [  c efgh]
       Utopia Black               [abcd]
Helvetica LT Std           [ab def  ijk]
       Arial Regular              (whatever the text in out.pdf requires)

... meaning that there are 3 copes of h.n.s "c" plus 2 copies of "d", "e" and "h" from *identical* fonts (presuming each input had the same version of h.n.s as verified by metrics or for the truly paranoid even glyph data checksums). You appear to think I want to produce:

       Helvetica Neue Sans        [abcdefghijk]
       Utopia Black               [abcd]
       Arial Regular              (whatever the text in out.pdf requires)

or even:

       Arial Regular              (out.pdf glyph usage plus [abcdefghijk])
       Utopia Black               [abcd]

... where Helvetica Neue Sans and Helvetica LT Std are "de-duplicated" despite not being true duplicates of each other, or in the latter case both are replaced with the "equivalent" (approximately) Arial Regular.

That is *not* what I want; that would be completely incorrect to do automatically.

On 03/06/2012 07:08 PM, mehdi houshmand wrote:
Font de-duping is intrinsically a post-process action, you need the
full document, with all fonts, before you can do any font de-duping.
PostScript does this very thing (to a much lesser extent) with the
<optimize-resources>  tag, as a post-process action.
I absolutely disagree that font optimization must be done in a second pass.

Font de-duplication requires knowledge of all the fonts in the document, yes. That doesn't make it necessarily a post-process operation. PDF is a wonderfully non-linear format, and it's trivial to delay writing out fonts until the end of the document. PDF simply doesn't care where the fonts appear in the document. Once you know the last content stream has been written out (say, just before you write the xref tables) you know no more new glyphs will be used and no new fonts will be referenced, so you can write out the fonts you need.

The only operation in PDF that is (almost) forced to be post-process is writing out linearized ("fast web view" or "web optimized") PDF. That's because web-optimized PDF must have a partial xref table and the trailer dictionary near the *start* of the file. It's actually still possible to create linearised pdf by streaming it out in a single pass, but you need to know more in advance about what you'll be writing out so in practice it's much simpler to linearise by post-processing.

Also, the requirements aren't clear here, what is it we want here? Let
me validate that, this shouldn't change the (I guess we can call it)
"canonical" PDF document. By that I mean if you rasterized a PDF
before and after this change they should be identical,
I agree.
When Acrobat does the font de-duping (I don't
remember how much control it gives you, but if there are levels of
de-duping I would have chosen the most aggressive), the documents
aren't identical.
That's because it's actually substituting fonts, replacing one font with another with non-identical metrics. That's not what I want to do, I want to *merge* overlapping subsets of fonts with identical metrics. Since the font dictionary gives the metric information it's practical to do this. If fonts don't have the same metrics, you don't de-dupe them because they're not duplicates.

"Optimizing" a PDF by substituting one font for another is a completely different and much bigger job. Replacement of one font with another non-identical font is a different job that may require rewriting of content streams (for encoding differences), the production of multiple font dictionaries with different encodings to remap different content streams to use one font file, etc. It's hairy and complicated and I don't want to go there.

There are aberrations caused by slight kerning
differences between various verisons of Arial. This may seem trivial
when compared to bloated PDFs, but it looks tacky and lowers the high
standard of documents.
If the metrics don't match, they're not the same font and they don't get merged. The glyph metrics in the font dictionary should be sufficient to handle this.

Having three partial subsets of Arial in a document, each slightly different versions with slightly different metrics, is something I can live with. The problem arises when you have 10 different mostly-overlapping subsets of the *exact* *same* *glyph* *data* from each of those, leaving you with *30* small-ish copies of Arial instead of 3 slightly larger ones.

The other issue is you have subset fonts created by FOP as well as
those imported by the pdf-image-plugin. You'd have to create some
bridge between the image loading framework and the font loading system
*cough* HACK *cough*.
Only if you want to handle de-dupe between fop-loaded fonts and fonts loaded from input PDFs. I don't think that's particularly vital, but it might not be as bad as you think either.

The font matching and subset merging system required for pdf-image to de-dupe fonts would have to track glyph metrics, font names, etc for every font seen, and would need to accumulate information on needed glyphs, etc until the end of output generation just before the xref is written. Fop must maintain used-glyph information as it stands, and already knows glyph metrics, so it's entirely practical for it to report that into the same system. From there, it's not too much of a stretch to see pdf-image recognising that fop is going to embed a font with the same name and metrics already and just merging its required-glyph list with fop's before fop generates the subset.

That's a significantly bigger project, though. Just being able to merge completely redundant glyph subsets where the glyph data and metrics are exactly identical between partially overlapping subsets being loaded by fop-pdf-image would be a nice start.

The best thing about all this it that it's practical to do it progressively.

  Alternatively, just thinking aloud here, if this
was done as a post-process *wink* *wink* *wry smile*...
While it can be done in post-process, I'm really not convinced it's necessary. FOP handles image scaling and resampling - why don't we do that in post-process, too? Just generate a monstrously huge PDF full of uncompressed images, then re-sample later?

The answer seems to be because it's practical to do it in one pass, it's nicer for users, and it works well.

Why does fop have font subsetting support? Subsetting can be done in post-process, all you have to do is read the content streams and determine which glyphs are used, then rewrite the font. It's done in a single pass because it's *much* easier to implement that way, when fop already knows the glyphs it's used. Same deal: it could be done in a post pass, but it isn't because it doesn't make sense to do so.

Font replacement and the substitution of non-identical fonts should be done in post, because it's not practical to do them in a way that's going to be easy, reliable and automatic, nor are there any obvious correct choices. We don't know if the document designer wants to replace their own copy of Helvetica with Adobe's multi-master version. On the other hand, it's pretty bloody obvious that the user won't want 100 copies each of "abcdefg...." glyphs from Helvetica LT Std that are *exactly* *the* *same* when they can have just one copy of each with no effect on document display.

Apologies if I may seem to be argumentative here, it's not my
intention, but I feel this is would be serious scope creep. I see the
pdf-image-plugin as a plugin that treats PDFs as images, nothing more.
If you want to stitch together PDFs, PDFBox is designed just for that.

The trouble is that fop-pdf-image exists because PDFs aren't just images. If they were, it'd be much easier to just rasterise them and import them in raster form.

FWIW, I'm not trying to use fop to "stitch together PDFs" - not in the sense of trying to use it to append, n-up, impose, etc complex PDF documents. I'm using small PDFs that are basically "images" - but represented as a combination of raster, text and bitmap data that should be included in the output document as efficiently as possible and without loss of fidelity. IOW, exactly what fop-pdf-image is for.

Craig Ringer

Reply via email to