I could use the number of unmapped unicode chars are in a page to decide whether a PDF should be parsed “normally” or OCR.
Do you think that would be a decent strategy? Also, does anybody know when 1.21 is due? :-) Giovanni On 4 Apr 2019, 13:06 +0200, Tim Allison <[email protected]>, wrote: > And with TIKA-2846 (thanks to Tilman), you will now be able to see how > many unmapped chars there were per page. If there's more than one > page, you'll get a parallel array of ints. These were the results on > your doc: > > 0: pdf:unmappedUnicodeCharsPerPage : 3242 > 0: pdf:charsPerPage : 3242 > > Note, you'll either have to retrieve the Tika Metadata object after > the parse or use the RecursiveParserWrapper (-j /rmeta). These stats > won't show up in the xhtml because they are calculated after the first > bit of content has been written. > > On Tue, Apr 2, 2019 at 4:52 AM Giovanni De Stefano (zxxz) > <[email protected]> wrote: > > > > Hello Tim, Peter, > > > > Thank you for your replies. > > > > It seems indeed that the only solution is to include Tesseract in my > > processing pipeline. > > > > I don’t know if it might be useful to future readers, but I noticed that > > *all* pdf created with PDF24 are subject to this behavior. > > > > I guess this might fall into the “obfuscation” approach some software adopt > > :-( > > > > Cheers, > > > > Giovanni > > On 2 Apr 2019, 04:48 +0200, Peter Murray-Rust <[email protected]>, wrote: > > > > I agree with Tim's analysis. > > > > Many "legacy" fonts (including unfortunately some of those used by LaTeX) > > are not mapped onto Unicode. There are two indications (codepoints and > > names which can often be used to create a partial mapping. I spent a *lot* > > of time doing this manually. For example > > > > > > WARN No Unicode mapping for .notdef (89) in font null > > > > WARN No Unicode mapping for 90 (90) in font null > > <<< > > The first field is the name , the second the codepoint. In your example the > > font (probably) uses codepoints consistently within that particular font, > > e.g. 89 is consistently the same character and different from 90. The names > > *may* differentiate characters. Here is my (handedited) entry for CMSY > > (used by LaTeX for symbols): > > > > <codePoint unicode="U+00B1" name=".notdef" note="PLUS-MINUS SIGN"/> > > > > But this will only work for this particularly font. > > > > If you are only dealing with anglophone alphanumeric from a single > > source/font you can probably work out a table. You are welcome to use mine > > (mainly from scientific / technical publishing) Beyond that OCR/Tesseract > > may help. (I use it a lot). However maths and non-ISO-LATIN is problematic. > > For example distinguishing between the many types of dash/minus/underline > > depend on having a system trained on these. Relative heights and size are a > > major problem > > > > In general, typesetters and their software are only concerned with the > > visual display and frequently use illiteracies (e.g. "=" + backspace + "/" > > for "not-equals". Anyone having work typeset in PDF should insist that a > > Unicode font is used. Better still avoid PDF. > > > > > > > > -- > > Peter Murray-Rust > > Reader Emeritus in Molecular Informatics > > Unilever Centre, Dept. Of Chemistry > > University of Cambridge > > CB2 1EW, UK > > +44-1223-763069
