The numbers I suggested were actually from Tim a week or 2 ago. Of course, the idea is to allow the user to adjust them, so if the default numbers don't work for a particular scenario, they can be changed.
It sounds like the best solution for the best vs fast discussion is to just make it an option and let the user decide I know that Tesseract has something similar with their best and fast models Peter Kronenberg | SENIOR AI ANALYTIC ENGINEER C: 703.887.5623 4303 W. 119th St., Leawood, KS 66209 WWW.TORCH.AI -----Original Message----- From: Nick Burch <[email protected]> Sent: Wednesday, April 14, 2021 10:02 AM To: [email protected] Subject: RE: Parsing PDF file - setting threshold of unmapped characters On Wed, 14 Apr 2021, Peter Kronenberg wrote: > Anyone have any thoughts on this? I think both an absolute and a percentage would be good, but I don't have enough experience to comment on your suggested numbers for those two thresholds, sorry! Your idea on best vs fast touches on much older discussions on what to do when we have multiple possible parsers available. For example, an external program that's slow but official and very reliable, or a java library that's quick but misses some edge cases. We never did manage to reach a conclusion on that though... Nick > Subject: RE: Parsing PDF file - setting threshold of unmapped > characters > > I’ve been thinking about this and I think it would be a good idea to > change the comparison of unmapped characters to a percentage. For > example, you suggested > > > unmappedUnicodeCharsPerPage > 10 && percentUnmappedUnicodeChars > 0.2 or > something? > > > > The percentage could be configurable. > > > > Another thought I had to was to have an AUTO_BEST and AUTO_FAST. AUTO_FAST > would have a higher threshold of Unmapped Characters, so that in most cases, > it would just extract text and not use OCR. The performance overhead of OCR > is very high for not a lot of benefit given that it extracts 99% of the text. > > AUTO_BEST would have a lower threshold before OCR is triggered. > > > > Or just keep AUTO and allow the threshold to be configured, either by > number of characters or percentage. The only downside to this is that > the user would have to understand it a little more to be able to set > the threshold properly, instead of AUTO just working magically > > > > What do you thin/ > > > Peter Kronenberg | Senior AI Analytic ENGINEER > C: 703.887.5623 > [Torch > AI]<https://us-east-2.protection.sophos.com?d=torch.ai&u=aHR0cDovL3d3d > y50b3JjaC5haS8=&i=NjAwMDY2MjNjNzQ1NDY0ODkyYTNmNTg3&t=dHRDUUJralFuRnRCU > 2tvcmRLNUUycFdBV2RmazdTZU0zZUZVM21GSXhobz0=&h=a835294803a7452c9dd75b10 > 08ffebe7> > 4303 W. 119th St., Leawood, KS 66209 > https://us-east-2.protection.sophos.com?d=torch.ai&u=d3d3LlRPUkNILkFJ&; > i=NjAwMDY2MjNjNzQ1NDY0ODkyYTNmNTg3&t=Nm1Pd1NUek94MUNheHppZ0RpaUZ4RVlYe > mhyTlhSa1M3Ly9FUFhXeDc5dz0=&h=a835294803a7452c9dd75b1008ffebe7<https:/ > /us-east-2.protection.sophos.com?d=torch.ai&u=aHR0cDovL3d3dy50b3JjaC5h > aS8=&i=NjAwMDY2MjNjNzQ1NDY0ODkyYTNmNTg3&t=dHRDUUJralFuRnRCU2tvcmRLNUUy > cFdBV2RmazdTZU0zZUZVM21GSXhobz0=&h=a835294803a7452c9dd75b1008ffebe7> > > > From: Tim Allison <[email protected]<mailto:[email protected]>> > Sent: Monday, April 5, 2021 1:49 PM > To: Peter Kronenberg > <[email protected]<mailto:[email protected]>> > Cc: [email protected]<mailto:[email protected]> > Subject: Re: Parsing PDF file > > Y. You understand perfectly! > > I want "auto" to be the best it can be and most generally applicable across > use cases. For users who want high performance/better control, you might > parse the PDF first with NO_OCR, and then make the determination on which > pages to run OCR based on those statistics pulled out in the first parse. > Another key statistic in the decision would be the out of vocabulary > measurement that you can get with an integration with tika-eval. > > So, in short, if there are clear, provable, general improvements to AUTO, we > should make them. If you want more refined control, let us know if the > current metadata can be improved to help you develop your application for > your use cases. > > On Mon, Apr 5, 2021 at 1:06 PM Peter Kronenberg > <[email protected]<mailto:[email protected]>> wrote: > You’re right that OCRing would result in slightly more accurate results in > this case. But the performance penalty is high. Wondering if there is some > intermediate option. > > I think I understand now why you are separately looking for unmapped > characters as well as total characters. If total characters is low, > we assume the page is an image and OCR. But if unmapped characters is > high, it might still be straight text, but the unmapped characters > will essentially result in unreadable characters > > From: Tim Allison <[email protected]<mailto:[email protected]>> > Sent: Monday, April 5, 2021 11:39 AM > To: Peter Kronenberg > <[email protected]<mailto:[email protected]>> > Cc: [email protected]<mailto:[email protected]> > Subject: Re: Parsing PDF file > > As for the metadata, we should add unique. Given that multiple parsers can > hit the same file, we need to record all of them (in this case: default, pdf, > tesseract). > > As for tweaking the settings...I'm not sure as I look at the extracted text > more. There are quite a few bad ligatures /unmapped unicode chars which > would render search for, e.g. "efficient", "affairs" useless. > > On Mon, Apr 5, 2021 at 10:40 AM Peter Kronenberg > <[email protected]<mailto:[email protected]>> wrote: > Yes, I think tweaking the criteria for Auto is a good idea. > And if the parser list was a Set, that would automatically eliminate > dups > > From: Tim Allison <[email protected]<mailto:[email protected]>> > Sent: Monday, April 5, 2021 10:15 AM > To: [email protected]<mailto:[email protected]> > Subject: Fwd: Parsing PDF file > > It looks like the ligatures don't have unicode mappings: > > "Division of Monetary A???airs" > > > if (totalCharsPerPage < 10 || unmappedUnicodeCharsPerPage > 10) > > The issue is that this file has > 10 unmapped unicode chars per page. > > We could change the heuristic to unmappedUnicodeCharsPerPage > 10 && > percentUnmappedUnicodeChars > 0.2 or something? > > We should also probably check to see if a parser is in the parsed by list > before re-adding it? > > > > 0: pdf:charsPerPage : 1579 > 0: pdf:charsPerPage : 1891 > 0: pdf:charsPerPage : 2283 > 0: pdf:charsPerPage : 2224 > 0: pdf:charsPerPage : 1619 > 0: pdf:charsPerPage : 2177 > 0: pdf:charsPerPage : 1626 > 0: pdf:charsPerPage : 1313 > 0: pdf:charsPerPage : 1652 > 0: pdf:charsPerPage : 1493 > 0: pdf:charsPerPage : 1136 > 0: pdf:charsPerPage : 1477 > 0: pdf:charsPerPage : 1264 > 0: pdf:charsPerPage : 1994 > 0: pdf:charsPerPage : 2062 > 0: pdf:charsPerPage : 1756 > 0: pdf:charsPerPage : 2007 > 0: pdf:charsPerPage : 2202 > 0: pdf:charsPerPage : 2105 > 0: pdf:charsPerPage : 2106 > 0: pdf:charsPerPage : 1895 > 0: pdf:charsPerPage : 1978 > 0: pdf:charsPerPage : 1826 > 0: pdf:charsPerPage : 1742 > 0: pdf:charsPerPage : 2073 > 0: pdf:charsPerPage : 1882 > 0: pdf:charsPerPage : 1497 > 0: pdf:charsPerPage : 282 > 0: pdf:charsPerPage : 606 > 0: pdf:charsPerPage : 948 > 0: pdf:charsPerPage : 418 > 0: pdf:charsPerPage : 266 > 0: pdf:charsPerPage : 830 > 0: pdf:charsPerPage : 259 > 0: pdf:charsPerPage : 716 > 0: pdf:charsPerPage : 961 > 0: pdf:charsPerPage : 1325 > 0: pdf:charsPerPage : 1478 > 0: pdf:docinfo:creator_tool : dvips 5.83 (MiKTeX 1.11d) Copyright 1998 > Radical Eye Software > 0: pdf:docinfo:producer : Acrobat Distiller 3.01 for Windows > 0: pdf:docinfo:title : Inel4shannon.dvi > 0: pdf:encrypted : false > 0: pdf:hasMarkedContent : false > 0: pdf:hasXFA : false > 0: pdf:hasXMP : false > 0: pdf:producer : Acrobat Distiller 3.01 for Windows > 0: pdf:unmappedUnicodeCharsPerPage : 109 > 0: pdf:unmappedUnicodeCharsPerPage : 120 > 0: pdf:unmappedUnicodeCharsPerPage : 113 > 0: pdf:unmappedUnicodeCharsPerPage : 120 > 0: pdf:unmappedUnicodeCharsPerPage : 94 > 0: pdf:unmappedUnicodeCharsPerPage : 112 > 0: pdf:unmappedUnicodeCharsPerPage : 178 > 0: pdf:unmappedUnicodeCharsPerPage : 74 > 0: pdf:unmappedUnicodeCharsPerPage : 132 > 0: pdf:unmappedUnicodeCharsPerPage : 189 > 0: pdf:unmappedUnicodeCharsPerPage : 165 > 0: pdf:unmappedUnicodeCharsPerPage : 145 > 0: pdf:unmappedUnicodeCharsPerPage : 132 > 0: pdf:unmappedUnicodeCharsPerPage : 186 > 0: pdf:unmappedUnicodeCharsPerPage : 162 > 0: pdf:unmappedUnicodeCharsPerPage : 145 > 0: pdf:unmappedUnicodeCharsPerPage : 119 > 0: pdf:unmappedUnicodeCharsPerPage : 138 > 0: pdf:unmappedUnicodeCharsPerPage : 115 > 0: pdf:unmappedUnicodeCharsPerPage : 99 > 0: pdf:unmappedUnicodeCharsPerPage : 107 > 0: pdf:unmappedUnicodeCharsPerPage : 108 > 0: pdf:unmappedUnicodeCharsPerPage : 116 > 0: pdf:unmappedUnicodeCharsPerPage : 174 > 0: pdf:unmappedUnicodeCharsPerPage : 138 > 0: pdf:unmappedUnicodeCharsPerPage : 101 > 0: pdf:unmappedUnicodeCharsPerPage : 61 > 0: pdf:unmappedUnicodeCharsPerPage : 90 > 0: pdf:unmappedUnicodeCharsPerPage : 239 > 0: pdf:unmappedUnicodeCharsPerPage : 614 > 0: pdf:unmappedUnicodeCharsPerPage : 216 > 0: pdf:unmappedUnicodeCharsPerPage : 101 > 0: pdf:unmappedUnicodeCharsPerPage : 502 > 0: pdf:unmappedUnicodeCharsPerPage : 103 > 0: pdf:unmappedUnicodeCharsPerPage : 427 > 0: pdf:unmappedUnicodeCharsPerPage : 629 > 0: pdf:unmappedUnicodeCharsPerPage : 347 > 0: pdf:unmappedUnicodeCharsPerPage : 327 > > On Mon, Apr 5, 2021 at 10:00 AM Peter Kronenberg > <[email protected]<mailto:[email protected]>> wrote: > Yes, 2.x > > From: Tim Allison <[email protected]<mailto:[email protected]>> > Sent: Monday, April 5, 2021 9:54 AM > To: [email protected]<mailto:[email protected]> > Subject: Re: Parsing PDF file > > Tika 2.x? Looking now. > > On Mon, Apr 5, 2021 at 8:55 AM Peter Kronenberg > <[email protected]<mailto:[email protected]>> wrote: > If I use OCRStrategy=no_ocr, the time it takes to process is orders of > magnitude faster and I don’t see the calls to OCRParser (obviously) Why is it > taking so long with auto? If the page does not meet the criteria for OCR, > then it shouldn’t be calling OCR at all, right? > > "X-TIKA:Parsed-By": "[org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.pdf.PDFParser]", > > > From: Peter Kronenberg > <[email protected]<mailto:[email protected]>> > Sent: Monday, April 5, 2021 8:48 AM > To: [email protected]<mailto:[email protected]> > Subject: RE: {EXTERNAL}Parsing PDF file > > This email was sent from outside your organisation, yet is displaying the > name of someone from your organisation. This often happens in phishing > attempts. Please only interact with this email if you know its source and > that the content is safe. > > Correction: I see one instance of PDFParser at the beginning, but why does it > then alternate between OCRParser and CompositeParser? > > From: Peter Kronenberg > <[email protected]<mailto:[email protected]>> > Sent: Monday, April 5, 2021 8:41 AM > To: [email protected]<mailto:[email protected]> > Subject: {EXTERNAL}Parsing PDF file > > This email was sent from outside your organisation, yet is displaying the > name of someone from your organisation. This often happens in phishing > attempts. Please only interact with this email if you know its source and > that the content is safe. > > CAUTION: This email originated from outside of the organization. DO NOT click > links or open attachments unless you recognize the sender and know the > content is safe. > Parsing the attached PDF file. It is a text file, not scanned. I’m using > OCR_Strategy=Auto, extractInlineImages=false > > The output contains the following in the metadata. I’m wondering 2 things. > First, why don’t I see PDFParser? > And 2nd, why does it keep calling the TesseractOCRParser? Once it determines > that it is a PDF file, wouldn’t it stick with that? > I’m asking because it seems to take longer to parse than I would > expect and I’m wondering if the OCRParser is adding extra overhead > > > "X-TIKA:Parsed-By":[org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.pdf.PDFParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.CompositeParser, > org.apache.tika.parser.ocr.TesseractOCRParser, > org.apache.tika.parser.Compo siteParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.T esseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parse r.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser, org.apache.tika.parser.CompositeParser, org.apache.tika.parser.ocr.TesseractOCRParser] > > Peter Kronenberg | Senior AI Analytic ENGINEER > C: 703.887.5623 > [Torch > AI]<https://us-east-2.protection.sophos.com/?d=torch.ai&u=aHR0cDovL3d3 > dy50b3JjaC5haS8=&i=NjAwMDY2MjNjNzQ1NDY0ODkyYTNmNTg3&t=dHRDUUJralFuRnRC > U2tvcmRLNUUycFdBV2RmazdTZU0zZUZVM21GSXhobz0=&h=38113a42f3384422af99199 > 9d363a651> > 4303 W. 119th St., Leawood, KS > 66209<https://us-east-2.protection.sophos.com?d=google.com&u=aHR0cHM6L > y93d3cuZ29vZ2xlLmNvbS9tYXBzL3NlYXJjaC80MzAzK1cuKzExOXRoK1N0LiwrTGVhd29 > vZCwrS1MrNjYyMDk_ZW50cnk9Z21haWwmc291cmNlPWc=&i=NjAwMDY2MjNjNzQ1NDY0OD > kyYTNmNTg3&t=N2FQazRValYxZ2cyRHZLcXZnb1AzcTVlQVc0SHJFYXdjMkFPemVSR1M1c > z0=&h=14c17a0e2f574c30b54332f7c4081ca7> > https://us-east-2.protection.sophos.com?d=torch.ai&u=d3d3LlRPUkNILkFJ&; > i=NjAwMDY2MjNjNzQ1NDY0ODkyYTNmNTg3&t=Nm1Pd1NUek94MUNheHppZ0RpaUZ4RVlYe > mhyTlhSa1M3Ly9FUFhXeDc5dz0=&h=a835294803a7452c9dd75b1008ffebe7<https:/ > /us-east-2.protection.sophos.com?d=torch.ai&u=aHR0cDovL3d3dy50b3JjaC5h > aS8=&i=NjAwMDY2MjNjNzQ1NDY0ODkyYTNmNTg3&t=dHRDUUJralFuRnRCU2tvcmRLNUUy > cFdBV2RmazdTZU0zZUZVM21GSXhobz0=&h=38113a42f3384422af991999d363a651> > > >
