[
https://issues.apache.org/jira/browse/TIKA-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114645#comment-18114645
]
Tim Allison commented on TIKA-4891:
-----------------------------------
[~lfoppiano] on the question of how many have pathological tags... it depends
of course. That's part of our task.
Bot reports:
{noformat}
Roughly one in nine. Of the 1457 tagged documents, 1303 (89%) have a tree that
describes the document: no gate hits, under 5% of text outside the tree,
under 15% marked artifact. The rest fall into overlapping buckets:
┌─────────────────────────────────────────────┬──────┬────────────────────────────────────────────────────────────────────────────────────┐
│ bucket │ docs │
what it looks like │
├─────────────────────────────────────────────┼──────┼────────────────────────────────────────────────────────────────────────────────────┤
│ tree present, no page has tagged text │ 33 │ a StructTreeRoot with
nothing wired to content; scans, exports that lost the MCIDs │
├─────────────────────────────────────────────┼──────┼────────────────────────────────────────────────────────────────────────────────────┤
│ at least one page rejected by the AUTO gate │ 77 │ of these only 8 have a
majority of pages rejected │
├─────────────────────────────────────────────┼──────┼────────────────────────────────────────────────────────────────────────────────────┤
│ over 30% of text marked /Artifact │ 17 │ body text tagged as
pagination; worst are Word 2010 to 2019 files at 40 to 60% │
├─────────────────────────────────────────────┼──────┼────────────────────────────────────────────────────────────────────────────────────┤
│ over 20% of text outside the tree │ 2 │ 3003017 (Word 2013,
later modified) at 54% │
└─────────────────────────────────────────────┴──────┴────────────────────────────────────────────────────────────────────────────────────┘
Counting a document once if it hits any bucket gives about 130 documents, so
the pathology rate is around 9%, and most of it is partial: the gate and the
artifact and untagged divs handle it page by page, which is why the text
numbers stay at dice 1.000. By producer the rate is flat, roughly 5% of Word
and 6% of Adobe PDF Library documents hit the gate, and no producer is
pathological as a class. Distiller
is the only one where a meaningful share, 3 of 41, have a tree with nothing
tagged, which fits print-driver output that inherits a tree from an earlier
version. Two caveats on the number. Dice cannot see reading-order damage, so
a tree that scrambles block order counts as clean here. And the artifact share
is a
proxy: a 40% artifact page could be a legitimate table of contents marked as
pagination, so the 17 need eyes before they become a gate rule.
{noformat}
> Improve handling of PDF/UA structural tags/marked content
> ---------------------------------------------------------
>
> Key: TIKA-4891
> URL: https://issues.apache.org/jira/browse/TIKA-4891
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
> Attachments: 0001869.pdf
>
>
> PDF/UA includes structural markup. We hacked out a standalone handler for
> this back in 1.x but haven't touched it in years.
>
> We should modernize our handling of structural tags and eventually consider
> turning that on by default. That decision will be based on evaluation on
> 1000s of PDFs. This is not a default switch to be taken lightly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
