[
https://issues.apache.org/jira/browse/TIKA-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048978#comment-15048978
]
Tim Allison commented on TIKA-1599:
-----------------------------------
bq. So I'm not seeing how the parser that CommonCrawl uses would factor into
that.
Right, agree. I was just seeing if they had also picked up JSoup or were using
something else.
bq. But I also don't have a good suggestion for how to automatically evaluate
the TagSoup vs. JSoup parse-off, other than simple measures like how many
failed, or maybe the amount of text extracted?
Y, agreed. For PDFBOX-3058, and as part of TIKA-1330, I have some alpha-level
eval code that will do these comparisons. Following [~tilman]'s
recommendation, I added counts for "common English" words.
> Switch from TagSoup to JSoup
> ----------------------------
>
> Key: TIKA-1599
> URL: https://issues.apache.org/jira/browse/TIKA-1599
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.7, 1.8
> Reporter: Ken Krugler
> Assignee: Ken Krugler
> Priority: Minor
>
> There are several Tika issues related to how TagSoup cleans up HTML
> ([TIKA-381], [TIKA-985], maybe [TIKA-715]), but TagSoup doesn't seem to be
> under active development.
> On the other hand I know of several projects that are now using
> [JSoup|https://github.com/jhy/jsoup], which is an active project (albeit only
> one main contributor) under the MIT license.
> I haven't looked into how hard it would be to switch this dependency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
