[
https://issues.apache.org/jira/browse/TIKA-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048786#comment-15048786
]
Tim Allison commented on TIKA-1599:
-----------------------------------
At least in the 1.x branch, we could leave in both, but configure one or the
other.
I completely agree with your point above about testing on common-crawl.
[~jnioche], do you have examples of where Tika was normalizing/filtering too
much?
I can't remember off the top of my head what CommonCrawl is using for their
text extraction...will take a look.
I hacked out a simple Jsoup wrapper and will add that branch to my github site
by the end of the day. Refactoring the AutoDetectReader so that there is a
standalone AutoEncodingDetector that loads detectors and runs against a stream
is more than I'd like to do, but we can decide on that refactoring later.
> Switch from TagSoup to JSoup
> ----------------------------
>
> Key: TIKA-1599
> URL: https://issues.apache.org/jira/browse/TIKA-1599
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.7, 1.8
> Reporter: Ken Krugler
> Assignee: Ken Krugler
> Priority: Minor
>
> There are several Tika issues related to how TagSoup cleans up HTML
> ([TIKA-381], [TIKA-985], maybe [TIKA-715]), but TagSoup doesn't seem to be
> under active development.
> On the other hand I know of several projects that are now using
> [JSoup|https://github.com/jhy/jsoup], which is an active project (albeit only
> one main contributor) under the MIT license.
> I haven't looked into how hard it would be to switch this dependency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
