[
https://issues.apache.org/jira/browse/TIKA-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768010#comment-15768010
]
Tim Allison commented on TIKA-1946:
-----------------------------------
My initial thought is wrong. CommonCrawl docs are often truncated at 1MB.
However, the docs with the EOF exceptions include quite a few from govdocs1,
which should be whole files, and many files from CommonCrawl are not near the
1MB threshold.
{noformat}
select count(1) from profiles;
select count(1) from parse_exceptions;
{noformat}
yields 535 files with 265 exceptions
{noformat}
select file_path, c.length
from parse_exceptions e
join profiles p on e.id=p.id
join containers c on c.container_id=p.container_id
where orig_stack_trace not like '%QPWTextExtractor.java:179%'
order by c.length desc
{noformat}
yields the 248 files with the EOF.
I'm going to take a look at some now.
> Add mime detection and parser for WordPerfect
> ---------------------------------------------
>
> Key: TIKA-1946
> URL: https://issues.apache.org/jira/browse/TIKA-1946
> Project: Tika
> Issue Type: Improvement
> Components: mime, parser
> Reporter: Nick C
> Fix For: 2.0, 1.15
>
>
> I noticed some code on github for parsing WordPerfect files
> (https://github.com/Norconex/importer) Also looks like the author
> [~pascal.essiembre] has contributed to Tika before
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
