[
https://issues.apache.org/jira/browse/TIKA-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004592#comment-14004592
]
Tim Allison edited comment on TIKA-1302 at 5/21/14 11:44 AM:
-------------------------------------------------------------
[~jnioche], very cool corpus. My dream would be to run Tika via hadoop against
a corpus that big, diverse and noisy whenever there's a commit (or better yet,
let developers upload their mods as a tika-server, go for a cup of coffee and
come back to see the results). For initial steps, govdocs1 seems like a great
start...perhaps we could include a random sample from CC downloaded to Apache
servers if that is consistent with both Apache and CC licenses?
[~chrismattmann], thank you for pointing out [~gostep]'s work!
[~gostep], please post a link to your work to this issue when it is published.
Are there any evaluation components that you'd like to contribute? Do you
think there would be a way to share your datasets? And, finally, I'm not sure
what you mean by all of the metadata is the same. I am just getting started
with the govdocs corpus...when I open up two different pdf files, the have
different pdf versions, different authors, different producers.
was (Author: talli...@mitre.org):
[~jnioche], very cool corpus. My dream would be to run Tika via hadoop against
a corpus that big, diverse and noisy whenever there's a commit (or better yet,
let developers upload their mods as a tika-server, go for a cup of coffee and
come back to see the results). For initial steps, govdocs1 seems like a decent
start...perhaps we could include a random sample from CC downloaded to Apache
servers if that is consistent with both Apache and CC licenses?
[~chrismattmann], thank you for pointing out [~gostep]'s work!
[~gostep], please post a link to your work to this issue when it is published.
Are there any evaluation components that you'd like to contribute? Do you
think there would be a way to share your datasets?
> Let's run Tika against a large batch of docs nightly
> ----------------------------------------------------
>
> Key: TIKA-1302
> URL: https://issues.apache.org/jira/browse/TIKA-1302
> Project: Tika
> Issue Type: Improvement
> Reporter: Tim Allison
>
> Many thanks to [~lewismc] for TIKA-1301! Once we get nightly builds up and
> running again, it might be fun to run Tika regularly against a large set of
> docs and report metrics.
> One excellent candidate corpus is govdocs1:
> http://digitalcorpora.org/corpora/files.
> Any other candidate corpora?
> [~willp-bl], have anything handy you'd like to contribute?
> [http://www.openplanetsfoundation.org/blogs/2014-03-21-tika-ride-characterising-web-content-nanite]
> ;)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
