[
https://issues.apache.org/jira/browse/TIKA-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas DiPiazza updated TIKA-3835:
------------------------------------
Summary: tika pipes parse cache - avoid re-parsing content that has not
changed (was: parse cache - avoid re-parsing content that has not changed)
> tika pipes parse cache - avoid re-parsing content that has not changed
> ----------------------------------------------------------------------
>
> Key: TIKA-3835
> URL: https://issues.apache.org/jira/browse/TIKA-3835
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 2.2.0
> Reporter: Nicholas DiPiazza
> Priority: Major
>
> Tika pipes should have an optional configuration to archive parsed results.
> So the process would be
> * pipe iterator has the next document: \{lastUpdated,docID}
> * if parse cache is enabled and parse cache contains \{lastUpdated,docID}
> ** Emit to the document to the emit queue and return.
> * Parse document
> * If parse cache is enabled, put into cache key=\{lastUpdated,docID},
> value=\{document,metadata}
> This will drastically improve full crawl times for customers using services
> especially cloud file services with strict rate limits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
