[
https://issues.apache.org/jira/browse/TIKA-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas DiPiazza updated TIKA-3835:
------------------------------------
Description:
Tika pipes should have an optional configuration to archive parsed results.
These archived results can be returned in the case that the same exact version
of a document had already been parsed previously, pull the parsed output from a
"parse cache" instead of repeating the fetch+parse.
In other words, skip the fetch+parse if you did it previously.
Benefits of this:
* When the tika pipe fetcher is using a cloud service, documents are rate
limited heavily. So if you manage to get a document and parse it, storing it
for future use is very important.
* Multi tier environments can be populated faster. Example: You are pulling
data from an app in dev, staging and production. When you run the tika pipe
job, it will parse each document 1 time. All the other environments can now
re-use the parsed output - saving days of run time (in my case).
** In other words, "full crawls" for your initial tika index on duplicate
environments is reduced to cache lookups.
So the process would be
* pipe iterator has the next document: \{lastUpdated,docID}
** pipe iterator documents have an optional field: *cache* _boolean -_
default=true. If cache=false, will not cache this doc.
* if parse cache is enabled and parse cache contains \{lastUpdated,docID}
** Get \{lastUpdated,docID} document from the cache and push to the emit queue
and return.
* Parse document
* If parse cache is enabled, put into cache key=\{lastUpdated,docID},
value=\{document,metadata}
The cache would need to be disk or network based storage because of the storage
size. In-memory cache would not be feasible.
The parser cache should be based on an interface so that the user can use
several varieties of implementations such as:
* File cache
* S3 implementation cache
* Others..
was:
Tika pipes should have an optional configuration to archive parsed results.
These archived results can be returned in the case that the same exact version
of a document had already been parsed previously, pull the parsed output from a
"parse cache" instead of repeating the fetch+parse.
In other words, skip the fetch+parse if you did it previously.
Benefits of this:
* When the tika pipe fetcher is using a cloud service, documents are rate
limited heavily. So if you manage to get a document and parse it, storing it
for future use is very important.
* Multi tier environments can be populated faster. Example: You are pulling
data from an app in dev, staging and production. When you run the tika pipe
job, it will parse each document 1 time. All the other environments can now
re-use the parsed output - saving days of run time (in my case).
** In other words, "full crawls" for your initial tika index on duplicate
environments is reduced to cache lookups.
So the process would be
* pipe iterator has the next document: \{lastUpdated,docID}
* if parse cache is enabled and parse cache contains \{lastUpdated,docID}
** Get \{lastUpdated,docID} document from the cache and push to the emit queue
and return.
* Parse document
* If parse cache is enabled, put into cache key=\{lastUpdated,docID},
value=\{document,metadata}
The cache would need to be disk or network based storage because of the storage
size. In-memory cache would not be feasible.
The parser cache should be based on an interface so that the user can use
several varieties of implementations such as:
* File cache
* S3 implementation cache
* Others..
> tika pipes parse cache - avoid re-parsing content that has not changed
> ----------------------------------------------------------------------
>
> Key: TIKA-3835
> URL: https://issues.apache.org/jira/browse/TIKA-3835
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 2.2.0
> Reporter: Nicholas DiPiazza
> Priority: Major
>
> Tika pipes should have an optional configuration to archive parsed results.
> These archived results can be returned in the case that the same exact
> version of a document had already been parsed previously, pull the parsed
> output from a "parse cache" instead of repeating the fetch+parse.
> In other words, skip the fetch+parse if you did it previously.
> Benefits of this:
> * When the tika pipe fetcher is using a cloud service, documents are rate
> limited heavily. So if you manage to get a document and parse it, storing it
> for future use is very important.
> * Multi tier environments can be populated faster. Example: You are pulling
> data from an app in dev, staging and production. When you run the tika pipe
> job, it will parse each document 1 time. All the other environments can now
> re-use the parsed output - saving days of run time (in my case).
> ** In other words, "full crawls" for your initial tika index on duplicate
> environments is reduced to cache lookups.
> So the process would be
> * pipe iterator has the next document: \{lastUpdated,docID}
> ** pipe iterator documents have an optional field: *cache* _boolean -_
> default=true. If cache=false, will not cache this doc.
> * if parse cache is enabled and parse cache contains \{lastUpdated,docID}
> ** Get \{lastUpdated,docID} document from the cache and push to the emit
> queue and return.
> * Parse document
> * If parse cache is enabled, put into cache key=\{lastUpdated,docID},
> value=\{document,metadata}
> The cache would need to be disk or network based storage because of the
> storage size. In-memory cache would not be feasible.
> The parser cache should be based on an interface so that the user can use
> several varieties of implementations such as:
> * File cache
> * S3 implementation cache
> * Others..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
