Author: chetanm
Date: Mon Jul 17 06:19:05 2017
New Revision: 1802096
URL: http://svn.apache.org/viewvc?rev=1802096&view=rev
Log:
OAK-6370 - Improve documentation for text pre-extraction
Modified:
jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/indexing.md
jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/lucene.md
jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/pre-extract-text.md
Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/indexing.md
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/indexing.md?rev=1802096&r1=1802095&r2=1802096&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/indexing.md (original)
+++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/indexing.md Mon Jul 17
06:19:05 2017
@@ -39,6 +39,8 @@
* [Cluster Setup](#nrt-indexing-cluster-setup)
* [Configuration](#nrt-indexing-config)
* [Reindexing](#reindexing)
+ * [Reducing reindexing times](#reduce-reindexing-times)
+ * [How to Abort Reindexing](#abort-reindex)
## <a name="overview"></a> Overview
@@ -512,7 +514,16 @@ Once reindexing starts, the following lo
Once reindexing is complete, the `reindex` flag is set to `false`
automatically.
-### How to Abort Reindexing
+### <a name="reduce-reindexing-times"></a> Reducing reindexing times
+
+If the index being reindexed has full text extraction configured then
reindexing can take long time as most of the
+time is spent in text extraction.
+For such cases its recommended to use text [pre-extraction
support](pre-extract-text.html).
+The text pre-extraction can be done before starting the actual reindexing.
This would then ensure that during reindexing
+time is not spent in performing text extraction and hence the actual time
taken for reindexing such an index gets reduced
+considerably.
+
+### <a name="abort-reindex"></a> How to Abort Reindexing
Building an index can be slow. It can be aborted (stopped before it is
finished),
for example if you detect there is an error in the index definition.
Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/lucene.md
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/lucene.md?rev=1802096&r1=1802095&r2=1802096&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/lucene.md (original)
+++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/lucene.md Mon Jul 17
06:19:05 2017
@@ -1120,52 +1120,7 @@ From the Luke UI shown you can access va
### <a name="text-extraction"></a>Pre-Extracting Text from Binaries
-`@since Oak 1.0.18, 1.2.3`
-
-Lucene indexing is performed in a single threaded mode. Extracting text from
-binaries is an expensive operation and slows down the indexing rate
considerably.
-For incremental indexing this mostly works fine but if performing a reindex
-or creating the index for the first time after migration then it increases the
-indexing time considerably.
-
-To speed up the Lucene indexing for such cases i.e. reindexing, we can
decouple
-the text extraction from actual indexing.
-
-1. Extract and store the extracted text from binaries via [oak-run
tool][oak-run-tika]
-2. Configure a `PreExtractedTextProvider` which can lookup extracted text and
- thus avoid text extraction at time of actual indexing
-
-Below are details around steps required for making using of this feature
-
-1. Generate the csv file containing binary file details
-
- java -cp tika-app-1.8.jar:oak-run.jar \
- org.apache.jackrabbit.oak.run.Main tika \
- --fds-path /path/to/datastore \
- --nodestore /path/to/segmentstore --data-file dump.csv generate
-
-2. Extract the text
-
- java -cp tika-app-1.8.jar:oak-run.jar \
- org.apache.jackrabbit.oak.run.Main tika \
- --data-file binary-stats.csv \
- --store-path ./store
- --fds-path /path/to/datastore extract
-
-3. Configure the `PreExtractedTextProvider` - Once the extraction is
performed
- configure a `PreExtractedTextProvider` within the application such that
Lucene
- indexer can make use of that to lookup extracted text.
-
- For this look for OSGi config for `Apache Jackrabbit Oak DataStore
PreExtractedTextProvider`
-
- 
-
-Once `PreExtractedTextProvider` is configured then upon reindexing Lucene
-indexer would make use of it to check if text needs to be extracted or not.
Check
-`TextExtractionStatsMBean` for various statistics around text extraction and
also
-to validate if `PreExtractedTextProvider` is being used.
-
-For more details on this feature refer to [OAK-2892][OAK-2892]
+Refer to [pre-extraction via oak-run](pre-extract-text.html).
### <a name="advanced-search-features"></a>Advanced search features
Modified:
jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/pre-extract-text.md
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/pre-extract-text.md?rev=1802096&r1=1802095&r2=1802096&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/pre-extract-text.md
(original)
+++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/query/pre-extract-text.md
Mon Jul 17 06:19:05 2017
@@ -39,7 +39,7 @@ binary properties.
Download following jars
-* oak-run 1.7.2
+* oak-run 1.7.4
Refer to [oak-run setup](../features/oak-run-nodestore-connection-options.md)
for details about connecting to different
types of NodeStore. Example below assume a setup consisting of
SegmentNodeStore and FileDataStore. Depending on setup
@@ -85,7 +85,7 @@ By default it scans whole repository. If
Once the csv file is generated we need to perform the text extraction. To do
that we would need to download the
[tika-app](https://tika.apache.org/download.html) jar from Tika downloads. You
should be able to use 1.15 version
-with Oak 1.7.2 jar.
+with Oak 1.7.4 jar.
To perform the text extraction use the `--extract` action
