Thanks for your reply! The another constraint with the single index approach is
our index locations are dynamic and search happened through APIs which
construct the index location based on certain input parameters.
However, I did modification in code to do fork process to complete the actions.
In that process Searcher object gets created and destroyed and I get all search
hits in an array that I process later. I did not use polysercher. This
increases the speed.
From: Nick Wellnhofer [mailto:wellnho...@aevum.de]
Sent: Friday, September 16, 2016 3:51 PM
Subject: Re: [lucy-user] Speed up Search with Lucy::Search::IndexSearcher and
Lucy::Search::PolySearcher from multiple index folders
On 14/09/2016 09:05, Gupta, Rajiv wrote:
> I'm creating indexes on multiple subfolders under one parent folder.
> Indexes are created on multiple folders since files are getting created in
> parallel and I want to avoid segment locking between multiple indexers.
> I did profiling using Devel::NYTProf<https://metacpan.org/pod/Devel::NYTProf>
> and found two places where the maximum time was taken:
> 1. While scanning the directory. (This I will try to solve by generating a
> list of directories while the application is generating the indexes).
> 2. When creating the searchers using Lucy::Search::IndexSearcher. This
> takes maximum time when running in loop for all indexed directories.
It sounds like you're working with an excessively large number of indices.
Maybe you should simply rethink your approach and use a single index? If you're
concerned about locking maybe a separate indexing process with some kind of
notification mechanism would help?