On 14/09/2016 09:05, Gupta, Rajiv wrote:
I'm creating indexes on multiple subfolders under one parent folder.
Indexes are created on multiple folders since files are getting created in
parallel and I want to avoid segment locking between multiple indexers.
I did profiling using Devel::NYTProf<https://metacpan.org/pod/Devel::NYTProf>
and found two places where the maximum time was taken:
1. While scanning the directory. (This I will try to solve by generating a
list of directories while the application is generating the indexes).
2. When creating the searchers using Lucy::Search::IndexSearcher. This takes
maximum time when running in loop for all indexed directories.
It sounds like you're working with an excessively large number of indices.
Maybe you should simply rethink your approach and use a single index? If
you're concerned about locking maybe a separate indexing process with some
kind of notification mechanism would help?