Re: Indexing time increase moving from Lucene 8 to 9

2024-04-17 Thread Adrien Grand
Hi Marc,

Nothing jumps to mind as a potential cause for this 2x regression. It would
be interesting to look at a profile.

On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
 wrote:

> Hello,
> I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall build can
> now support Java 11. The quick first step of renaming packages and
> importing the new libraries has gone well.  I'm even seeing a nice
> performance bump in our average query time. I am however seeing a dramatic
> increase in our indexing time.  We are indexing ~3.1 million documents each
> with about 100 attributes used for facet filter, and sorting; no lexical
> text search.  Our indexing time has jumped from ~1k seconds to ~2k
> seconds.  I have yet to profile the individual aspects of how we convert
> our data to records vs time for the index writer to accept the documents.
> I'm curious if other users discovered this for their migrations at some
> point.  Or if there are some changes to defaults that I did not see in the
> migration guide that would account for this?  Looking at the logs I can see
> that as we are indexing the documents we commit every 10 minutes.
> Thank you,
> Marc
>


-- 
Adrien


Indexing time increase moving from Lucene 8 to 9

2024-04-17 Thread Marc Davenport
Hello,
I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall build can
now support Java 11. The quick first step of renaming packages and
importing the new libraries has gone well.  I'm even seeing a nice
performance bump in our average query time. I am however seeing a dramatic
increase in our indexing time.  We are indexing ~3.1 million documents each
with about 100 attributes used for facet filter, and sorting; no lexical
text search.  Our indexing time has jumped from ~1k seconds to ~2k
seconds.  I have yet to profile the individual aspects of how we convert
our data to records vs time for the index writer to accept the documents.
I'm curious if other users discovered this for their migrations at some
point.  Or if there are some changes to defaults that I did not see in the
migration guide that would account for this?  Looking at the logs I can see
that as we are indexing the documents we commit every 10 minutes.
Thank you,
Marc