Re: Indexing time increase moving from Lucene 8 to 9

2024-04-26 Thread Marc Davenport
Hello,
Thanks Matt. I also had run a test dramatically increasing the LRU cache,
but in the end it was still better for our case to run with the previous
cache.  We won't encounter the bug that the switch to LRU cache addresses
(for now).  After returning to the previous cache implementation we
actually saw an improvement in our overall indexing time and some
performance improvements across the board.
Thanks for all the feedback.
Marc

On Wed, Apr 24, 2024 at 9:47 AM Matt Davis  wrote:

> Marc,
>
> We also ran into this problem on updating to Lucene 9.5.  We found it
> sufficient in our use case to just bump up LRU cache in the constructor to
> a high enough value to not pose a performance problem.  The default value
> of 4k was way too low for our use case with millions of unique facet
> values. We also tested with 64k and it was not enough for our use case.  We
> have had no issues with a large value.
>
> taxoWriter = new DirectoryTaxonomyWriter(directory,
> OpenMode.CREATE_OR_APPEND, new LruTaxonomyWriterCache(32 * 1024 * 1024));
>
> Full context here:
>
> https://github.com/zuliaio/zuliasearch/blob/4.1.1/zulia-server/src/main/java/io/zulia/server/index/ShardWriteManager.java#L89
>
> Matt
>
> On Tue, Apr 23, 2024 at 3:56 AM Dawid Weiss  wrote:
>
> > Thanks for the follow-up, Marc. I'm not familiar with this part of the
> code
> > but reading through the original issue that changed this, the rationale
> > was to avoid a memleak from a thread local. The LRU cache has
> > synchronized blocks sprinkled all over it - again, I haven't checked but
> it
> > seems the overhead
> > will be there (regardless of cache size?).
> >
> > This said, it seems like you can use your own cache implementation,
> similar
> > to what you can see in tests - TestDirectoryTaxonomyWriter.java
> > or TestConcurrentFacetedIndexing.java. This would
> > allow you to plug in the previous implementation (or something even more
> > fine-tuned to your needs)?
> >
> > Dawid
> >
> >
> >
> > On Mon, Apr 22, 2024 at 10:29 PM Marc Davenport
> >  wrote:
> >
> > > Hello,
> > > I've done bisect between 9.4.2 and 9.5 and found the PR affecting my
> > > particular set up : https://github.com/apache/lucene/pull/12093
> > > This is the switch from UTF8TaxonomyWriterCache to an
> > > LruTaxonomyWriterCache.   I don't see a way to control the size of this
> > > cache to never expel items and match the previous behavior.
> > > Marc
> > >
> > >
> > > On Fri, Apr 19, 2024 at 4:39 PM Marc Davenport <
> madavenp...@cargurus.com
> > >
> > > wrote:
> > >
> > > > Hello,
> > > > Thanks for the leads. I haven't yet gone as far as doing a git
> bisect,
> > > but
> > > > I have found that the big jump in time is in the call to
> > > > facetsConfig.build(taxonomyWriter, doc);  I made a quick and dirty
> > > > instrumented version of the FacetsConfig class and found that calls
> to
> > > > TaxonomyWriter.add(FacetLabel) are significantly slower for me.
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/lucene/blob/releases/lucene/9.5.0/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L383
> > > >
> > > > I don't know what is special about my documents that I would be
> seeing
> > > > this change.  I'm going to start dropping groups of our facets from
> the
> > > > documents and seeing if there is some threshold that I'm hitting.
> I'll
> > > > probably start with our hierarchies which are not particularly large,
> > but
> > > > are the most suspect.
> > > >
> > > > Thanks for any input,
> > > > Marc
> > > >
> > > > 9.4.2
> > > > Time(ms) per Document
> > > > facetConfig.build   : 0.9882365
> > > > Taxo Add: 0.8334876
> > > >
> > > > 9.5
> > > > facetConfig.build   : 11.037549
> > > > Taxo Add: 10.915726
> > > >
> > > > On Fri, Apr 19, 2024 at 2:56 AM Dawid Weiss 
> > > wrote:
> > > >
> > > >> Hi Marc,
> > > >>
> > > >> You could try git bisect lucene repository to pinpoint the commit
> that
> > > >> caused what you're observing. It'll take some time to build but
> it's a
> > > >> logarithmic bisection and you'd know for sure where the problem is.
> > > >>
> > > >> D.
> > > >>
> > > >> On Thu, Apr 18, 2024 at 11:16 PM Marc Davenport
> > > >>  wrote:
> > > >>
> > > >> > Hi Adrien et al,
> > > >> > I've been doing some investigation today and it looks like
> whatever
> > > the
> > > >> > change is, it happens between 9.4.2 and 9.5.0.
> > > >> > I made a smaller test set up for our code that mocks our documents
> > and
> > > >> just
> > > >> > runs through the indexing portion of our code sending in batches
> of
> > 4k
> > > >> > documents at a time. This way I can run it locally.
> > > >> > 9.4.2: ~1200-2000 documents per second
> > > >> > 9.5.0: ~150-400 documents per second
> > > >> >
> > > >> > I'll continue investigating, but nothing in the release notes
> jumped
> > > >> out to
> > > >> > me.
> > > >> > https://lucene.apache.org/core/9_10_0/changes/Changes.html#v9.5.0
> > > >> >
> > > >> > Sorry I don't have 

Re: Indexing time increase moving from Lucene 8 to 9

2024-04-24 Thread Matt Davis
Marc,

We also ran into this problem on updating to Lucene 9.5.  We found it
sufficient in our use case to just bump up LRU cache in the constructor to
a high enough value to not pose a performance problem.  The default value
of 4k was way too low for our use case with millions of unique facet
values. We also tested with 64k and it was not enough for our use case.  We
have had no issues with a large value.

taxoWriter = new DirectoryTaxonomyWriter(directory,
OpenMode.CREATE_OR_APPEND, new LruTaxonomyWriterCache(32 * 1024 * 1024));

Full context here:
https://github.com/zuliaio/zuliasearch/blob/4.1.1/zulia-server/src/main/java/io/zulia/server/index/ShardWriteManager.java#L89

Matt

On Tue, Apr 23, 2024 at 3:56 AM Dawid Weiss  wrote:

> Thanks for the follow-up, Marc. I'm not familiar with this part of the code
> but reading through the original issue that changed this, the rationale
> was to avoid a memleak from a thread local. The LRU cache has
> synchronized blocks sprinkled all over it - again, I haven't checked but it
> seems the overhead
> will be there (regardless of cache size?).
>
> This said, it seems like you can use your own cache implementation, similar
> to what you can see in tests - TestDirectoryTaxonomyWriter.java
> or TestConcurrentFacetedIndexing.java. This would
> allow you to plug in the previous implementation (or something even more
> fine-tuned to your needs)?
>
> Dawid
>
>
>
> On Mon, Apr 22, 2024 at 10:29 PM Marc Davenport
>  wrote:
>
> > Hello,
> > I've done bisect between 9.4.2 and 9.5 and found the PR affecting my
> > particular set up : https://github.com/apache/lucene/pull/12093
> > This is the switch from UTF8TaxonomyWriterCache to an
> > LruTaxonomyWriterCache.   I don't see a way to control the size of this
> > cache to never expel items and match the previous behavior.
> > Marc
> >
> >
> > On Fri, Apr 19, 2024 at 4:39 PM Marc Davenport  >
> > wrote:
> >
> > > Hello,
> > > Thanks for the leads. I haven't yet gone as far as doing a git bisect,
> > but
> > > I have found that the big jump in time is in the call to
> > > facetsConfig.build(taxonomyWriter, doc);  I made a quick and dirty
> > > instrumented version of the FacetsConfig class and found that calls to
> > > TaxonomyWriter.add(FacetLabel) are significantly slower for me.
> > >
> > >
> > >
> >
> https://github.com/apache/lucene/blob/releases/lucene/9.5.0/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L383
> > >
> > > I don't know what is special about my documents that I would be seeing
> > > this change.  I'm going to start dropping groups of our facets from the
> > > documents and seeing if there is some threshold that I'm hitting.  I'll
> > > probably start with our hierarchies which are not particularly large,
> but
> > > are the most suspect.
> > >
> > > Thanks for any input,
> > > Marc
> > >
> > > 9.4.2
> > > Time(ms) per Document
> > > facetConfig.build   : 0.9882365
> > > Taxo Add: 0.8334876
> > >
> > > 9.5
> > > facetConfig.build   : 11.037549
> > > Taxo Add: 10.915726
> > >
> > > On Fri, Apr 19, 2024 at 2:56 AM Dawid Weiss 
> > wrote:
> > >
> > >> Hi Marc,
> > >>
> > >> You could try git bisect lucene repository to pinpoint the commit that
> > >> caused what you're observing. It'll take some time to build but it's a
> > >> logarithmic bisection and you'd know for sure where the problem is.
> > >>
> > >> D.
> > >>
> > >> On Thu, Apr 18, 2024 at 11:16 PM Marc Davenport
> > >>  wrote:
> > >>
> > >> > Hi Adrien et al,
> > >> > I've been doing some investigation today and it looks like whatever
> > the
> > >> > change is, it happens between 9.4.2 and 9.5.0.
> > >> > I made a smaller test set up for our code that mocks our documents
> and
> > >> just
> > >> > runs through the indexing portion of our code sending in batches of
> 4k
> > >> > documents at a time. This way I can run it locally.
> > >> > 9.4.2: ~1200-2000 documents per second
> > >> > 9.5.0: ~150-400 documents per second
> > >> >
> > >> > I'll continue investigating, but nothing in the release notes jumped
> > >> out to
> > >> > me.
> > >> > https://lucene.apache.org/core/9_10_0/changes/Changes.html#v9.5.0
> > >> >
> > >> > Sorry I don't have anything more rigorous yet.  I'm doing this
> > >> > investigation in parallel with some other things.
> > >> > But any insight or suggestions on areas to look would be
> appreciated.
> > >> > Thank you,
> > >> > Marc
> > >> >
> > >> > On Wed, Apr 17, 2024 at 4:18 PM Adrien Grand 
> > wrote:
> > >> >
> > >> > > Hi Marc,
> > >> > >
> > >> > > Nothing jumps to mind as a potential cause for this 2x regression.
> > It
> > >> > would
> > >> > > be interesting to look at a profile.
> > >> > >
> > >> > > On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
> > >> > >  wrote:
> > >> > >
> > >> > > > Hello,
> > >> > > > I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our
> overall
> > >> build
> > >> > > can
> > >> > > > now support Java 11. The quick first step of renaming packages

Re: Indexing time increase moving from Lucene 8 to 9

2024-04-23 Thread Dawid Weiss
Thanks for the follow-up, Marc. I'm not familiar with this part of the code
but reading through the original issue that changed this, the rationale
was to avoid a memleak from a thread local. The LRU cache has
synchronized blocks sprinkled all over it - again, I haven't checked but it
seems the overhead
will be there (regardless of cache size?).

This said, it seems like you can use your own cache implementation, similar
to what you can see in tests - TestDirectoryTaxonomyWriter.java
or TestConcurrentFacetedIndexing.java. This would
allow you to plug in the previous implementation (or something even more
fine-tuned to your needs)?

Dawid



On Mon, Apr 22, 2024 at 10:29 PM Marc Davenport
 wrote:

> Hello,
> I've done bisect between 9.4.2 and 9.5 and found the PR affecting my
> particular set up : https://github.com/apache/lucene/pull/12093
> This is the switch from UTF8TaxonomyWriterCache to an
> LruTaxonomyWriterCache.   I don't see a way to control the size of this
> cache to never expel items and match the previous behavior.
> Marc
>
>
> On Fri, Apr 19, 2024 at 4:39 PM Marc Davenport 
> wrote:
>
> > Hello,
> > Thanks for the leads. I haven't yet gone as far as doing a git bisect,
> but
> > I have found that the big jump in time is in the call to
> > facetsConfig.build(taxonomyWriter, doc);  I made a quick and dirty
> > instrumented version of the FacetsConfig class and found that calls to
> > TaxonomyWriter.add(FacetLabel) are significantly slower for me.
> >
> >
> >
> https://github.com/apache/lucene/blob/releases/lucene/9.5.0/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L383
> >
> > I don't know what is special about my documents that I would be seeing
> > this change.  I'm going to start dropping groups of our facets from the
> > documents and seeing if there is some threshold that I'm hitting.  I'll
> > probably start with our hierarchies which are not particularly large, but
> > are the most suspect.
> >
> > Thanks for any input,
> > Marc
> >
> > 9.4.2
> > Time(ms) per Document
> > facetConfig.build   : 0.9882365
> > Taxo Add: 0.8334876
> >
> > 9.5
> > facetConfig.build   : 11.037549
> > Taxo Add: 10.915726
> >
> > On Fri, Apr 19, 2024 at 2:56 AM Dawid Weiss 
> wrote:
> >
> >> Hi Marc,
> >>
> >> You could try git bisect lucene repository to pinpoint the commit that
> >> caused what you're observing. It'll take some time to build but it's a
> >> logarithmic bisection and you'd know for sure where the problem is.
> >>
> >> D.
> >>
> >> On Thu, Apr 18, 2024 at 11:16 PM Marc Davenport
> >>  wrote:
> >>
> >> > Hi Adrien et al,
> >> > I've been doing some investigation today and it looks like whatever
> the
> >> > change is, it happens between 9.4.2 and 9.5.0.
> >> > I made a smaller test set up for our code that mocks our documents and
> >> just
> >> > runs through the indexing portion of our code sending in batches of 4k
> >> > documents at a time. This way I can run it locally.
> >> > 9.4.2: ~1200-2000 documents per second
> >> > 9.5.0: ~150-400 documents per second
> >> >
> >> > I'll continue investigating, but nothing in the release notes jumped
> >> out to
> >> > me.
> >> > https://lucene.apache.org/core/9_10_0/changes/Changes.html#v9.5.0
> >> >
> >> > Sorry I don't have anything more rigorous yet.  I'm doing this
> >> > investigation in parallel with some other things.
> >> > But any insight or suggestions on areas to look would be appreciated.
> >> > Thank you,
> >> > Marc
> >> >
> >> > On Wed, Apr 17, 2024 at 4:18 PM Adrien Grand 
> wrote:
> >> >
> >> > > Hi Marc,
> >> > >
> >> > > Nothing jumps to mind as a potential cause for this 2x regression.
> It
> >> > would
> >> > > be interesting to look at a profile.
> >> > >
> >> > > On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
> >> > >  wrote:
> >> > >
> >> > > > Hello,
> >> > > > I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall
> >> build
> >> > > can
> >> > > > now support Java 11. The quick first step of renaming packages and
> >> > > > importing the new libraries has gone well.  I'm even seeing a nice
> >> > > > performance bump in our average query time. I am however seeing a
> >> > > dramatic
> >> > > > increase in our indexing time.  We are indexing ~3.1 million
> >> documents
> >> > > each
> >> > > > with about 100 attributes used for facet filter, and sorting; no
> >> > lexical
> >> > > > text search.  Our indexing time has jumped from ~1k seconds to ~2k
> >> > > > seconds.  I have yet to profile the individual aspects of how we
> >> > convert
> >> > > > our data to records vs time for the index writer to accept the
> >> > documents.
> >> > > > I'm curious if other users discovered this for their migrations at
> >> some
> >> > > > point.  Or if there are some changes to defaults that I did not
> see
> >> in
> >> > > the
> >> > > > migration guide that would account for this?  Looking at the logs
> I
> >> can
> >> > > see
> >> > > > that as we are indexing the documents we commit 

Re: Indexing time increase moving from Lucene 8 to 9

2024-04-22 Thread Marc Davenport
Hello,
I've done bisect between 9.4.2 and 9.5 and found the PR affecting my
particular set up : https://github.com/apache/lucene/pull/12093
This is the switch from UTF8TaxonomyWriterCache to an
LruTaxonomyWriterCache.   I don't see a way to control the size of this
cache to never expel items and match the previous behavior.
Marc


On Fri, Apr 19, 2024 at 4:39 PM Marc Davenport 
wrote:

> Hello,
> Thanks for the leads. I haven't yet gone as far as doing a git bisect, but
> I have found that the big jump in time is in the call to
> facetsConfig.build(taxonomyWriter, doc);  I made a quick and dirty
> instrumented version of the FacetsConfig class and found that calls to
> TaxonomyWriter.add(FacetLabel) are significantly slower for me.
>
>
> https://github.com/apache/lucene/blob/releases/lucene/9.5.0/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L383
>
> I don't know what is special about my documents that I would be seeing
> this change.  I'm going to start dropping groups of our facets from the
> documents and seeing if there is some threshold that I'm hitting.  I'll
> probably start with our hierarchies which are not particularly large, but
> are the most suspect.
>
> Thanks for any input,
> Marc
>
> 9.4.2
> Time(ms) per Document
> facetConfig.build   : 0.9882365
> Taxo Add: 0.8334876
>
> 9.5
> facetConfig.build   : 11.037549
> Taxo Add: 10.915726
>
> On Fri, Apr 19, 2024 at 2:56 AM Dawid Weiss  wrote:
>
>> Hi Marc,
>>
>> You could try git bisect lucene repository to pinpoint the commit that
>> caused what you're observing. It'll take some time to build but it's a
>> logarithmic bisection and you'd know for sure where the problem is.
>>
>> D.
>>
>> On Thu, Apr 18, 2024 at 11:16 PM Marc Davenport
>>  wrote:
>>
>> > Hi Adrien et al,
>> > I've been doing some investigation today and it looks like whatever the
>> > change is, it happens between 9.4.2 and 9.5.0.
>> > I made a smaller test set up for our code that mocks our documents and
>> just
>> > runs through the indexing portion of our code sending in batches of 4k
>> > documents at a time. This way I can run it locally.
>> > 9.4.2: ~1200-2000 documents per second
>> > 9.5.0: ~150-400 documents per second
>> >
>> > I'll continue investigating, but nothing in the release notes jumped
>> out to
>> > me.
>> > https://lucene.apache.org/core/9_10_0/changes/Changes.html#v9.5.0
>> >
>> > Sorry I don't have anything more rigorous yet.  I'm doing this
>> > investigation in parallel with some other things.
>> > But any insight or suggestions on areas to look would be appreciated.
>> > Thank you,
>> > Marc
>> >
>> > On Wed, Apr 17, 2024 at 4:18 PM Adrien Grand  wrote:
>> >
>> > > Hi Marc,
>> > >
>> > > Nothing jumps to mind as a potential cause for this 2x regression. It
>> > would
>> > > be interesting to look at a profile.
>> > >
>> > > On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
>> > >  wrote:
>> > >
>> > > > Hello,
>> > > > I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall
>> build
>> > > can
>> > > > now support Java 11. The quick first step of renaming packages and
>> > > > importing the new libraries has gone well.  I'm even seeing a nice
>> > > > performance bump in our average query time. I am however seeing a
>> > > dramatic
>> > > > increase in our indexing time.  We are indexing ~3.1 million
>> documents
>> > > each
>> > > > with about 100 attributes used for facet filter, and sorting; no
>> > lexical
>> > > > text search.  Our indexing time has jumped from ~1k seconds to ~2k
>> > > > seconds.  I have yet to profile the individual aspects of how we
>> > convert
>> > > > our data to records vs time for the index writer to accept the
>> > documents.
>> > > > I'm curious if other users discovered this for their migrations at
>> some
>> > > > point.  Or if there are some changes to defaults that I did not see
>> in
>> > > the
>> > > > migration guide that would account for this?  Looking at the logs I
>> can
>> > > see
>> > > > that as we are indexing the documents we commit every 10 minutes.
>> > > > Thank you,
>> > > > Marc
>> > > >
>> > >
>> > >
>> > > --
>> > > Adrien
>> > >
>> >
>>
>


Re: Indexing time increase moving from Lucene 8 to 9

2024-04-19 Thread Marc Davenport
Hello,
Thanks for the leads. I haven't yet gone as far as doing a git bisect, but
I have found that the big jump in time is in the call to
facetsConfig.build(taxonomyWriter, doc);  I made a quick and dirty
instrumented version of the FacetsConfig class and found that calls to
TaxonomyWriter.add(FacetLabel) are significantly slower for me.

https://github.com/apache/lucene/blob/releases/lucene/9.5.0/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L383

I don't know what is special about my documents that I would be seeing this
change.  I'm going to start dropping groups of our facets from the
documents and seeing if there is some threshold that I'm hitting.  I'll
probably start with our hierarchies which are not particularly large, but
are the most suspect.

Thanks for any input,
Marc

9.4.2
Time(ms) per Document
facetConfig.build   : 0.9882365
Taxo Add: 0.8334876

9.5
facetConfig.build   : 11.037549
Taxo Add: 10.915726

On Fri, Apr 19, 2024 at 2:56 AM Dawid Weiss  wrote:

> Hi Marc,
>
> You could try git bisect lucene repository to pinpoint the commit that
> caused what you're observing. It'll take some time to build but it's a
> logarithmic bisection and you'd know for sure where the problem is.
>
> D.
>
> On Thu, Apr 18, 2024 at 11:16 PM Marc Davenport
>  wrote:
>
> > Hi Adrien et al,
> > I've been doing some investigation today and it looks like whatever the
> > change is, it happens between 9.4.2 and 9.5.0.
> > I made a smaller test set up for our code that mocks our documents and
> just
> > runs through the indexing portion of our code sending in batches of 4k
> > documents at a time. This way I can run it locally.
> > 9.4.2: ~1200-2000 documents per second
> > 9.5.0: ~150-400 documents per second
> >
> > I'll continue investigating, but nothing in the release notes jumped out
> to
> > me.
> > https://lucene.apache.org/core/9_10_0/changes/Changes.html#v9.5.0
> >
> > Sorry I don't have anything more rigorous yet.  I'm doing this
> > investigation in parallel with some other things.
> > But any insight or suggestions on areas to look would be appreciated.
> > Thank you,
> > Marc
> >
> > On Wed, Apr 17, 2024 at 4:18 PM Adrien Grand  wrote:
> >
> > > Hi Marc,
> > >
> > > Nothing jumps to mind as a potential cause for this 2x regression. It
> > would
> > > be interesting to look at a profile.
> > >
> > > On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
> > >  wrote:
> > >
> > > > Hello,
> > > > I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall
> build
> > > can
> > > > now support Java 11. The quick first step of renaming packages and
> > > > importing the new libraries has gone well.  I'm even seeing a nice
> > > > performance bump in our average query time. I am however seeing a
> > > dramatic
> > > > increase in our indexing time.  We are indexing ~3.1 million
> documents
> > > each
> > > > with about 100 attributes used for facet filter, and sorting; no
> > lexical
> > > > text search.  Our indexing time has jumped from ~1k seconds to ~2k
> > > > seconds.  I have yet to profile the individual aspects of how we
> > convert
> > > > our data to records vs time for the index writer to accept the
> > documents.
> > > > I'm curious if other users discovered this for their migrations at
> some
> > > > point.  Or if there are some changes to defaults that I did not see
> in
> > > the
> > > > migration guide that would account for this?  Looking at the logs I
> can
> > > see
> > > > that as we are indexing the documents we commit every 10 minutes.
> > > > Thank you,
> > > > Marc
> > > >
> > >
> > >
> > > --
> > > Adrien
> > >
> >
>


Re: Indexing time increase moving from Lucene 8 to 9

2024-04-19 Thread Dawid Weiss
Hi Marc,

You could try git bisect lucene repository to pinpoint the commit that
caused what you're observing. It'll take some time to build but it's a
logarithmic bisection and you'd know for sure where the problem is.

D.

On Thu, Apr 18, 2024 at 11:16 PM Marc Davenport
 wrote:

> Hi Adrien et al,
> I've been doing some investigation today and it looks like whatever the
> change is, it happens between 9.4.2 and 9.5.0.
> I made a smaller test set up for our code that mocks our documents and just
> runs through the indexing portion of our code sending in batches of 4k
> documents at a time. This way I can run it locally.
> 9.4.2: ~1200-2000 documents per second
> 9.5.0: ~150-400 documents per second
>
> I'll continue investigating, but nothing in the release notes jumped out to
> me.
> https://lucene.apache.org/core/9_10_0/changes/Changes.html#v9.5.0
>
> Sorry I don't have anything more rigorous yet.  I'm doing this
> investigation in parallel with some other things.
> But any insight or suggestions on areas to look would be appreciated.
> Thank you,
> Marc
>
> On Wed, Apr 17, 2024 at 4:18 PM Adrien Grand  wrote:
>
> > Hi Marc,
> >
> > Nothing jumps to mind as a potential cause for this 2x regression. It
> would
> > be interesting to look at a profile.
> >
> > On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
> >  wrote:
> >
> > > Hello,
> > > I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall build
> > can
> > > now support Java 11. The quick first step of renaming packages and
> > > importing the new libraries has gone well.  I'm even seeing a nice
> > > performance bump in our average query time. I am however seeing a
> > dramatic
> > > increase in our indexing time.  We are indexing ~3.1 million documents
> > each
> > > with about 100 attributes used for facet filter, and sorting; no
> lexical
> > > text search.  Our indexing time has jumped from ~1k seconds to ~2k
> > > seconds.  I have yet to profile the individual aspects of how we
> convert
> > > our data to records vs time for the index writer to accept the
> documents.
> > > I'm curious if other users discovered this for their migrations at some
> > > point.  Or if there are some changes to defaults that I did not see in
> > the
> > > migration guide that would account for this?  Looking at the logs I can
> > see
> > > that as we are indexing the documents we commit every 10 minutes.
> > > Thank you,
> > > Marc
> > >
> >
> >
> > --
> > Adrien
> >
>


Re: Indexing time increase moving from Lucene 8 to 9

2024-04-18 Thread Gautam Worah
Does your application see a lot of document updates/deletes?
GITHUB#11761 could have potentially affected you. Whenever I see large
indexing times, my first suspicion is towards increased merge activity.

Regards,
Gautam Worah.


On Thu, Apr 18, 2024 at 2:14 PM Marc Davenport
 wrote:

> Hi Adrien et al,
> I've been doing some investigation today and it looks like whatever the
> change is, it happens between 9.4.2 and 9.5.0.
> I made a smaller test set up for our code that mocks our documents and just
> runs through the indexing portion of our code sending in batches of 4k
> documents at a time. This way I can run it locally.
> 9.4.2: ~1200-2000 documents per second
> 9.5.0: ~150-400 documents per second
>
> I'll continue investigating, but nothing in the release notes jumped out to
> me.
> https://lucene.apache.org/core/9_10_0/changes/Changes.html#v9.5.0
>
> Sorry I don't have anything more rigorous yet.  I'm doing this
> investigation in parallel with some other things.
> But any insight or suggestions on areas to look would be appreciated.
> Thank you,
> Marc
>
> On Wed, Apr 17, 2024 at 4:18 PM Adrien Grand  wrote:
>
> > Hi Marc,
> >
> > Nothing jumps to mind as a potential cause for this 2x regression. It
> would
> > be interesting to look at a profile.
> >
> > On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
> >  wrote:
> >
> > > Hello,
> > > I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall build
> > can
> > > now support Java 11. The quick first step of renaming packages and
> > > importing the new libraries has gone well.  I'm even seeing a nice
> > > performance bump in our average query time. I am however seeing a
> > dramatic
> > > increase in our indexing time.  We are indexing ~3.1 million documents
> > each
> > > with about 100 attributes used for facet filter, and sorting; no
> lexical
> > > text search.  Our indexing time has jumped from ~1k seconds to ~2k
> > > seconds.  I have yet to profile the individual aspects of how we
> convert
> > > our data to records vs time for the index writer to accept the
> documents.
> > > I'm curious if other users discovered this for their migrations at some
> > > point.  Or if there are some changes to defaults that I did not see in
> > the
> > > migration guide that would account for this?  Looking at the logs I can
> > see
> > > that as we are indexing the documents we commit every 10 minutes.
> > > Thank you,
> > > Marc
> > >
> >
> >
> > --
> > Adrien
> >
>


Re: Indexing time increase moving from Lucene 8 to 9

2024-04-18 Thread Marc Davenport
Hi Adrien et al,
I've been doing some investigation today and it looks like whatever the
change is, it happens between 9.4.2 and 9.5.0.
I made a smaller test set up for our code that mocks our documents and just
runs through the indexing portion of our code sending in batches of 4k
documents at a time. This way I can run it locally.
9.4.2: ~1200-2000 documents per second
9.5.0: ~150-400 documents per second

I'll continue investigating, but nothing in the release notes jumped out to
me.
https://lucene.apache.org/core/9_10_0/changes/Changes.html#v9.5.0

Sorry I don't have anything more rigorous yet.  I'm doing this
investigation in parallel with some other things.
But any insight or suggestions on areas to look would be appreciated.
Thank you,
Marc

On Wed, Apr 17, 2024 at 4:18 PM Adrien Grand  wrote:

> Hi Marc,
>
> Nothing jumps to mind as a potential cause for this 2x regression. It would
> be interesting to look at a profile.
>
> On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
>  wrote:
>
> > Hello,
> > I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall build
> can
> > now support Java 11. The quick first step of renaming packages and
> > importing the new libraries has gone well.  I'm even seeing a nice
> > performance bump in our average query time. I am however seeing a
> dramatic
> > increase in our indexing time.  We are indexing ~3.1 million documents
> each
> > with about 100 attributes used for facet filter, and sorting; no lexical
> > text search.  Our indexing time has jumped from ~1k seconds to ~2k
> > seconds.  I have yet to profile the individual aspects of how we convert
> > our data to records vs time for the index writer to accept the documents.
> > I'm curious if other users discovered this for their migrations at some
> > point.  Or if there are some changes to defaults that I did not see in
> the
> > migration guide that would account for this?  Looking at the logs I can
> see
> > that as we are indexing the documents we commit every 10 minutes.
> > Thank you,
> > Marc
> >
>
>
> --
> Adrien
>


Re: Indexing time increase moving from Lucene 8 to 9

2024-04-17 Thread Adrien Grand
Hi Marc,

Nothing jumps to mind as a potential cause for this 2x regression. It would
be interesting to look at a profile.

On Wed, Apr 17, 2024 at 9:32 PM Marc Davenport
 wrote:

> Hello,
> I'm finally migrating Lucene from 8.11.2 to 9.10.0 as our overall build can
> now support Java 11. The quick first step of renaming packages and
> importing the new libraries has gone well.  I'm even seeing a nice
> performance bump in our average query time. I am however seeing a dramatic
> increase in our indexing time.  We are indexing ~3.1 million documents each
> with about 100 attributes used for facet filter, and sorting; no lexical
> text search.  Our indexing time has jumped from ~1k seconds to ~2k
> seconds.  I have yet to profile the individual aspects of how we convert
> our data to records vs time for the index writer to accept the documents.
> I'm curious if other users discovered this for their migrations at some
> point.  Or if there are some changes to defaults that I did not see in the
> migration guide that would account for this?  Looking at the logs I can see
> that as we are indexing the documents we commit every 10 minutes.
> Thank you,
> Marc
>


-- 
Adrien