Yes, this changed in 8.x:
- 8.0 moved the terms index off-heap for non-PK fields with
MMapDirectory. https://github.com/apache/lucene/issues/9681
- Then in 8.6 the FST was moved off-heap all the time.
https://github.com/apache/lucene/issues/10297
More generally, there's a few files that are no longer loaded in heap
in 8.x. It should be possible to load them back in heap by doing
something like that (beware, I did not actually test this code):
class MyHeapDirectory extends FilterDirectory {
MyHeapDirectory(Directory in) {
super(in);
}
@Override
public IndexInput openInput(String name, IOContext context) throws
IOException {
if (context.load == false) {
return super.openInput(name, context);
} else {
try (IndexInput in = super.openInput(name, context)) {
byte[] bytes = new byte[Math.toIntExact(in.length())];
in.readBytes(bytes, bytes.length);
ByteBuffer bb =
ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer();
return new ByteBuffersIndexInput(new
ByteBuffersDataInput(Collections.singletonList(bb)),
"ByteBuffersIndexInput(" + name + ")");
}
}
}
}
On Tue, Jun 6, 2023 at 3:41 PM Rahul Goswami <rahul196...@gmail.com> wrote:
>
> Thanks Adrien. Is this behavior of FST something that has changed in Lucene
> 8.x (from 7.x)?
> Also, is the terms index not loaded into memory anymore in 8.x?
>
> To your point on MMapDirectoryFactory, it is much faster as you
> anticipated, but the indexes commonly being >1 TB makes the Windows machine
> freeze to a point I sometimes can't even connect to the VM.
> SimpleFSDirectory works well for us from that standpoint.
>
> To add, both NIOFS and SimpleFS have similar indexing benchmarks on
> Windows. I understand it is because of the Java bug which synchronizes
> internally in the native call for NIOFs.
>
> -Rahul
>
> On Tue, Jun 6, 2023 at 9:32 AM Adrien Grand <jpou...@gmail.com> wrote:
>
> > +Alan Woodward helped me better understand what is going on here.
> > BufferedIndexInput (used by NIOFSDirectory and SimpleFSDirectory)
> > doesn't play well with the fact that the FST reads bytes backwards:
> > every call to readByte() triggers a refill of 1kB because it wants to
> > read the byte that is just before what the buffer contains.
> >
> > On Tue, Jun 6, 2023 at 2:07 PM Adrien Grand <jpou...@gmail.com> wrote:
> > >
> > > My best guess based on your description of the issue is that
> > > SimpleFSDirectory doesn't like the fact that the terms index now reads
> > > data directly from the directory instead of loading the terms index in
> > > heap. Would you be able to run the same benchmark with MMapDirectory
> > > to check if it addresses the regression?
> > >
> > >
> > > On Tue, Jun 6, 2023 at 5:47 AM Rahul Goswami <rahul196...@gmail.com>
> > wrote:
> > > >
> > > > Hello,
> > > > We started experiencing slowness with atomic updates in Solr after
> > > > upgrading from 7.7.2 to 8.11.1. Running several tests revealed the
> > > > slowness to be in RealTimeGet's SolrIndexSearcher.getFirstMatch() call
> > > > which eventually calls Lucene's SegmentTermsEnum.seekExact()..
> > > >
> > > > In the benchmarks I ran, 8.11.1 is about 10x slower than 7.7.2. After
> > > > discussion on the Solr mailing list I created the below JIRA:
> > > >
> > > > https://issues.apache.org/jira/browse/SOLR-16838
> > > >
> > > > The thread dumps collected show a lot of threads stuck in the
> > > > FST.findTargetArc()
> > > > method. Testing environment details:
> > > >
> > > > Environment details:
> > > > - Java 11 on Windows server
> > > > - Xms1536m Xmx3072m
> > > > - Indexing client code running 15 parallel threads indexing in batches
> > of
> > > > 1000 on a standalone core.
> > > > - using SimpleFSDirectoryFactory (since Mmap doesn't quite work well
> > on
> > > > Windows for our index sizes which commonly run north of 1 TB)
> > > >
> > > >
> > https://drive.google.com/drive/folders/1q2DPNTYQEU6fi3NeXIKJhaoq3KPnms0h?usp=sharing
> > > >
> > > > Is there a known issue with slowness with TermsEnum.seekExact() in
> > Lucene
> > > > 8.x ?
> > > >
> > > > Thanks,
> > > > Rahul
> > >
> > >
> > >
> > > --
> > > Adrien
> >
> >
> >
> > --
> > Adrien
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: java-user-h...@lucene.apache.org
> >
> >
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
