Thanks Adrien. I spent some time trying to understand the readByte() in ReverseRandomAccessReader (through FST) and compare with 7.x. Although I don't understand ALL of the details and reasoning for always loading the FST (and in turn the term index) off-heap (as discussed in https://github.com/apache/lucene/issues/10297 ) I understand that this is essentially causing disk access for every single byte during readByte().
Does this warrant a JIRA for regression? As mentioned, I am noticing a 10x slowdown in SegmentTermsEnum.seekExact() affecting atomic update performance . For setups like mine that can't use mmap due to large indexes this would be a legit regression, no? - Rahul On Tue, Jun 6, 2023 at 10:09 AM Adrien Grand <jpou...@gmail.com> wrote: > Yes, this changed in 8.x: > - 8.0 moved the terms index off-heap for non-PK fields with > MMapDirectory. https://github.com/apache/lucene/issues/9681 > - Then in 8.6 the FST was moved off-heap all the time. > https://github.com/apache/lucene/issues/10297 > > More generally, there's a few files that are no longer loaded in heap > in 8.x. It should be possible to load them back in heap by doing > something like that (beware, I did not actually test this code): > > class MyHeapDirectory extends FilterDirectory { > > MyHeapDirectory(Directory in) { > super(in); > } > > @Override > public IndexInput openInput(String name, IOContext context) throws > IOException { > if (context.load == false) { > return super.openInput(name, context); > } else { > try (IndexInput in = super.openInput(name, context)) { > byte[] bytes = new byte[Math.toIntExact(in.length())]; > in.readBytes(bytes, bytes.length); > ByteBuffer bb = > ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer(); > return new ByteBuffersIndexInput(new > ByteBuffersDataInput(Collections.singletonList(bb)), > "ByteBuffersIndexInput(" + name + ")"); > } > } > } > > } > > On Tue, Jun 6, 2023 at 3:41 PM Rahul Goswami <rahul196...@gmail.com> > wrote: > > > > Thanks Adrien. Is this behavior of FST something that has changed in > Lucene > > 8.x (from 7.x)? > > Also, is the terms index not loaded into memory anymore in 8.x? > > > > To your point on MMapDirectoryFactory, it is much faster as you > > anticipated, but the indexes commonly being >1 TB makes the Windows > machine > > freeze to a point I sometimes can't even connect to the VM. > > SimpleFSDirectory works well for us from that standpoint. > > > > To add, both NIOFS and SimpleFS have similar indexing benchmarks on > > Windows. I understand it is because of the Java bug which synchronizes > > internally in the native call for NIOFs. > > > > -Rahul > > > > On Tue, Jun 6, 2023 at 9:32 AM Adrien Grand <jpou...@gmail.com> wrote: > > > > > +Alan Woodward helped me better understand what is going on here. > > > BufferedIndexInput (used by NIOFSDirectory and SimpleFSDirectory) > > > doesn't play well with the fact that the FST reads bytes backwards: > > > every call to readByte() triggers a refill of 1kB because it wants to > > > read the byte that is just before what the buffer contains. > > > > > > On Tue, Jun 6, 2023 at 2:07 PM Adrien Grand <jpou...@gmail.com> wrote: > > > > > > > > My best guess based on your description of the issue is that > > > > SimpleFSDirectory doesn't like the fact that the terms index now > reads > > > > data directly from the directory instead of loading the terms index > in > > > > heap. Would you be able to run the same benchmark with MMapDirectory > > > > to check if it addresses the regression? > > > > > > > > > > > > On Tue, Jun 6, 2023 at 5:47 AM Rahul Goswami <rahul196...@gmail.com> > > > wrote: > > > > > > > > > > Hello, > > > > > We started experiencing slowness with atomic updates in Solr after > > > > > upgrading from 7.7.2 to 8.11.1. Running several tests revealed the > > > > > slowness to be in RealTimeGet's SolrIndexSearcher.getFirstMatch() > call > > > > > which eventually calls Lucene's SegmentTermsEnum.seekExact().. > > > > > > > > > > In the benchmarks I ran, 8.11.1 is about 10x slower than 7.7.2. > After > > > > > discussion on the Solr mailing list I created the below JIRA: > > > > > > > > > > https://issues.apache.org/jira/browse/SOLR-16838 > > > > > > > > > > The thread dumps collected show a lot of threads stuck in the > > > > > FST.findTargetArc() > > > > > method. Testing environment details: > > > > > > > > > > Environment details: > > > > > - Java 11 on Windows server > > > > > - Xms1536m Xmx3072m > > > > > - Indexing client code running 15 parallel threads indexing in > batches > > > of > > > > > 1000 on a standalone core. > > > > > - using SimpleFSDirectoryFactory (since Mmap doesn't quite work > well > > > on > > > > > Windows for our index sizes which commonly run north of 1 TB) > > > > > > > > > > > > > > https://drive.google.com/drive/folders/1q2DPNTYQEU6fi3NeXIKJhaoq3KPnms0h?usp=sharing > > > > > > > > > > Is there a known issue with slowness with TermsEnum.seekExact() in > > > Lucene > > > > > 8.x ? > > > > > > > > > > Thanks, > > > > > Rahul > > > > > > > > > > > > > > > > -- > > > > Adrien > > > > > > > > > > > > -- > > > Adrien > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > > > -- > Adrien > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
