Please feel free to provide a pull request that adds the ability to load the FST off heap to WFSTCompletionLookup. I think it's an oversight and it'd be a good addition.
Dawid On Tue, Dec 27, 2022 at 10:35 AM marcos rebelo <ole...@gmail.com> wrote: > > I have the same impression, even if I'm using the MMapDirectory. The data > is on heap. > > For my use case, it's a huge waste of memory :( 90% of my data could be > correctly organised and kept in disk. > > Thanks for the support > > Best regards > Marcos Rebelo > > On Tue, 27 Dec 2022, 09:11 Dawid Weiss, <dawid.we...@gmail.com> wrote: > > > Looking at the code briefly, I think WFSTCompletionLookup uses on heap > > store for the fst. You'd have to load it with off heap fst store instead: > > > > > > https://github.com/apache/lucene/blob/1b9d98d6ec079e950bdd37137082f81400d3bc2e/lucene/core/src/java/org/apache/lucene/util/fst/OffHeapFSTStore.java > > > > but I don't think there is an API in WFSTCompletionLookup that would allow > > you to do that. > > > > D. > > > > On Fri, Dec 23, 2022 at 5:00 PM marcos rebelo <ole...@gmail.com> wrote: > > > > > Hey all! > > > > > > I'm loading multiple WFST with ~1.1 Gb and the JVM memory increases > > > proportionally. Looks like the file is stored in memory, meaning not > > using > > > Memory Mapped Files at all. > > > > > > Example code: > > > > > > In the following code we setup the Lucene to use /tmp/deleteme2 for the > > > memory mapped file and we load the file from /tmp/deleteme/file.wfst via > > an > > > InputStream. > > > > > > After file load I list the files on /tmp/deleteme2 and nothing is found, > > > but I'm able to query the WFST file. > > > > > > @Test > > > @SneakyThrows > > > void WFSTLoad() throws IOException { > > > Path wfstPath = Paths.get("/tmp/deleteme2"); > > > Path wfstFilePath = Paths.get("/tmp/deleteme/file.wfst"); > > > > > > var directory = new MMapDirectory(wfstPath); > > > > > > WFSTCompletionLookup wfst = > > > new WFSTCompletionLookup(directory, "temp"); > > > > > > try (var is = new FileInputStream(wfstFilePath.toFile())) { > > > wfst.load(is); > > > System.out.println("FILE LOADED"); > > > } > > > > > > Files.list(wfstPath).forEach(System.out::println); > > > System.out.println("FILES LISTED"); > > > > > > assertThat(wfst.get("qwert123qwert")).isEqualTo(123); > > > } > > > > > > What am I doing wrong? > > > > > > Thanks for the support > > > > > > Best Regards > > > Marcos Rebelo > > > > > > -- > > > > > > *Marcos Bruno Gomes Rebelo Engineering Manager / Data Scientist / > > Software > > > Engineer* > > > Linkedin: https://www.linkedin.com/in/oleber/ > > > *Adding value to your data. Specialized in Search and Recommendation > > > Systems* > > > Technologies: Elastic, Spark, Scala, Jupiter Notebook, Python, ... > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
