Hi Daniel! Long time no see :) Anyway, Marcin forwarded me with your question so I quickly follow.
In theory it is possible to build an automaton that is larger than Java heap (by early disk-flushing of parts that are immutable). The problem arises at runtime; although you could map a file buffer and use it for lookups, automaton traversals are typically not "local" so they'd cause a lot of disk-swapping and this in general would be super-slow (even on a ssd disk swap). This is the primary reason I originally decided to write everything based on a simple Java array. What's your data and why do you need to cram everything in RAM? Perhaps there's some other options I could recommend? Dawid > ---------- Forwarded message ---------- > From: Daniel Naber <daniel.na...@languagetool.org> > Date: 2014-06-11 22:28 GMT+02:00 > Subject: Morfologik for large data sets > To: LanguageTool Developer List <languagetool-devel@lists.sourceforge.net> > > > Hi, > > has anybody used Morfologik for data sets larger than the computer's > memory? Is it even possible, or does everything need to be loaded into > RAM? > > Regards > Daniel > > > ------------------------------------------------------------------------------ > HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions > Find What Matters Most in Your Big Data with HPCC Systems > Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. > Leverages Graph Analysis for Fast Processing & Easy Data Exploration > http://p.sf.net/sfu/hpccsystems > _______________________________________________ > Languagetool-devel mailing list > Languagetool-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/languagetool-devel > > ------------------------------------------------------------------------------ HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing & Easy Data Exploration http://p.sf.net/sfu/hpccsystems _______________________________________________ Languagetool-devel mailing list Languagetool-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/languagetool-devel
