John, Are you indicating that later Lucene releases might have a config setting that can control the write I/O timeout? If so, do you happen to know where it is or how to set it? I did quick Googling, but all I get back is the write lock timeout which is set to one second by default. Thanks /Jong On Tue, Oct 2, 2012 at 9:38 AM, Nader, John P <john.na...@cengage.com>wrote:
> We've been in production on Lucene over NFS for about 4 years now. Though > we've had performance issues related to NFS (similar to those mentioned on > this thread), we've only seen some reliability issues. Index writing I/O > timeout exceptions are the primary issue. We've addressed these by > implementing retry logic. This kept transactional consistency and avoided > corruption. I can't recall specifically where these timeouts occurred, > but I do remember that on our version of Lucene at the time (3.0.2) the > timeout was not configurable, and defaulted to 5 seconds. Had it been > configurable, we could have reduced how frequently we needed to rollback > and retry. > > On another point, NFS performance can be greatly enhanced with servers > that have extra memory for mapping the index files. We found after > initial warmup and loading of Lucene that queries performance very well > since most of the data needed to execute them is cached locally. > > Also, keep in mind that local disk is not a free ride either. This takes > you down data replication. You end up repeating indexing once per > replica. You also may have to move the indices around as you > add/remove/restart nodes. We are moving to this architecture with a new > product, so I am just now starting to understand the trade-offs. > > Hope that helps. > > -John > > > > > On 10/2/12 8:01 AM, "Jong Kim" <jong.luc...@gmail.com> wrote: > > >Thank you all for reply. > > > >So it soudns like it is a known fact that the performance would suffer > >rather significantly when the index files are accessed over NFS. But how > >about reliability and robustness (which seems even more important)? Isn't > >there any increased possibility for intermittent errors such as index file > >corruption (due to cache inconsistency, difference in delete semantics, > >etc.) when using NFS? Has anyone run into such trouble? Or is it strictly > >just a performance issue? > > > >/Jong > >On Tue, Oct 2, 2012 at 5:17 AM, Paul Libbrecht <p...@hoplahup.net> wrote: > > > >> My experience in the Lucene 1.x times were a factor of at least four in > >> writing to NFS and about two when reading from there. I'd discourage > >>this > >> as much as possible! > >> > >> (rsync is way more your friend for transporting and replication à la > >>solr > >> should also be considered) > >> > >> paul > >> > >> > >> Le 2 oct. 2012 à 11:10, Ian Lea a écrit : > >> > >> > You'll certainly need to factor in the performance of NFS versus local > >> disks. > >> > > >> > My experience is that smallish low activity indexes work just fine on > >> > NFS, but large high activity indexes are not so good, particularly if > >> > you have a lot of modifications to the index. > >> > > >> > You may want to install a custom IndexDeletionPolicy. See the > >> > javadocs for details with specific reference to NFS. > >> > > >> > > >> > -- > >> > Ian. > >> > > >> > On Tue, Oct 2, 2012 at 3:21 AM, Vitaly Funstein <vfunst...@gmail.com> > >> wrote: > >> >> How tolerant is your project of decreased search and indexing > >> performance? > >> >> You could probably write a simple test that compares search and write > >> >> speeds of local and NFS-mounted indexes and make the decision based > >>on > >> the > >> >> results. > >> >> > >> >> On Mon, Oct 1, 2012 at 3:06 PM, Jong Kim <jong.luc...@gmail.com> > >>wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> According to the Lucene In Action (Second Edition), the section > >>2.11.2 > >> >>> "Accessing an index over a remote file system" explains that there > >>are > >> >>> issues related to accessing a Lucene index across remote file system > >> >>> including NFS. > >> >>> > >> >>> I'm particuarly interested in NFS compatibility, and wondering if > >> there has > >> >>> been any work done to solve or mitigate this problem. Has this issue > >> been > >> >>> addressed? If not, are there some reliable work-arounds that make > >>this > >> >>> possible at the expense of some sacrifice in other areas? > >> >>> > >> >>> Any information would be greatly appreciated, since my project > >>heavily > >> >>> depends on the feasibility of this. > >> >>> > >> >>> Thanks > >> >>> /Jong > >> >>> > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> > For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
