--- "Kevin A. Burton" <[EMAIL PROTECTED]> wrote: > Otis Gospodnetic wrote: > > >I like option 3. I've done it before, and it worked well. I dealt > >with very small indices, though, and if your indices are several > tens > >or hundred gigs, this may be hard for you. > > > >Option 4: search can be performed on an index that is being modified > >(update, delete, insert, optimize). You'd just have to make sure > not > >to recreate new IndexSearcher too frequently, if your index is being > >modified often. Just change it every X index modification or every > X > >minutes, and you'll be fine. > > > > > Right now I'm thinking about #4... Disk may be cheap but a fast RAID > 10 > array with 100G twice isn't THAT cheap... That's the worse case
Yes, but not everything needs to be on a fast RAID (you probabably are using SCSI disks in RAID, which is what makes it expensive. RAID requires only a RAID controller). You could have a Searcher machine with a set of cheap EIDE disks, and use those as a copy target disks, which are not searched. Once you transfer your indices there, you copy them on fast SCSI RAID disks. > Also... since the new indexes are SO small (~100M) the merges would > probably be easier on the machine than just doing a whole new write. > Of > course it's hard to make that argument with a 100G RAID array but > we're > using rysnc to avoid distribution of network IO so the CPU > computation > and network read would slow things down. > > The only way around this is the re-upload the whole 100G index but > even > over gigabit ethernet this will take 15 minutes. This doesn't scale > as we add more searchers. I wonder what happens if you try compressing the indices before copying them over the network. I wonder if it makes a difference whether you use compound vs. traditional directories. I wonder what the index size is if you use DbDirectory instead of FSDirectory. Otis --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
