1. What is the difference when you used different vm parameters? 2 What merge policy and optimization strategy did you use? 3. How did you use the commit or flush ?
Qi On Wed, Oct 20, 2010 at 2:05 PM, Sahin Buyrukbilen < sahin.buyrukbi...@gmail.com> wrote: > Thank you so much for this infor. it looks pretty complicated for me but I > will try. > > > > On Wed, Oct 20, 2010 at 1:18 AM, Johnbin Wang <johnbin.w...@gmail.com > >wrote: > > > You can start a fixedThreadPool to index all these files in the multhread > > way. Every thread execute an index task which could index a part of all > the > > files. In the index task, when indexing 10000 files, you need execute the > > indexWrite.commit() method to flush all the index add operation to disk > > file. > > > > If you need index all these files into only one index file, you need to > > hold > > only one indexWriter instance among all the index thread. > > > > Hope it's helpful. > > > > > > > > On Wed, Oct 20, 2010 at 1:05 PM, Sahin Buyrukbilen < > > sahin.buyrukbi...@gmail.com> wrote: > > > > > Thank you Johnbin, > > > do you know which parameter I have to play with? > > > > > > On Wed, Oct 20, 2010 at 12:59 AM, Johnbin Wang <johnbin.w...@gmail.com > > > >wrote: > > > > > > > I think you can write index file once every 10,000 files or less have > > > been > > > > read. > > > > > > > > On Wed, Oct 20, 2010 at 12:11 PM, Sahin Buyrukbilen < > > > > sahin.buyrukbi...@gmail.com> wrote: > > > > > > > > > Hi all, > > > > > > > > > > I have to index about 4.5Million txt files. When I run the my > > indexing > > > > > application through Eclipse, I get this error : "Exception in > thread > > > > "main" > > > > > java.lang.OutOfMemoryError: Java heap space" > > > > > > > > > > eclipse -vmargs -Xmx2000m -Xss8192k > > > > > > > > > > eclipse -vmargs -Xms40M -Xmx2G > > > > > > > > > > I tried running Eclipse with above memory parameters, but still > had > > > the > > > > > same error. The architecture of my computer is AMD x2 64bit 2GHz > > > > processor, > > > > > Ubuntu 10.04 LTS 64bit. java-6-openjdk. > > > > > > > > > > Anybody has a suggestion? > > > > > > > > > > thank you. > > > > > Sahin. > > > > > > > > > > > > > > > > > > > > > -- > > > > cheers, > > > > Johnbin Wang > > > > > > > > > > > > > > > -- > > cheers, > > Johnbin Wang > > >
