RE: Large index files
Hi I think (a) would be a better choice [I have done it on Linux upt to 7GB , it's pretty faster then doing the same on win2000 PF] with regards Karthik -Original Message- From: Rupinder Singh Mazara [mailto:[EMAIL PROTECTED] Sent: Friday, July 23, 2004 5:55 PM To: Lucene Users List Subject: Large index files Hi all I am using lucene to index a large dataset, it so happens 10% of this data yields indexes of 400MB, in all likelihood it is possible the index may go upto 7GB. My deployment will be on a linux/tomcat system, what will be a better solution a) create one large index and hope linux does not mind b) generate 7-10 indexes based on some criteria and glue them together using MultiReader, in this case I may cross the MAX file handles limit of Tomcat ? regards - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Large index files
As long as your kernel has Large File Support, then you should be fine. Most modern distro's support 2GB files now out of the box. John On Fri, 2004-07-23 at 13:44, Karthik N S wrote: Hi I think (a) would be a better choice [I have done it on Linux upt to 7GB , it's pretty faster then doing the same on win2000 PF] with regards Karthik -Original Message- From: Rupinder Singh Mazara [mailto:[EMAIL PROTECTED] Sent: Friday, July 23, 2004 5:55 PM To: Lucene Users List Subject: Large index files Hi all I am using lucene to index a large dataset, it so happens 10% of this data yields indexes of 400MB, in all likelihood it is possible the index may go upto 7GB. My deployment will be on a linux/tomcat system, what will be a better solution a) create one large index and hope linux does not mind b) generate 7-10 indexes based on some criteria and glue them together using MultiReader, in this case I may cross the MAX file handles limit of Tomcat ? regards - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- John Moylan RT ePublishing, Montrose House, Donnybrook, Dublin 4 T: +353 1 2083564 E: [EMAIL PROTECTED] ** The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Please note that emails to, from and within RT may be subject to the Freedom of Information Act 1997 and may be liable to disclosure. ** - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Large index files
I'm a little confused by this. I thought Lucene keeps creating new files as the index gets bigger and any single file doesn't ever get all that big. Is that not the case? Thanks, Joel Shellman John Moylan wrote: As long as your kernel has Large File Support, then you should be fine. Most modern distro's support 2GB files now out of the box. John On Fri, 2004-07-23 at 13:44, Karthik N S wrote: Hi I think (a) would be a better choice [I have done it on Linux upt to 7GB , it's pretty faster then doing the same on win2000 PF] with regards Karthik -Original Message- From: Rupinder Singh Mazara [mailto:[EMAIL PROTECTED] Sent: Friday, July 23, 2004 5:55 PM To: Lucene Users List Subject: Large index files Hi all I am using lucene to index a large dataset, it so happens 10% of this data yields indexes of 400MB, in all likelihood it is possible the index may go upto 7GB. My deployment will be on a linux/tomcat system, what will be a better solution a) create one large index and hope linux does not mind b) generate 7-10 indexes based on some criteria and glue them together using MultiReader, in this case I may cross the MAX file handles limit of Tomcat ? regards - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Large index files
by optimizing the created in index(es) u can reduce multiple files into a smaller set of files and on some file systems might be a good idea to optimize once in while -Original Message- From: Joel Shellman [mailto:[EMAIL PROTECTED] Sent: 23 July 2004 14:38 To: Lucene Users List Subject: Re: Large index files I'm a little confused by this. I thought Lucene keeps creating new files as the index gets bigger and any single file doesn't ever get all that big. Is that not the case? Thanks, Joel Shellman John Moylan wrote: As long as your kernel has Large File Support, then you should be fine. Most modern distro's support 2GB files now out of the box. John On Fri, 2004-07-23 at 13:44, Karthik N S wrote: Hi I think (a) would be a better choice [I have done it on Linux upt to 7GB , it's pretty faster then doing the same on win2000 PF] with regards Karthik -Original Message- From: Rupinder Singh Mazara [mailto:[EMAIL PROTECTED] Sent: Friday, July 23, 2004 5:55 PM To: Lucene Users List Subject: Large index files Hi all I am using lucene to index a large dataset, it so happens 10% of this data yields indexes of 400MB, in all likelihood it is possible the index may go upto 7GB. My deployment will be on a linux/tomcat system, what will be a better solution a) create one large index and hope linux does not mind b) generate 7-10 indexes based on some criteria and glue them together using MultiReader, in this case I may cross the MAX file handles limit of Tomcat ? regards - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
