Well, the code is correct, because it can work by avoiding this trap. But it failed to act as a good API.
I learned the inside details from you. I am not the only one that's trapped. And more users will likely be trapped again, unless javadoc to describe the close() function is changed. Actually, I didn't look at the javadoc of close(), because, shouldn't close() means close(), not uncontrollably delayed resource releasing? So I fear just changing the javadoc is not enough. -- Chris Lu ------------------------- Instant Scalable Full-Text Search On Any Database/Application site: http://www.dbsight.net demo: http://search.dbsight.com Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes DBSight customer, a shopping comparison site, (anonymous per request) got 2.6 Million Euro funding! On Wed, Sep 10, 2008 at 1:03 PM, robert engels <[EMAIL PROTECTED]>wrote: > Always your prerogative. > > On Sep 10, 2008, at 1:15 PM, Chris Lu wrote: > > Actually I am done with it by simply downgrading and not to use r659602 > and later.The old version is more clean and consistent with the API and > close() does mean close, not something complicated and unknown to most > users, which almost feels like a trap. And later on, if no changes happened > for this file, I will have to upgrade Lucene and manually remove the patch > Lucene-1195. > > -- > Chris Lu > ------------------------- > Instant Scalable Full-Text Search On Any Database/Application > site: http://www.dbsight.net > demo: http://search.dbsight.com > Lucene Database Search in 3 minutes: > http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes > DBSight customer, a shopping comparison site, (anonymous per request) got > 2.6 Million Euro funding! > > On Wed, Sep 10, 2008 at 10:56 AM, robert engels <[EMAIL PROTECTED]>wrote: > >> Why not just use reopen() and be done with it??? >> >> On Sep 10, 2008, at 12:48 PM, Chris Lu wrote: >> >> Yeah, the timing is different. But it's an unknown, undetermined, and >> uncontrollable time... >> We can not ask the user, >> >> while(memory is low){ >> sleep(1000); >> } >> do_the_real_thing_an_hour_later >> >> >> -- >> Chris Lu >> ------------------------- >> Instant Scalable Full-Text Search On Any Database/Application >> site: http://www.dbsight.net >> demo: http://search.dbsight.com >> Lucene Database Search in 3 minutes: >> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >> DBSight customer, a shopping comparison site, (anonymous per request) got >> 2.6 Million Euro funding! >> >> On Wed, Sep 10, 2008 at 10:39 AM, robert engels <[EMAIL PROTECTED]>wrote: >> >>> Close() does work - it is just that the memory may not be freed until >>> much later... >>> When working with VERY LARGE objects, this can be a problem. >>> >>> On Sep 10, 2008, at 12:36 PM, Chris Lu wrote: >>> >>> Thanks for the analysis, really appreciate it, and I agree with it. >>> But... >>> This is really a normal J2EE use case. The threads seldom die. >>> Doesn't that mean closing the RAMDirectory doesn't work for J2EE >>> applications? >>> And only reopen() works? >>> And close() doesn't release the resources? duh... >>> >>> I can only say this is a problem to be cleaned up. >>> >>> -- >>> Chris Lu >>> ------------------------- >>> Instant Scalable Full-Text Search On Any Database/Application >>> site: http://www.dbsight.net >>> demo: http://search.dbsight.com >>> Lucene Database Search in 3 minutes: >>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>> DBSight customer, a shopping comparison site, (anonymous per request) got >>> 2.6 Million Euro funding! >>> >>> >>> On Wed, Sep 10, 2008 at 9:10 AM, robert engels <[EMAIL PROTECTED]>wrote: >>> >>>> You do not need to create a new RAMDirectory - just write to the >>>> existing one, and then reopen() the IndexReader using it. >>>> This will prevent lots of big objects being created. This may be the >>>> source of your problem. >>>> >>>> Even if the Segment is closed, the ThreadLocal will no longer be >>>> referenced, but there will still be a reference to the SegmentTermEnum >>>> (which will be cleared when the thread dies, or "most likely" when new >>>> thread locals on that thread a created, so here is a potential problem. >>>> >>>> Thread 1 does a search, creates a thread local that references the >>>> RAMDir (A). >>>> Thread 2 does a search, creates a thread local that references the >>>> RAMDir (A). >>>> >>>> All readers, are closed on RAMDir (A). >>>> >>>> A new RAMDir (B) is opened. >>>> >>>> There may still be references in the thread local maps to RAMDir A >>>> (since no new thread local have been created yet). >>>> >>>> So you may get OOM depending on the size of the RAMDir (since you would >>>> need room for more than 1). If you extend this out with lots of threads >>>> that don't run very often, you can see how you could easily run out of >>>> memory. "I think" that ThreadLocal should use a ReferenceQueue so stale >>>> object slots can be reclaimed as soon as the key is dereferenced - but that >>>> is an issue for SUN. >>>> >>>> This is why you don't want to create new RAMDirs. >>>> >>>> A good rule of thumb - don't keep references to large objects in >>>> ThreadLocal (especially indirectly). If needed, use a "key", and then read >>>> the cache using a the "key". >>>> This would be something for the Lucene folks to change. >>>> >>>> On Sep 10, 2008, at 10:44 AM, Chris Lu wrote: >>>> >>>> I am really want to find out where I am doing wrong, if that's the >>>> case. >>>> >>>> Yes. I have made certain that I closed all Readers/Searchers, and >>>> verified that through memory profiler. >>>> Yes. I am creating new RAMDirectory. But that's the problem. I need to >>>> update the content. Sure, if no content update and everything the same, of >>>> course no OOM. >>>> >>>> Yes. No guarantee of the thread schedule. But that's the problem. If >>>> Lucene is using ThreadLocal to cache lots of things by the Thread as the >>>> key, and no idea when it'll be released. Of course ThreadLocal is not >>>> Lucene's problem... >>>> >>>> Chris >>>> >>>> On Wed, Sep 10, 2008 at 8:34 AM, robert engels <[EMAIL PROTECTED]>wrote: >>>> >>>>> It is basic Java. Threads are not guaranteed to run on any sort of >>>>> schedule. If you create lots of large objects in one thread, releasing >>>>> them >>>>> in another, there is a good chance you will get an OOM (since the >>>>> releasing >>>>> thread may not run before the OOM occurs)... This is not Lucene specific >>>>> by >>>>> any means. >>>>> It is a misunderstanding on your part about how GC works. >>>>> >>>>> I assume you must at some point be creating new RAMDirectories - >>>>> otherwise the memory would never really increase, since the >>>>> IndexReader/enums/etc are not very large... >>>>> >>>>> When you create a new RAMDirectories, you need to BE CERTAIN !!! that >>>>> the other IndexReaders/Searchers using the old RAMDirectory are ALL >>>>> CLOSED, >>>>> otherwise their memory will still be in use, which leads to your OOM... >>>>> >>>>> >>>>> On Sep 10, 2008, at 10:16 AM, Chris Lu wrote: >>>>> >>>>> I do not believe I am making any mistake. Actually I just got an email >>>>> from another user, complaining about the same thing. And I am having the >>>>> same usage pattern. >>>>> After the reader is opened, the RAMDirectory is shared by several >>>>> objects. >>>>> There is one instance of RAMDirectory in the memory, and it is holding >>>>> lots of memory, which is expected. >>>>> >>>>> If I close the reader in the same thread that has opened it, the >>>>> RAMDirectory is gone from the memory. >>>>> If I close the reader in other threads, the RAMDirectory is left in the >>>>> memory, referenced along the tree I draw in the first email. >>>>> >>>>> I do not think the usage is wrong. Period. >>>>> >>>>> ------------------------------------- >>>>> >>>>> Hi, >>>>> >>>>> i found a forum post from you here [1] where you mention that you >>>>> have a memory leak using the lucene ram directory. I'd like to ask you >>>>> if you already have resolved the problem and how you did it or maybe >>>>> you know where i can read about the solution. We are using >>>>> RAMDirectory too and figured out, that over time the memory >>>>> consumption raises and raises until the system breaks down but only >>>>> when we performing much index updates. if we only create the index and >>>>> don't do nothing except searching it, it work fine. >>>>> >>>>> maybe you can give me a hint or a link, >>>>> greetz, >>>>> >>>>> ------------------------------------- >>>>> >>>>> -- >>>>> Chris Lu >>>>> ------------------------- >>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>> site: http://www.dbsight.net >>>>> demo: http://search.dbsight.com >>>>> Lucene Database Search in 3 minutes: >>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>> DBSight customer, a shopping comparison site, (anonymous per request) >>>>> got 2.6 Million Euro funding! >>>>> >>>>> On Wed, Sep 10, 2008 at 7:12 AM, robert engels <[EMAIL PROTECTED]>wrote: >>>>> >>>>>> Sorry, but I am fairly certain you are mistaken. >>>>>> If you only have a single IndexReader, the RAMDirectory will be shared >>>>>> in all cases. >>>>>> >>>>>> The only memory growth is any buffer space allocated by an IndexInput >>>>>> (used in many places and cached). >>>>>> >>>>>> Normally the IndexInput created by a RAMDirectory do not have any >>>>>> buffer allocated, since the underlying store is already in memory. >>>>>> >>>>>> You have some other problem in your code... >>>>>> >>>>>> On Sep 10, 2008, at 1:10 AM, Chris Lu wrote: >>>>>> >>>>>> Actually, even I only use one IndexReader, some resources are cached >>>>>> via the ThreadLocal cache, and can not be released unless all threads do >>>>>> the >>>>>> close action. >>>>>> >>>>>> SegmentTermEnum itself is small, but it holds RAMDirectory along the >>>>>> path, which is big. >>>>>> >>>>>> -- >>>>>> Chris Lu >>>>>> ------------------------- >>>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>>> site: http://www.dbsight.net >>>>>> demo: http://search.dbsight.com >>>>>> Lucene Database Search in 3 minutes: >>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>>> DBSight customer, a shopping comparison site, (anonymous per request) >>>>>> got 2.6 Million Euro funding! >>>>>> On Tue, Sep 9, 2008 at 10:43 PM, robert engels <[EMAIL PROTECTED] >>>>>> > wrote: >>>>>> >>>>>>> You do not need a pool of IndexReaders... >>>>>>> It does not matter what class it is, what matters is the class that >>>>>>> ultimately holds the reference. >>>>>>> >>>>>>> If the IndexReader is never closed, the SegmentReader(s) is never >>>>>>> closed, so the thread local in TermInfosReader is not cleared (because >>>>>>> the >>>>>>> thread never dies). So you will get one SegmentTermEnum, per thread * >>>>>>> per >>>>>>> segment. >>>>>>> >>>>>>> The SegmentTermEnum is not a large object, so even if you had 100 >>>>>>> threads, and 100 segments, for 10k instances, seems hard to believe >>>>>>> that is >>>>>>> the source of your memory issue. >>>>>>> >>>>>>> The SegmentTermEnum is cached by thread since it needs to enumerate >>>>>>> the terms, not having a per thread cache, would lead to lots of random >>>>>>> access when multiple threads read the index - very slow. >>>>>>> >>>>>>> You need to keep in mind, what if every thread was executing a search >>>>>>> simultaneously - you would still have 100x100 SegmentTermEnum instances >>>>>>> anyway ! The only way to prevent that would be to create and destroy >>>>>>> the >>>>>>> SegmentTermEnum on each call (opening and seeking to the proper spot) - >>>>>>> which would be SLOW SLOW SLOW. >>>>>>> >>>>>>> On Sep 10, 2008, at 12:19 AM, Chris Lu wrote: >>>>>>> >>>>>>> I have tried to create an IndexReader pool and dynamically create >>>>>>> searcher. But the memory leak is the same. It's not related to the >>>>>>> Searcher >>>>>>> class specifically, but the SegmentTermEnum in TermInfosReader. >>>>>>> >>>>>>> -- >>>>>>> Chris Lu >>>>>>> ------------------------- >>>>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>>>> site: http://www.dbsight.net >>>>>>> demo: http://search.dbsight.com >>>>>>> Lucene Database Search in 3 minutes: >>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>>>> DBSight customer, a shopping comparison site, (anonymous per request) >>>>>>> got 2.6 Million Euro funding! >>>>>>> >>>>>>> On Tue, Sep 9, 2008 at 10:14 PM, robert engels < >>>>>>> [EMAIL PROTECTED]> wrote: >>>>>>> >>>>>>>> A searcher uses an IndexReader - the IndexReader is slow to open, >>>>>>>> not a Searcher. And searchers can share an IndexReader. >>>>>>>> You want to create a single shared (across all threads/users) >>>>>>>> IndexReader (usually), and create an Searcher as needed and dispose. >>>>>>>> It is >>>>>>>> VERY CHEAP to create the Searcher. >>>>>>>> >>>>>>>> I am fairly certain the javadoc on Searcher is incorrect. The >>>>>>>> warning "For performance reasons it is recommended to open only one >>>>>>>> IndexSearcher and use it for all of your searches" is not true in >>>>>>>> the case where an IndexReader is passed to the ctor. >>>>>>>> >>>>>>>> Any caching should USUALLY be performed at the IndexReader level. >>>>>>>> >>>>>>>> You are most likely using the "path" ctor, and that is the source of >>>>>>>> your problems, as multiple IndexReader instances are being created, >>>>>>>> and thus >>>>>>>> the memory use. >>>>>>>> >>>>>>>> >>>>>>>> On Sep 9, 2008, at 11:44 PM, Chris Lu wrote: >>>>>>>> >>>>>>>> On J2EE environment, usually there is a searcher pool with several >>>>>>>> searchers open. The speed to opening a large index for every user >>>>>>>> is not acceptable. >>>>>>>> >>>>>>>> -- >>>>>>>> Chris Lu >>>>>>>> ------------------------- >>>>>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>>>>> site: http://www.dbsight.net >>>>>>>> demo: http://search.dbsight.com >>>>>>>> Lucene Database Search in 3 minutes: >>>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>>>>> DBSight customer, a shopping comparison site, (anonymous per >>>>>>>> request) got 2.6 Million Euro funding! >>>>>>>> >>>>>>>> On Tue, Sep 9, 2008 at 9:03 PM, robert engels < >>>>>>>> [EMAIL PROTECTED]> wrote: >>>>>>>> >>>>>>>>> You need to close the searcher within the thread that is using it, >>>>>>>>> in order to have it cleaned up quickly... usually right after you >>>>>>>>> display >>>>>>>>> the page of results. >>>>>>>>> If you are keeping multiple searcher refs across multiple threads >>>>>>>>> for paging/whatever, you have not coded it correctly. >>>>>>>>> >>>>>>>>> Imagine 10,000 users - storing a searcher for each one is not going >>>>>>>>> to work... >>>>>>>>> >>>>>>>>> On Sep 9, 2008, at 10:21 PM, Chris Lu wrote: >>>>>>>>> >>>>>>>>> Right, in a sense I can not release it from another thread. But >>>>>>>>> that's the problem. >>>>>>>>> >>>>>>>>> It's a J2EE environment, all threads are kind of equal. It's simply >>>>>>>>> not possible to iterate through all threads to close the searcher, >>>>>>>>> thus >>>>>>>>> releasing the ThreadLocal cache. >>>>>>>>> Unless Lucene is not recommended for J2EE environment, this has to >>>>>>>>> be fixed. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Chris Lu >>>>>>>>> ------------------------- >>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>>>>>> site: http://www.dbsight.net >>>>>>>>> demo: http://search.dbsight.com >>>>>>>>> Lucene Database Search in 3 minutes: >>>>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>>>>>> DBSight customer, a shopping comparison site, (anonymous per >>>>>>>>> request) got 2.6 Million Euro funding! >>>>>>>>> >>>>>>>>> On Tue, Sep 9, 2008 at 8:14 PM, robert engels < >>>>>>>>> [EMAIL PROTECTED]> wrote: >>>>>>>>> >>>>>>>>>> Your code is not correct. You cannot release it on another thread >>>>>>>>>> - the first thread may creating hundreds/thousands of instances >>>>>>>>>> before the >>>>>>>>>> other thread ever runs... >>>>>>>>>> >>>>>>>>>> On Sep 9, 2008, at 10:10 PM, Chris Lu wrote: >>>>>>>>>> >>>>>>>>>> If I release it on the thread that's creating the searcher, by >>>>>>>>>> setting searcher=null, everything is fine, the memory is released >>>>>>>>>> very >>>>>>>>>> cleanly. >>>>>>>>>> My load test was to repeatedly create a searcher on a RAMDirectory >>>>>>>>>> and release it on another thread. The test will quickly go to OOM >>>>>>>>>> after >>>>>>>>>> several runs. I set the heap size to be 1024M, and the RAMDirectory >>>>>>>>>> is of >>>>>>>>>> size 250M. Using some profiling tool, the used size simply stepped >>>>>>>>>> up pretty >>>>>>>>>> obviously by 250M. >>>>>>>>>> >>>>>>>>>> I think we should not rely on something that's a "maybe" behavior, >>>>>>>>>> especially for a general purpose library. >>>>>>>>>> >>>>>>>>>> Since it's a multi-threaded env, the thread that's creating the >>>>>>>>>> entries in the LRU cache may not go away quickly(actually most, if >>>>>>>>>> not all, >>>>>>>>>> application servers will try to reuse threads), so the LRU cache, >>>>>>>>>> which uses >>>>>>>>>> thread as the key, can not be released, so the SegmentTermEnum which >>>>>>>>>> is in >>>>>>>>>> the same class can not be released. >>>>>>>>>> >>>>>>>>>> And yes, I close the RAMDirectory, and the fileMap is released. I >>>>>>>>>> verified that through the profiler by directly checking the values >>>>>>>>>> in the >>>>>>>>>> snapshot. >>>>>>>>>> >>>>>>>>>> Pretty sure the reference tree wasn't like this using code before >>>>>>>>>> this commit, because after close the searcher in another thread, the >>>>>>>>>> RAMDirectory totally disappeared from the memory snapshot. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Chris Lu >>>>>>>>>> ------------------------- >>>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>>>>>>> site: http://www.dbsight.net >>>>>>>>>> demo: http://search.dbsight.com >>>>>>>>>> Lucene Database Search in 3 minutes: >>>>>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>>>>>>> DBSight customer, a shopping comparison site, (anonymous per >>>>>>>>>> request) got 2.6 Million Euro funding! >>>>>>>>>> >>>>>>>>>> On Tue, Sep 9, 2008 at 5:03 PM, Michael McCandless < >>>>>>>>>> [EMAIL PROTECTED]> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Chris Lu wrote: >>>>>>>>>>> >>>>>>>>>>> The problem should be similar to what's talked about on this >>>>>>>>>>>> discussion. >>>>>>>>>>>> >>>>>>>>>>>> http://lucene.markmail.org/message/keosgz2c2yjc7qre?q=ThreadLocal >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The "rough" conclusion of that thread is that, technically, this >>>>>>>>>>> isn't a memory leak but rather a "delayed freeing" problem. Ie, it >>>>>>>>>>> may take >>>>>>>>>>> longer, possibly much longer, than you want for the memory to be >>>>>>>>>>> freed. >>>>>>>>>>> >>>>>>>>>>> There is a memory leak for Lucene search from Lucene-1195.(svn >>>>>>>>>>>> r659602, May23,2008) >>>>>>>>>>>> >>>>>>>>>>>> This patch brings in a ThreadLocal cache to TermInfosReader. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> One thing that confuses me: TermInfosReader was already using a >>>>>>>>>>> ThreadLocal to cache the SegmentTermEnum instance. What was added >>>>>>>>>>> in this >>>>>>>>>>> commit (for LUCENE-1195) was an LRU cache storing Term -> TermInfo >>>>>>>>>>> instances. But it seems like it's the SegmentTermEnum instance >>>>>>>>>>> that you're >>>>>>>>>>> tracing below. >>>>>>>>>>> >>>>>>>>>>> It's usually recommended to keep the reader open, and reuse it >>>>>>>>>>>> when >>>>>>>>>>>> possible. In a common J2EE application, the http requests are >>>>>>>>>>>> usually >>>>>>>>>>>> handled by different threads. But since the cache is >>>>>>>>>>>> ThreadLocal, the cache >>>>>>>>>>>> are not really usable by other threads. What's worse, the cache >>>>>>>>>>>> can not be >>>>>>>>>>>> cleared by another thread! >>>>>>>>>>>> >>>>>>>>>>>> This leak is not so obvious usually. But my case is using >>>>>>>>>>>> RAMDirectory, >>>>>>>>>>>> having several hundred megabytes. So one un-released resource is >>>>>>>>>>>> obvious to >>>>>>>>>>>> me. >>>>>>>>>>>> >>>>>>>>>>>> Here is the reference tree: >>>>>>>>>>>> org.apache.lucene.store.RAMDirectory >>>>>>>>>>>> |- directory of org.apache.lucene.store.RAMFile >>>>>>>>>>>> |- file of org.apache.lucene.store.RAMInputStream >>>>>>>>>>>> |- base of >>>>>>>>>>>> org.apache.lucene.index.CompoundFileReader$CSIndexInput >>>>>>>>>>>> |- input of org.apache.lucene.index.SegmentTermEnum >>>>>>>>>>>> |- value of >>>>>>>>>>>> java.lang.ThreadLocal$ThreadLocalMap$Entry >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> So you have a RAMDir that has several hundred MB stored in it, >>>>>>>>>>> that you're done with yet through this path Lucene is keeping it >>>>>>>>>>> alive? >>>>>>>>>>> >>>>>>>>>>> Did you close the RAMDir? (which will null its fileMap and >>>>>>>>>>> should also free your memory). >>>>>>>>>>> >>>>>>>>>>> Also, that reference tree doesn't show the ThreadResources class >>>>>>>>>>> that was added in that commit -- are you sure this reference tree >>>>>>>>>>> wasn't >>>>>>>>>>> before the commit? >>>>>>>>>>> >>>>>>>>>>> Mike >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Chris Lu >>>>>>>>>> ------------------------- >>>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>>>>>>> site: http://www.dbsight.net >>>>>>>>>> demo: http://search.dbsight.com >>>>>>>>>> Lucene Database Search in 3 minutes: >>>>>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>>>>>>> DBSight customer, a shopping comparison site, (anonymous per >>>>>>>>>> request) got 2.6 Million Euro funding! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Chris Lu >>>> ------------------------- >>>> Instant Scalable Full-Text Search On Any Database/Application >>>> site: http://www.dbsight.net >>>> demo: http://search.dbsight.com >>>> Lucene Database Search in 3 minutes: >>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>> DBSight customer, a shopping comparison site, (anonymous per request) >>>> got 2.6 Million Euro funding! >>>> >>>> >>>> >>> >>> >>> >>> >> >> >> > > > >
