There is https://issues.apache.org/jira/browse/LUCENE-3837 but I suppose it's too far from completion.
On Wed, Apr 4, 2012 at 2:48 PM, Ravish Bhagdev <ravish.bhag...@gmail.com>wrote: > Updating a single field is not possible in solr. The whole record has to > be rewritten. > > 300 MB is still not that big a file. Have you tried doing the indexing (if > its only a one time thing) by giving it ~2 GB or xmx? > > A single file with that size is strange! May I ask what is it? > > Rav > > On Tue, Apr 3, 2012 at 7:32 PM, vybe3142 <vybe3...@gmail.com> wrote: > > > > > Some days ago, I posted about an issue with SOLR running out of memory > when > > attempting to index large text files (say 300 MB ). Details at > > > > > http://lucene.472066.n3.nabble.com/Solr-Tika-crashing-when-attempting-to-index-large-files-td3846939.html > > > > Two things I need to point out: > > > > 1. I don't need Tika for content extraction as the files are already in > > plain text format. > > 2. The heap space error was caused by a futile Tika/SOLR attempt at > > creating > > the corresponding huge XML document in memory > > > > I've decided to develop a custom handler that > > 1. reads the file text directly > > 2. attempts to create a SOLR document and directly add the text data to > the > > corresponding field. > > > > One approach I've taken is to read manageable chunks of text data > > sequentially from the file and process. We've used this approach > > sucessfully > > with Lucene in the past and I'm attempting to make it work with SOLR > too. I > > got most of the work done yesterday, but need a bit of guidance w.r.t. > > point > > 2. > > > > How can I achieve updating the same field multiple times. Looking at the > > SOLR source, processor.addField() merely > > a. adds to the in-memory field map and > > b. attempts to write EVERYTHING to the index later on. > > > > In my situation, (a) eventually causes a heap space error. > > > > > > > > > > Here's part of the handler code. > > > > > > > > Thanks much > > > > Thanks > > > > -- > > View this message in context: > > > http://lucene.472066.n3.nabble.com/Incremantally-updating-a-VERY-LARGE-field-Is-this-possibe-tp3881945p3881945.html > > Sent from the Solr - User mailing list archive at Nabble.com. > > > -- Sincerely yours Mikhail Khludnev ge...@yandex.ru <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
