Hi Jayson,
It is on my list of things to do. I've been having a very busy week
and and am also working all weekend. I hope to get to it next week
sometime, if no-one else has taken it.
cheers,
-mike
On 8-May-09, at 10:15 PM, jayson.minard wrote:
First cut of updated handler now in:
Thanks Mike, I'm running it in a few environments that do not have
post-commit hooks and so far have not seen any issues. A white-box review
will be helpful in seeing things that may rarely occur, or if I had any
misuse if internal data structures that I do not know well enough to
measure.
--j
of the memory consumption by this queue if commit takes
much longer to complete.
Thanks,
Siddharth
-Original Message-
From: jayson.minard [mailto:jayson.min...@gmail.com]
Sent: Saturday, May 09, 2009 10:45 AM
To: solr-user@lucene.apache.org
Subject: Re: Autocommit blocking adds? AutoCommit
-Original Message-
From: jayson.minard [mailto:jayson.min...@gmail.com]
Sent: Saturday, May 09, 2009 10:45 AM
To: solr-user@lucene.apache.org
Subject: Re: Autocommit blocking adds? AutoCommit Speedup?
First cut of updated handler now in:
https://issues.apache.org/jira/browse/SOLR
Siddharth,
The settings you have in your solrconfig for ramBufferSizeMB and
maxBufferedDocs control how much memory may be used during indexing besides
any overhead with the documents being in-flight at a given moment
(deserialized into memory but not yet handed to lucene). There are
streaming
Indexing speed comes down to a lot of factors. The settings as talked about
above, VM settings, the size of the documents, how many are sent at a time,
how active you can keep the indexer (i.e. one thread sending documents lets
the indexer relax whereas N threads keeps pressure on the indexer),
Any pointers to this newer more concurrent behavior in lucene? I can try an
experiment where I downgrade the iwCommit lock to the iwAccess lock to allow
updates to happen during commit.
Would you expect that to work?
Thanks for bootstrapping me on this.
Jim
Yonik Seeley-2 wrote:
On
On Fri, May 8, 2009 at 4:27 PM, Jim Murphy jim.mur...@pobox.com wrote:
Any pointers to this newer more concurrent behavior in lucene?
At the API level we care about, IndexWriter.commit() instead of close()
Also, we shouldn't have to worry about other parts of the code closing
the writer on us
Created issue:
https://issues.apache.org/jira/browse/SOLR-1155
Jim Murphy wrote:
Any pointers to this newer more concurrent behavior in lucene? I can try
an experiment where I downgrade the iwCommit lock to the iwAccess lock to
allow updates to happen during commit.
Would you
Yonik Seeley-2 wrote:
...your code snippit elided and edited below ...
Don't take this code as correct (or even compiling) but is this the essence?
I moved shared access to the writer inside the read lock and kept the other
non-commit bits to the write lock. I'd need to rethink the
Can we move this to patch files within the JIRA issue please. Will make it
easier to review and help out a as a patch to current trunk.
--j
Jim Murphy wrote:
Yonik Seeley-2 wrote:
...your code snippit elided and edited below ...
Don't take this code as correct (or even
First cut of updated handler now in:
https://issues.apache.org/jira/browse/SOLR-1155
Needs review from those that know Lucene better, and double check for errors
in locking or other areas of the code. Thanks.
--j
jayson.minard wrote:
Can we move this to patch files within the JIRA issue
On Thu, May 7, 2009 at 5:03 PM, Jim Murphy jim.mur...@pobox.com wrote:
Question 1: I see in DirectUpdateHandler2 that there is a read/Write lock
used between addDoc and commit.
My mental model of the process was this: clients can add/update documents
until the auto commit threshold was hit.
Interesting. So is there a JIRA ticket open for this already? Any chance of
getting it into 1.4? Its seriously kicking out butts right now. We write
into our masters with ~50ms response times till we hit the autocommit then
add/update response time is 10-30 seconds. Ouch.
I'd be willing to
On Thu, May 7, 2009 at 8:37 PM, Jim Murphy jim.mur...@pobox.com wrote:
Interesting. So is there a JIRA ticket open for this already? Any chance of
getting it into 1.4?
No ticket currently open, but IMO it could make it for 1.4.
Its seriously kicking out butts right now. We write
into our
15 matches
Mail list logo
