Re: distributed deletes working?

2012-02-27 Thread Jamie Johnson
Thanks Mark. I'll pull the latest trunk today and run with that. On Sun, Feb 26, 2012 at 10:37 AM, Mark Miller markrmil...@gmail.com wrote: Are there any outstanding issues that I should be aware of? Not that I know of - we where trying to track down an issue around peer sync recovery

Re: distributed deletes working?

2012-02-26 Thread Mark Miller
Are there any outstanding issues that I should be aware of? Not that I know of - we where trying to track down an issue around peer sync recovery that our ChaosMonkey* tests were tripping, but looks like Yonik may have tracked that down last night. * The ChaosMonkey tests randomly start,

Re: distributed deletes working?

2012-02-24 Thread Mark Miller
On Feb 22, 2012, at 9:54 PM, Jamie Johnson wrote: Perhaps if you could give me the steps you're using to test I can find an error in what I'm doing. I've tested a few ways - one of them as Sami has already explained. Could you try one more time from the latest? A few issues have been

Re: distributed deletes working?

2012-02-24 Thread Jamie Johnson
I'm pulling the latest now. Once I've rebuilt and setup the test I'll forward all the logs on to you. Again thanks for looking into this. On Fri, Feb 24, 2012 at 9:20 PM, Mark Miller markrmil...@gmail.com wrote: On Feb 22, 2012, at 9:54 PM, Jamie Johnson wrote: Perhaps if you could give me

Re: distributed deletes working?

2012-02-24 Thread Jamie Johnson
Pulling the latest version seems to have fixed whatever issue that previously existed so everything appears to be working properly. I'm seeing updates make it to the downed server once it recovers and deletes (even by query, which I wasn't sure would work) being forwarded as well. So everything

Re: distributed deletes working?

2012-02-24 Thread Mark Miller
No problem! Only wish I had found what the issue was sooner - I suspected a couple other issues we found could have been related, but since I had not duplicated it I could not be sure - the early use and feedback is invaluable though. On Feb 24, 2012, at 10:53 PM, Jamie Johnson wrote:

Re: distributed deletes working?

2012-02-24 Thread Jamie Johnson
I'm always excited to try the new stuff, we've been running off of a fairly old version from solrbranch which I'd like to move off of, this is the first step. Are there any outstanding issues that I should be aware of? I've only run simple tests thus far, but plan to run some more comprehensive

Re: distributed deletes working?

2012-02-22 Thread Jamie Johnson
I know everyone is busy, but I was wondering if anyone had found anything with this? Any suggestions on what I could be doing wrong would be greatly appreciated. On Fri, Feb 17, 2012 at 4:08 PM, Mark Miller markrmil...@gmail.com wrote: On Feb 17, 2012, at 3:56 PM, Jamie Johnson wrote: id

Re: distributed deletes working?

2012-02-22 Thread Mark Miller
Yonik did fix an issue around peer sync and deletes a few days ago - long chance that was involved? Otherwise, neither Sami nor I have replicated these results so far. On Feb 22, 2012, at 8:56 PM, Jamie Johnson wrote: I know everyone is busy, but I was wondering if anyone had found anything

Re: distributed deletes working?

2012-02-22 Thread Jamie Johnson
Perhaps if you could give me the steps you're using to test I can find an error in what I'm doing. On Wed, Feb 22, 2012 at 9:24 PM, Mark Miller markrmil...@gmail.com wrote: Yonik did fix an issue around peer sync and deletes a few days ago - long chance that was involved? Otherwise, neither

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
Thanks Mark. I'm still seeing some issues while indexing though. I have the same setup describe in my previous email. I do some indexing to the cluster with everything up and everything looks good. I then take down one instance which is running 2 cores (shard2 slice 1 and shard 1 slice 2) and

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
and having looked at this closer, shouldn't the down node not be marked as active when I stop that solr instance? On Fri, Feb 17, 2012 at 10:04 AM, Jamie Johnson jej2...@gmail.com wrote: Thanks Mark.  I'm still seeing some issues while indexing though.  I have the same setup describe in my

Re: distributed deletes working?

2012-02-17 Thread Sami Siren
On Fri, Feb 17, 2012 at 5:10 PM, Jamie Johnson jej2...@gmail.com wrote: and having looked at this closer, shouldn't the down node not be marked as active when I stop that solr instance? Currently the shard state is not updated in the cloudstate when a node goes down. This behavior should

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
Thanks Sami, so long at it's expected ;) In regards to the replication not working the way I think it should, am I missing something or is it simply not working the way I think? On Fri, Feb 17, 2012 at 11:01 AM, Sami Siren ssi...@gmail.com wrote: On Fri, Feb 17, 2012 at 5:10 PM, Jamie Johnson

Re: distributed deletes working?

2012-02-17 Thread Sami Siren
On Fri, Feb 17, 2012 at 6:03 PM, Jamie Johnson jej2...@gmail.com wrote: Thanks Sami, so long at it's expected ;) In regards to the replication not working the way I think it should, am I missing something or is it simply not working the way I think? It should work. I also tried to reproduce

Re: distributed deletes working?

2012-02-17 Thread Mark Miller
On Feb 17, 2012, at 11:03 AM, Jamie Johnson wrote: Thanks Sami, so long at it's expected ;) Yeah, its expected - we always use both the live nodes info and state to determine the full state for a shard. In regards to the replication not working the way I think it should, am I missing

Re: distributed deletes working?

2012-02-17 Thread Yonik Seeley
On Fri, Feb 17, 2012 at 11:13 AM, Mark Miller markrmil...@gmail.com wrote: When exactly is this build from? Yeah... I just checked in a fix yesterday dealing with sync while indexing is going on. -Yonik lucidimagination.com

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
I stop the indexing, stop the shard, then start indexing again. So shouldn't need Yonik's latest fix? In regards to how far out of sync, it's completely out of sync, meaning index 100 documents to the cluster (40 on shard1 60 on shard2) then stop the instance, index 100 more, when I bring the

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
I'm seeing the following. Do I need a _version_ long field in my schema? Feb 17, 2012 1:15:50 PM org.apache.solr.update.processor.LogUpdateProcessor finish INFO: {delete=[f2c29abe-2e48-4965-adfb-8bd611293ff0]} 0 0 Feb 17, 2012 1:15:50 PM org.apache.solr.common.SolrException log SEVERE:

Re: distributed deletes working?

2012-02-17 Thread Yonik Seeley
On Fri, Feb 17, 2012 at 1:27 PM, Jamie Johnson jej2...@gmail.com wrote: I'm seeing the following.  Do I need a _version_ long field in my schema? Yep... versions are the way we keep things sane (shuffled updates to a replica can be correctly reordered, etc). -Yonik lucidimagination.com

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
Ok, so I'm making some progress now. With _version_ in the schema (forgot about this because I remember asking about it before) deletes across the cluster work when I delete by id. Updates work as well if a node is down it recovered fine. Something that didn't work though was if a node was down

Re: distributed deletes working?

2012-02-17 Thread Yonik Seeley
On Fri, Feb 17, 2012 at 1:38 PM, Jamie Johnson jej2...@gmail.com wrote: Something that didn't work though was if a node was down when a delete happened and then comes back up, that node still listed the id I deleted.  Is this currently supported? Yes, that should work fine. Are you still

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
Yes, still seeing that. Master has 8 items, replica has 9. So the delete didn't seem to work when the node was down. On Fri, Feb 17, 2012 at 1:41 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Fri, Feb 17, 2012 at 1:38 PM, Jamie Johnson jej2...@gmail.com wrote: Something that didn't

Re: distributed deletes working?

2012-02-17 Thread Mark Miller
Hmm...just tried this with only deletes, and the replica sync'd fine for me. Is this with your multi core setup or were you trying with instances? On Feb 17, 2012, at 1:52 PM, Jamie Johnson wrote: Yes, still seeing that. Master has 8 items, replica has 9. So the delete didn't seem to work

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
This was with the cloud-dev solrcloud-start.sh script (after that I've used solrcloud-start-existing.sh). Essentially I run ./solrcloud-start-existing.sh index docs kill 1 of the solr instances (using kill -9 on the pid) delete a doc from running instances restart killed solr instance on doing

Re: distributed deletes working?

2012-02-17 Thread Yonik Seeley
On Fri, Feb 17, 2012 at 2:07 PM, Jamie Johnson jej2...@gmail.com wrote: This was with the cloud-dev solrcloud-start.sh script (after that I've used solrcloud-start-existing.sh). Essentially I run ./solrcloud-start-existing.sh index docs kill 1 of the solr instances (using kill -9 on the pid)

Re: distributed deletes working?

2012-02-17 Thread Mark Miller
You are committing in that mix right? On Feb 17, 2012, at 2:07 PM, Jamie Johnson wrote: This was with the cloud-dev solrcloud-start.sh script (after that I've used solrcloud-start-existing.sh). Essentially I run ./solrcloud-start-existing.sh index docs kill 1 of the solr instances (using

Re: distributed deletes working?

2012-02-17 Thread Jamie Johnson
yes committing in the mix. id field is a UUID. On Fri, Feb 17, 2012 at 3:22 PM, Mark Miller markrmil...@gmail.com wrote: You are committing in that mix right? On Feb 17, 2012, at 2:07 PM, Jamie Johnson wrote: This was with the cloud-dev solrcloud-start.sh script (after that I've used

Re: distributed deletes working?

2012-02-17 Thread Mark Miller
On Feb 17, 2012, at 3:56 PM, Jamie Johnson wrote: id field is a UUID. Strange - was using UUID's myself in same test this morning... I'll try again soon. - Mark Miller lucidimagination.com

Re: distributed deletes working?

2012-02-16 Thread Mark Miller
Yup - deletes are fine. On Thu, Feb 16, 2012 at 8:56 PM, Jamie Johnson jej2...@gmail.com wrote: With solr-2358 being committed to trunk do deletes and updates get distributed/routed like adds do? Also when a down shard comes back up are the deletes/updates forwarded as well? Reading the jira