Re: Dead node, but clusterstate.json says active, won't sync on restart

2014-01-29 Thread Greg Preston
to merge the live nodes info with the state to know the real state. - Mark http://www.about.me/markrmiller On Jan 28, 2014, at 12:31 PM, Greg Preston gpres...@marinsoftware.com wrote: ** Using solrcloud 4.4.0 ** I had to kill a running solrcloud node. There is still a replica

Re: Dead node, but clusterstate.json says active, won't sync on restart

2014-01-29 Thread Greg Preston
: What's in the logs of the node that won't recover on restart after clearing the index and tlog - Mark On Jan 29, 2014, at 11:41 AM, Greg Preston gpres...@marinsoftware.com wrote: If you removed the tlog and index and restart it should resync, or something is really crazy

Dead node, but clusterstate.json says active, won't sync on restart

2014-01-28 Thread Greg Preston
** Using solrcloud 4.4.0 ** I had to kill a running solrcloud node. There is still a replica for that shard, so everything is functional. We've done some indexing while the node was killed. I'd like to bring back up the downed node and have it resync from the other replica. But when I restart

Re: Dead node, but clusterstate.json says active, won't sync on restart

2014-01-28 Thread Greg Preston
) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:219) -Greg On Tue, Jan 28, 2014 at 9:53 AM, Shawn Heisey s...@elyograg.org wrote: On 1/28/2014 10:31 AM, Greg Preston wrote: ** Using solrcloud 4.4.0 ** I had to kill a running solrcloud node. There is still a replica

Re: Possible memory leak after segment merge? (related to DocValues?)

2013-12-30 Thread Greg Preston
you very much for the suggestion. -Greg On Fri, Dec 27, 2013 at 4:25 AM, Michael McCandless luc...@mikemccandless.com wrote: Likely this is for field norms, which use doc values under the hood. Mike McCandless http://blog.mikemccandless.com On Thu, Dec 26, 2013 at 5:03 PM, Greg Preston

Re: Possible memory leak after segment merge? (related to DocValues?)

2013-12-27 Thread Greg Preston
, which use doc values under the hood. Mike McCandless http://blog.mikemccandless.com On Thu, Dec 26, 2013 at 5:03 PM, Greg Preston gpres...@marinsoftware.com wrote: Does anybody with knowledge of solr internals know why I'm seeing instances of Lucene42DocValuesProducer when I don't have any

Re: Possible memory leak after segment merge? (related to DocValues?)

2013-12-26 Thread Greg Preston
Does anybody with knowledge of solr internals know why I'm seeing instances of Lucene42DocValuesProducer when I don't have any fields that are using DocValues? Or am I misunderstanding what this class is for? -Greg On Mon, Dec 23, 2013 at 12:07 PM, Greg Preston gpres...@marinsoftware.com wrote

Possible memory leak after segment merge? (related to DocValues?)

2013-12-23 Thread Greg Preston
Hello, I'm loading up our solr cloud with data (from a solrj client) and running into a weird memory issue. I can reliably reproduce the problem. - Using Solr Cloud 4.4.0 (also replicated with 4.6.0) - 24 solr nodes (one shard each), spread across 3 physical hosts, each host has 256G of memory

Re: Possible memory leak after segment merge? (related to DocValues?)

2013-12-23 Thread Greg Preston
possible the OOM errors will stop occurring. Joel Joel Bernstein Search Engineer at Heliosearch On Mon, Dec 23, 2013 at 3:07 PM, Greg Preston gpres...@marinsoftware.comwrote: Hello, I'm loading up our solr cloud with data (from a solrj client) and running into a weird memory issue

Re: adding a node to SolrCloud

2013-12-23 Thread Greg Preston
Yes, I'm well aware of the performance implications, many of which are mitigated by 2TB of SSD and 512GB RAM I've got a very similar setup in production. 2TB SSD, 256G RAM (128G heaps), and 1 - 1.5 TB of index per node. We're in the process of splitting that to multiple JVMs per host. GC

Re: Possible memory leak after segment merge? (related to DocValues?)

2013-12-23 Thread Greg Preston
. Then watch the memory as the replaying occurs with the smaller tlog. Joel Joel Bernstein Search Engineer at Heliosearch On Mon, Dec 23, 2013 at 4:17 PM, Greg Preston gpres...@marinsoftware.comwrote: Hi Joel, Thanks for the suggestion. I could see how decreasing autoCommit time

Re: adding a node to SolrCloud

2013-12-23 Thread Greg Preston
, make sure you don't define a non-relative dataDir in solrconfig.xml, or you may run into issues with cores trying to use the same data dir. -Greg On Mon, Dec 23, 2013 at 2:16 PM, David Santamauro david.santama...@gmail.com wrote: On 12/23/2013 05:03 PM, Greg Preston wrote: Yes, I'm well

How to always tokenize on underscore?

2013-09-25 Thread Greg Preston
[Using SolrCloud 4.4.0] I have a text field where the data will sometimes be delimited by whitespace, and sometimes by underscore. For example, both of the following are possible input values: Group_EN_1000232142_blah_1000232142abc_foo Group EN 1000232142 blah 1000232142abc foo What I'd like

Re: How to always tokenize on underscore?

2013-09-25 Thread Greg Preston
Krupansky -Original Message- From: Greg Preston Sent: Wednesday, September 25, 2013 5:43 PM To: solr-user@lucene.apache.org Subject: How to always tokenize on underscore? [Using SolrCloud 4.4.0] I have a text field where the data will sometimes be delimited by whitespace

Re: Solr 4.3: Recovering from Too many values for UnInvertedField faceting on field

2013-09-03 Thread Greg Preston
Our index is too large to uninvert on the fly, so we've been looking into using DocValues to keep a particular field uninverted at index time. See http://wiki.apache.org/solr/DocValues I don't know if this will solve your problem, but it might be worth trying it out. -Greg On Tue, Sep 3, 2013

Re: Question about SOLR-5017 - Allow sharding based on the value of a field

2013-08-28 Thread Greg Preston
I don't know about SOLR-5017, but why don't you want to use parent_id as a shard key? So if you've got a doc with a key of abc123 and a parent_id of 456, just use a key of 456!abc123 and all docs with the same parent_id will go to the same shard. We're doing something similar and limiting

Shard splitting error: cannot uncache file=_1.nvm

2013-08-27 Thread Greg Preston
I haven't been able to successfully split a shard with Solr 4.4.0 If I have an empty index, or all documents would go to one side of the split, I hit SOLR-5144. But if I avoid that case, I consistently get this error: 290391 [qtp243983770-60] INFO

Re: SOLR Prevent solr of modifying fields when update doc

2013-08-24 Thread Greg Preston
the complete document. On 08/23/2013 09:03 AM, Greg Preston wrote: Perhaps an atomic update that only changes the fields you want to change? -Greg On Fri, Aug 23, 2013 at 4:16 AM, Luís Portela Afonso meligalet...@gmail.com wrote: Hi thanks by the answer, but the uniqueId is generated by me

Re: SOLR Prevent solr of modifying fields when update doc

2013-08-23 Thread Greg Preston
Perhaps an atomic update that only changes the fields you want to change? -Greg On Fri, Aug 23, 2013 at 4:16 AM, Luís Portela Afonso meligalet...@gmail.com wrote: Hi thanks by the answer, but the uniqueId is generated by me. But when solr indexes and there is an update in a doc, it deletes

Autosuggest on very large index

2013-08-20 Thread Greg Preston
Using 4.4.0 - I would like to be able to do an autosuggest query against one of the fields in our index and have the results be limited by an fq. I can get exactly the results I want with a facet query using a facet.prefix, but the first query takes ~5 minutes to run on our QA env (~240M docs).

Re: Autosuggest on very large index

2013-08-20 Thread Greg Preston
The filter query would be on a different field (clientId) than the field we want to autosuggest on (title). Or are you proposing we index a compound field that would be clientId+titleTokens so we would then prefix the suggester with clientId+userInput ? Interesting idea. -Greg On Tue, Aug 20,

Re: Autosuggest on very large index

2013-08-20 Thread Greg Preston
a lot more hardware. 2. You need to scale back your ambitions. -- Jack Krupansky -Original Message- From: Greg Preston Sent: Tuesday, August 20, 2013 2:00 PM To: solr-user@lucene.apache.org Subject: Autosuggest on very large index Using 4.4.0 - I would like to be able to do

Re: Getting the shard a document lives on in resultset

2013-08-20 Thread Greg Preston
I know I've done this in a search via the admin console, but I can't remember/find the exact syntax right now... -Greg On Tue, Aug 20, 2013 at 12:56 PM, AdamP adamph...@gmail.com wrote: Hi, We have several shards which we're querying across using distributed search. This initial search only

Re: Getting the shard a document lives on in resultset

2013-08-20 Thread Greg Preston
Found it. Add [shard] to your fl. -Greg On Tue, Aug 20, 2013 at 1:24 PM, Greg Preston gpres...@marinsoftware.com wrote: I know I've done this in a search via the admin console, but I can't remember/find the exact syntax right now... -Greg On Tue, Aug 20, 2013 at 12:56 PM, AdamP adamph

Re: Share splitting at 23 million documents - OOM

2013-08-16 Thread Greg Preston
Have you tried it with a smaller number of documents? I haven't been able to successfully split a shard with 4.4.0 with even a handful of docs. -Greg On Fri, Aug 16, 2013 at 7:09 AM, Harald Kirsch harald.kir...@raytion.comwrote: Hi all. Using the example setup of solr-4.4.0, I was able to

Re: Split Shard Error - maxValue must be non-negative

2013-08-13 Thread Greg Preston
I'm running into the same issue using composite routing keys when all of the shard keys end up in one of the subshards. -Greg On Tue, Aug 13, 2013 at 9:34 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: Scratch that. I obviously didn't pay attention to the stack trace. There is no

[4.4.0] Shard splitting failure (simplified case)

2013-08-12 Thread Greg Preston
I've simplified things from my previous email, and I'm still seeing errors. Using solr 4.4.0 with two nodes, starting with a single shard. Collection is named marin, host names are dumbo and solrcloud1. I bring up an empty cloud and index 50 documents. I can query them and everything looks

Re: What gets written to the other shards?

2013-08-12 Thread Greg Preston
Are you manually setting the shard on each document? If not, documents will be hashed across all the shards. -Greg On Mon, Aug 12, 2013 at 3:50 PM, Thierry Thelliez thierry.thelliez.t...@gmail.com wrote: Hello, I am trying to set a four shard system for the first time. I do not

Re: Shard splitting failure, with and without composite hashing

2013-08-11 Thread Greg Preston
attention than 4.1 FWIW, Erick On Fri, Aug 9, 2013 at 7:32 PM, Greg Preston gpres...@marinsoftware.com wrote: Howdy, I'm trying to test shard splitting, and it's not working for me. I've got a 4 node cloud with a single collection and 2 shards. I've indexed 170k small documents

Shard splitting failure, with and without composite hashing

2013-08-09 Thread Greg Preston
Howdy, I'm trying to test shard splitting, and it's not working for me. I've got a 4 node cloud with a single collection and 2 shards. I've indexed 170k small documents, and I'm using the compositeId router, with an internal client id as the shard key, with 4 distinct values across the data