Re: OutOfMemoryError

2013-03-27 Thread Arkadi Colson
I upgraded java to version 7 and everything seems to be stable now! BR, Arkadi On 03/25/2013 09:54 PM, Shawn Heisey wrote: On 3/25/2013 1:34 AM, Arkadi Colson wrote: I changed my system memory to 12GB. Solr now gets -Xms2048m -Xmx8192m as parameters. I also added -XX:+UseG1GC to the java

Disc space and replication

2013-03-27 Thread Arkadi Colson
Hi When replication is down for some time or an instance crashed for some reason replication will always start over again from the beginning. This means it will copy the whole shard over of about 150GB. So we need at least a disc of about 300 GB. I've read somewhere that Solr will

Elasticsearch with kerberos

2013-03-27 Thread Debika Mukherjee
Hi, Is there any integration of Solr with Kerberos? Thanks and regards, Debika Mukherjee CLOUD BBSR VOIP 6743071561 CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are

Re: [ScriptUpdateProcessor] Params aren't being picked up from solrconfig

2013-03-27 Thread Rene Nederhand
I cannot believe I've looked over this :} Thanks for helping me out. It works fine now. I'd like to contribute to the wiki pagehttp://wiki.apache.org/solr/ScriptUpdateProcessorand add a python example. So, if anyone could allow me write access or tell me how to do this without, I'd be happy to

Re: [ScriptUpdateProcessor] Params aren't being picked up from solrconfig

2013-03-27 Thread Steve Rowe
Hi Rene, Thanks for offering to help with wiki documentation. You'll need to register on the wiki first, then tell us your wiki username, and we'll add you to ContributorsGroup, which will allow you to make edits. Steve On Mar 27, 2013, at 7:40 AM, Rene Nederhand r...@nederhand.net wrote: I

Re: Disc space and replication

2013-03-27 Thread Mark Miller
On Mar 27, 2013, at 3:57 AM, Arkadi Colson ark...@smartbit.be wrote: Hi When replication is down for some time or an instance crashed for some reason replication will always start over again from the beginning. This means it will copy the whole shard over of about 150GB. So we need at

Re: Solrcloud 4.1 Collection with multiple slices only use

2013-03-27 Thread Chris R
So - I must be missing something very basic here and I've gone back to the Wiki example. After setting up the two shard example in the first tutorial and indexing the three example documents, look at the shards in the Admin UI. The documents are stored in the index where the update with directed

Using multiple text files for Suggestor dictionarys

2013-03-27 Thread Eric Wilson
I'm using the Suggester component for autocomplete. I have a variety of types of suggestions that I would like to offer, such as locations, company names, products, and dictionary words. These lists vary in size and volatility, so keeping them all in the same text file is not the most convenient.

Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely

2013-03-27 Thread Furkan KAMACI
Hi Nate; This may be out of topic however could you explain that why you want to use Tomcat instead of Jetty or Embedded Jetty? 2013/3/27 Michael Della Bitta michael.della.bi...@appinions.com You're using the blocking IO connector, which isn't so great for heavy loads. Give this a shot...

How do I recover the position and offset a highlight for solr (4.1/4.2)?

2013-03-27 Thread Skealler Nametic
Hi, I would like to retrieve the position and offset of each highlighting found. I searched on the internet, but I have not found the exact solution to my problem...

Re: Elasticsearch with kerberos

2013-03-27 Thread Shawn Heisey
On 3/27/2013 5:29 AM, Debika Mukherjee wrote: Is there any integration of Solr with Kerberos? I am pretty sure that the answer is no. Solr has no security features at all - it is intended to live where regular users cannot get to it. Thanks, Shawn

Querying a transitive closure?

2013-03-27 Thread Jack Park
This is a question about isA? We want to know if M isA B isA?(M,B) For some M, one might be able to look into M to see its type or which class(es) for which it is a subClass. We're talking taxonomic queries now. But, for some M, one might need to ripple up the transitive closure, looking at

Re: Elasticsearch with kerberos

2013-03-27 Thread Otis Gospodnetic
Debika, Did you really mean to ask about Solr or ElasticSearch (see subject)? I think your best bet is ManifoldCF, where I see some mention of it http://search-lucene.com/?q=kerberos Otis -- Solr ElasticSearch Support http://sematext.com/ On Wed, Mar 27, 2013 at 11:55 AM, Shawn Heisey

Re: Querying a transitive closure?

2013-03-27 Thread Otis Gospodnetic
Hi Jack, Is this really about HTTP and Solr vs. SolrCloud or more whether Solr(Cloud) is the right tool for the job and if so how to structure the schema and queries to make such lookups efficient? Otis -- Solr ElasticSearch Support http://sematext.com/ On Wed, Mar 27, 2013 at 12:53 PM,

Re: Elasticsearch with kerberos

2013-03-27 Thread Chris Hostetter
: Is there any integration of Solr with Kerberos? : I am pretty sure that the answer is no. Solr has no security features at : all - it is intended to live where regular users cannot get to it. The key question is how you define integration of Solr with Kerberos ? what is your goal? How

Solr Cloud update process

2013-03-27 Thread Walter Underwood
What do people do for updating, say from 4.1 to 4.2.1, on a live cluster? I need to help our release engineering team create the Jenkins scripts for deployment. wunder -- Walter Underwood wun...@wunderwood.org

Solr 4.1 SolrCloud with 1 shard and 3 replicas

2013-03-27 Thread Bill Au
I am running Solr 4.1. I have set up SolrCloud with 1 leader and 3 replicas, 4 nodes total. Do query requests send to a node only query the replica on that node, or are they load-balanced to the entire cluster? Bill

Re: Solr Cloud update process

2013-03-27 Thread Shawn Heisey
On 3/27/2013 12:34 PM, Walter Underwood wrote: What do people do for updating, say from 4.1 to 4.2.1, on a live cluster? I need to help our release engineering team create the Jenkins scripts for deployment. Aside from replacing the .war file and restarting your container, there hopefully

Re: Querying a transitive closure?

2013-03-27 Thread Jack Park
Hi Otis, I fully expect to grow to SolrCloud -- many shards. For now, it's solo. But, my thinking relates to cloud. I look for ways to reduce the number of HTTP round trips through SolrJ. Maybe you have some ideas? Thanks Jack On Wed, Mar 27, 2013 at 10:04 AM, Otis Gospodnetic

Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely

2013-03-27 Thread Nate Fox
Update: issue resolved! Cranking up the maxThreads did the trick. Default is 200. I went with 2500 for grins and giggles and things work great. Now, even if I overwhelm the box with too many requests, when the requests back off the box continues to respond. And when I slam the server after it's

Re: Solr 4.1 SolrCloud with 1 shard and 3 replicas

2013-03-27 Thread Mark Miller
They are load-balanced across the cluster unless you pass the distrib=false param. - Mark On Mar 27, 2013, at 2:51 PM, Bill Au bill.w...@gmail.com wrote: I am running Solr 4.1. I have set up SolrCloud with 1 leader and 3 replicas, 4 nodes total. Do query requests send to a node only query

Re: Solr 4.1 SolrCloud with 1 shard and 3 replicas

2013-03-27 Thread Erik Hatcher
Requests to a node in your example would be answered by that node (no need to distribute; it's a single shard system) and it would not internally be routed otherwise either. Ultimately it is up to the client to load-balance the initial requests into a SolrCloud cluster, but internally in a

Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely

2013-03-27 Thread Shawn Heisey
On 3/27/2013 1:16 PM, Nate Fox wrote: I have a couple quick followup questions: - What is the downside of having a maxThreads, acceptCount and maxConnections really high? Obviously defaults are there for a reason - I'd like to know what the reasoning is. - Any reason I shouldnt use Tomcat? I

Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely

2013-03-27 Thread Mark Miller
On Mar 27, 2013, at 3:29 PM, Shawn Heisey s...@elyograg.org wrote: The maxThreads parameter in the jetty config that's included with Solr is set to 1 Yonik raised this at some point if I remember right - it helps avoid some distrib deadlock issue. - Mark

Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Timothy Potter
When running in SolrCloud mode, does it make sense to disable distributed mode for warming queries? i.e. distrib=false in my warming query config I actually asked this on Erik's informative Webinar this morning but had to drop off before I heard the answer ... so Erik might have answered this

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Mark Miller
Yup. You only want to warm locally. We should add that to the wiki. - Mark On Mar 27, 2013, at 3:54 PM, Timothy Potter thelabd...@gmail.com wrote: When running in SolrCloud mode, does it make sense to disable distributed mode for warming queries? i.e. distrib=false in my warming query config

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Timothy Potter
Ok - thanks for confirming Mark - I'll add that to the wiki. Cheers, Tim On Wed, Mar 27, 2013 at 1:59 PM, Mark Miller markrmil...@gmail.com wrote: Yup. You only want to warm locally. We should add that to the wiki. - Mark On Mar 27, 2013, at 3:54 PM, Timothy Potter thelabd...@gmail.com

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread santoash
This is interesting. I'm looking into doing something similar too. Quick question: Would you be targeting each of the shard with exactly the same set of queries? On Mar 27, 2013, at 12:59 PM, Mark Miller markrmil...@gmail.com wrote: Yup. You only want to warm locally. We should add that to

Re: Solr 4.1 SolrCloud with 1 shard and 3 replicas

2013-03-27 Thread Bill Au
Thanks for the info, Erik. I had gone through the tutorial in the SolrCloud Wiki and verified that queries are load balanced in the two shard cluster with shard replicas setup. I was wondering if I need to explicitly specify distrib=false in my single shard setup. Glad to see that Solr is doing

Query on all dynamic fields or wildcard field query

2013-03-27 Thread Luis Lebolo
Hi All, First I have to apologize and admit that I'm asking this question before doing any real research =( Was hoping for some preliminary help before I start this endeavor tomorrow. So here goes: Can I query for a value in multiple (wildcarded) fields? For example, if I have dynamic fields

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Timothy Potter
In our case, yes - same non-distrib query is warmed on each node. Seems like you'd need something a little more dynamic than statically configured warming queries in solrconfig.xml for targeting specfic shards. Tim On Wed, Mar 27, 2013 at 2:04 PM, santoash santo...@me.com wrote: This is

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Joel Bernstein
This jira looks like it addresses this. https://issues.apache.org/jira/browse/SOLR-3081 I'll run a quick test. On Wed, Mar 27, 2013 at 5:41 PM, Timothy Potter thelabd...@gmail.comwrote: In our case, yes - same non-distrib query is warmed on each node. Seems like you'd need something a

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Joel Bernstein
I ran a quick test and distrib=false is being tacked on automatically. Here is the log record: INFO: [collection1] webapp=null path=null params={sort=price+ascevent=newSearcherq=solrdistrib=false} hits=1 status=0 QTime=17 So I think this is OK. On Wed, Mar 27, 2013 at 6:02 PM, Joel

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Mark Miller
Ah, interesting. Forgot about doing that issue entirely. - Mark On Mar 27, 2013, at 6:25 PM, Joel Bernstein joels...@gmail.com wrote: I ran a quick test and distrib=false is being tacked on automatically. Here is the log record: INFO: [collection1] webapp=null path=null

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Timothy Potter
lol - you know you're a bad ass when you've forgotten more about Solr cloud than the rest of us know ;-) On Wed, Mar 27, 2013 at 4:41 PM, Mark Miller markrmil...@gmail.com wrote: Ah, interesting. Forgot about doing that issue entirely. - Mark On Mar 27, 2013, at 6:25 PM, Joel Bernstein

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Joel Bernstein
That was a good fix Mark. I had this picture in my head of a large Solr Cloud sending around thousands of simultaneous searches and crashing itself. On Wed, Mar 27, 2013 at 6:47 PM, Timothy Potter thelabd...@gmail.comwrote: lol - you know you're a bad ass when you've forgotten more about Solr

Re: Query on all dynamic fields or wildcard field query

2013-03-27 Thread Jack Krupansky
No, but you can use the dismax feature of the dismax and edismax query parsers to specify a static list of any number of fields to be searched for terms in a query that do not have an explicit field specified. And, no harm filing a Jira to request support for a wildcard field search feature.

Re: Solr index Backup and restore of large indexs

2013-03-27 Thread Joel Bernstein
Hi, Are you running Solr Cloud or Master/Slave? I'm assuming with 1TB a day you're sharding. With master/slave you can configure incremental index replication to another core. The backup core can be local on the server, on a separate sever or in a separate data center. With Solr Cloud replicas

Solr sorting and relevance

2013-03-27 Thread scallawa
We are using solr for search on our ecommerce site that primarily sells clothing. We index search terms based on a title field and description field. We want to be able to sort by most relevant and what we have more inventory (there is a field for that). We have done some coding outside of

Re: Solrcloud 4.1 Collection with multiple slices only use

2013-03-27 Thread Erick Erickson
First, three documents isn't enough to really test. The formula for assigning shards is to hash on the unique ID. It _is_ possible that all three just happened to land on the same shard. If you index all 32 docs in the example dir and they're all on the same shard, we should talk. Second, a

Re: Warming queries and Solr Cloud - just curious ...

2013-03-27 Thread Erick Erickson
Tim: Unfortunately, due to the increase in spam pages from bots, we had to lock down the Solr wiki. Post a request for us to add your Wiki ID (and give us the ID!) to the list of authorized IDs and we'll get you added (just takes a second). Or send me (or Steve Rowe) a private e-mail if you'd

Re: Solrcloud 4.1 Collection with multiple slices only use

2013-03-27 Thread corgone
I realized my error shortly, more docs, better spread. I continued to do some testing to see how I could manually lay out the shards in what I thought was a more organized manner and with more descriptive names than the numshards parameter alone produced. I also gen'd up a few thousand docs

Re: Querying a transitive closure?

2013-03-27 Thread Otis Gospodnetic
Hi Jack, I don't fully understand the exact taxonomy structure and your needs, but in terms of reducing the number of HTTP round trips, you can do it by writing a custom SearchComponent that, upon getting the initial request, does everything locally, meaning that it talks to the local/specified

Re: Querying a transitive closure?

2013-03-27 Thread Jack Park
Hi Otis, That's essentially the answer I was looking for: each shard (are we talking master + replicas?) has the plug-in custom query handler. I need to build it to find out. What I mean is that there is a taxonomy, say one with a single root for sake of illustration, which grows all the

Re: Solr sorting and relevance

2013-03-27 Thread Joel Bernstein
It sounds like you might be able to get the mix you want with three different boosts: 1) High boost on title 2) Lower boost on description 3) Function query boost on inventory The high boost on title will help push products with matches in the title to the top. The function query boost on

Re: Could not load config for solrconfig.xml

2013-03-27 Thread A. Lotfi
  Hi Hoss, Thank you for replying to my question, The solrconfig.xml in the example-DIH in solr download is exactly the same like the links you posted in your reply, so where is the big difference ? I think I typed a mistake in my last question, instead of saying db-data-config.xml I said

Could not load config for solrconfig.xml

2013-03-27 Thread A. Lotfi
Hi, I am trying solr with an oracle database, It's working but I have on the top of the page an exception : SolrCore Initialization Failures  solr:  org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not load config for solrconfig.xml Here is my

Re: Too many fields to Sort in Solr

2013-03-27 Thread adityab
Hi Joel, you are correct, boost function populates the field cache. Well i am not aware of docValue, so while trying the example you provided i see the error when i define the field type Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is configured with a docValues format,