Question about optimize call - Request read Timeout

2011-12-02 Thread Torsten Krah
Hi, got a question about index optimizing. At midnight i am calling optimize(true, true) on my SolrServer instance. However this does fail with: org.apache.solr.client.solrj.SolrServerException: java.net.SocketTimeoutException: Read timed out at

Re: Multithreaded DIH bug

2011-12-02 Thread Mikhail Khludnev
Hello, AFAIK Particularly this exception is not a big deal. It's just one of the evidence of the fact that CachedSqlEntityProcessor doesn't work in multiple threads at 3.x and 4.0. It's discussed at http://search-lucene.com/m/0DNn32L2UBv the most problem here is the following messages in the log

Solr's FieldValueCache and Lucene's FieldCache

2011-12-02 Thread RT RT
Hi, I'm trying to understand caching, looking on the wiki I can see that Solr directly or indirectly uses Lucene's fieldCache and also it has it's own fieldValueCache. They seem to store the same entries - field values. How does Solr use both of these caches, or are they the same thing? 

Best practise to automatically change a field value for a specific period of time

2011-12-02 Thread Mark Schoy
Hi, I have an solr index for an online shop with a field price which contains the standard price of a product. But in the database, the shop owner can specify a period of time with an alternative price. For example: standard price is $20.00, but 12/24/11 08:00am to 12/26/11 11:59pm = $12.59 Of

Memory Leak in Solr?

2011-12-02 Thread Samarendra Pratap
Hi everyone, A couple of days back I encountered a weird problem of continuously increasing memory consumption. I am not sure if this is a problem of java or Solr (3.4). Till 3 days ago, we were running Solr 3.4 instance with following java command line options java -server -*Xms2048m*

Re: Best practise to automatically change a field value for a specific period of time

2011-12-02 Thread darren
Solr doesn't support these kind of business rules inside of it. Not intended to. Thusly, you will have to manage them externally. What's unstable about a cronjob? You will have to run your business rules externally, then apply the necessary field updates to the documents in Solr, ensuring the

Re: Best practise to automatically change a field value for a specific period of time

2011-12-02 Thread Michael Kuhlmann
Hi Mark, I'm sure you can manage this using function queries somehow, but this is rather complicated, esp. if you both want to return the price and sort on it. I'd rather update the index as soon as a campaign starts or ends. At least that's how we did it when I worked for online shops.

Re: Dealing with dashes with solr.PatternReplaceCharFilterFactory

2011-12-02 Thread Koji Sekiguchi
(11/12/02 4:20), Aaron Wong wrote: Hi all, We're encountering a problem with querying terms with dashes (and other non-alphanumeric characters). For example, we use PatternReplaceCharFilterFactory to replace dashes with blank characters for both index and query, however any terms with dashes in

Re: Best practise to automatically change a field value for a specific period of time

2011-12-02 Thread Morten Lied Johansen
On 02. des. 2011 12:21, Mark Schoy wrote: Hi, I have an solr index for an online shop with a field price which contains the standard price of a product. But in the database, the shop owner can specify a period of time with an alternative price. For example: standard price is $20.00, but

Re: Best practise to automatically change a field value for a specific period of time

2011-12-02 Thread Mathias Hodler
Hi Morten, thanks, this is a very good solution. I also found another solution: Creating a custom ValueSourceParser for price sorting which considered the standard price and the campaign price. In my special case I think your approach isn't working, because i also need result grouping and this

Re: Best practise to automatically change a field value for a specific period of time

2011-12-02 Thread Mark Schoy
Hi Morten, thanks, this is a very good solution. I also found another solution:Creating a custom ValueSourceParser for price sorting which consideredthe standard price and the campaign price. In my special case I think your approach isn't working, because i alsoneed result grouping and this cant

Re: HBase Datasource

2011-12-02 Thread Gian Marco Tagliani
Hi, In my company we have the same need, import from Hbase into Solr We just started a project here: http://code.google.com/p/hbase-solr-dataimport/ We want to provide an easy way to import data from HBase, similar to SqlEntityProcessor Gian Marco On Fri, Nov 11, 2011 at 4:37 AM, Fuad

Re: Configuring the Distributed

2011-12-02 Thread Mark Miller
So I dunno. You are running a zk server and running in zk mode right? You don't need to / shouldn't set a shards or self param. The shards are figured out from Zookeeper. You always want to use the distrib-update-chain. Eventually it will probably be part of the default chain and auto turn in zk

Re: highlight issue

2011-12-02 Thread Ravish Bhagdev
Also, not entirely sure wild-cards are supported in text based fields, only on strings. Although things may have changed in recent versions of Solr, I am not sure. R On Thu, Dec 1, 2011 at 3:55 PM, Koji Sekiguchi k...@r.email.ne.jp wrote: Suppose my search query is *Rak*.In my database i have

Re: Multithreaded DIH bug

2011-12-02 Thread Mark
Thanks for the info On 12/2/11 1:29 AM, Mikhail Khludnev wrote: Hello, AFAIK Particularly this exception is not a big deal. It's just one of the evidence of the fact that CachedSqlEntityProcessor doesn't work in multiple threads at 3.x and 4.0. It's discussed at

Re: Configuring the Distributed

2011-12-02 Thread Jamie Johnson
Glad to hear I don't need to set shards/self, but removing them didn't seem to change what I'm seeing. Doing this still results in 2 documents 1 on 8983 and 1 on 7574. String key = 1; SolrInputDocument solrDoc = new SolrInputDocument(); solrDoc.setField(key,

Simplification of boolean query failed

2011-12-02 Thread Mark D Sievers
I've put the question nicely formatted on StackOverflow here http://stackoverflow.com/questions/8360257/solr-lucene-why-is-this-or-query-failing-when-the-two-individual-queries-suc Here is that question Verbatim: I have a Solr document schema with with a solr.TrieDateField and noticed this

Re: Configuring the Distributed

2011-12-02 Thread Mark Miller
They are unused params, so removing them wouldn't help anything. You might just want to wait till we are further along before playing with it. Or if you submit your full self contained test, I can see what's going on (eg its still unclear if you have started setting numShards?). I can do a

Re: Configuring the Distributed

2011-12-02 Thread Jamie Johnson
So I'm a fool. I did set the numShards, the issue was so trivial it's embarrassing. I did indeed have it setup as a replica, the shard names in solr.xml were both shard1. This worked as I expected now. On Fri, Dec 2, 2011 at 1:02 PM, Mark Miller markrmil...@gmail.com wrote: They are unused

Multivalued field

2011-12-02 Thread Alan Miller
Hi I have a webapp that plots a bunch of time series Data which are just doubles coupled with a timestamp Every chart in my webapp has a reportid in my db and i am wondering if it would be effective to usr solr to serve the data th my app instead of keeping the data in my rdbms. Currently im

Solr Suggester - building terms from both a field and dictionary

2011-12-02 Thread Jithin
Hi, I have a use case where in I need to provide autocomplete from both values in a index field and a dictionary file. I am planning to use Solr Suggester. On reading the documentation I am getting the idea that terms can be either from a field or from a dictionary but not both. Can this behavior

Re: Out of memory during the indexing

2011-12-02 Thread Jeff Crump
Can anyone advise techniques for limiting the size of the RAM buffers to begin with? As the index grows, I shouldn't have to keep increasing the heap. We have a high-ingest, low-query-rate environment and I'm not as much concerned with the query-time caches as I am with the segment core

Solr Suggester - perform case insensitive search

2011-12-02 Thread Jithin
Hi, Is it possible to do case insensitive suggestions via Solr suggester. On reading the documentation it seems like there is no option for that. Can anyone please give suggestions on how to deal with this. -- View this message in context:

Re: Huge Performance: Solr distributed search

2011-12-02 Thread Tom Gullo
Interesting info. You should look into using Solid State Drives. I moved my search engine to SSD and saw dramatic improvements. -- View this message in context: http://lucene.472066.n3.nabble.com/Huge-Performance-Solr-distributed-search-tp3530627p346.html Sent from the Solr - User

Problem Commiting to Large Index

2011-12-02 Thread Marty Trenouth
We are creating an index of about 500 - 600M records fairly small documents. We are currently @ 252M+ records and adding documents at a rate of about 2k - 3k per second in multithreaded 1K batches sent from multiple servers. Our commits started bombing out with out of Memory Exceptions with

Re: Solr 3.5 very slow (performance)

2011-12-02 Thread Chris Hostetter
: But I stil don't understand why in solr 1.4 that option was efficient : but in solr 3.5 not Good question ... i'm not really sure. Frankly i forgot that option even existed (i missread your stack trace and thought it was just generating the DocSet for faceting ... that's why i wanted to see

Re: Configuring the Distributed

2011-12-02 Thread Mark Miller
Ah, okay - you are setting the shards in solr.xml - thats still an option to force a node to a particular shard - but if you take that out, shards will be auto assigned. By the way, because of the version code, distrib deletes don't work at the moment - will get to that next week. - Mark On

Re: Integrating Surround Query Parser

2011-12-02 Thread simon
Take a look at https://issues.apache.org/jira/browse/SOLR-2703, which integrates the surround parser into Solr trunk. There's a dependency on a Lucene patch which resolves a caching problem ( https://issues.apache.org/jira/browse/LUCENE-2945 ) which also wasn't backported to earlier versions of

Re: Integrating Surround Query Parser

2011-12-02 Thread simon
oops, didn't see all of the thread before I hit send. Good work, Erik On Fri, Dec 2, 2011 at 5:21 PM, simon mtnes...@gmail.com wrote: Take a look at https://issues.apache.org/jira/browse/SOLR-2703, which integrates the surround parser into Solr trunk. There's a dependency on a Lucene patch

Re: Memory Leak in Solr?

2011-12-02 Thread Chris Hostetter
: Till 3 days ago, we were running Solr 3.4 instance with following java : command line options : java -server -*Xms2048m* -*Xmx4096m* -Dsolr.solr.home=etc -jar start.jar : : Then we increased the memory with following options and restarted the : server : java -server *-**Xms4096m* -*Xmx10g*

Re: Boosted documents not appearing higher than less-boosted ones for equal relevancy.

2011-12-02 Thread Chris Hostetter
: I don't think there is a way of seeing the boosts from the index, as : those are encoded as norms (together with length normalization). You can : see the norms with Luke if you want to and in the debugQuery output the ... :

Re: Configuring the Distributed

2011-12-02 Thread Jamie Johnson
How does it determine the number of shards to create? How many replicas to create? On Fri, Dec 2, 2011 at 4:30 PM, Mark Miller markrmil...@gmail.com wrote: Ah, okay - you are setting the shards in solr.xml - thats still an option to force a node to a particular shard - but if you take that

Re: Configuring the Distributed

2011-12-02 Thread Jamie Johnson
I think I see it.so if I understand this correctly you specify numShards as a system property, as new nodes come up they check ZK to see if they should be a new shard or a replica based on if numShards is met. A few questions if a master goes down does a replica get promoted? If a new shard

Re: Configuring the Distributed

2011-12-02 Thread Jamie Johnson
So I just tried this out, seems like it does the things I asked about. Really really cool stuff, it's progressed quite a bit in the time since I took a snapshot of the branch. Last question, how do you change numShards? Is there a command you can use to do this now? I understand there will be