IndexWrite in Lucene/Solr 3.5 is slower?
We are upgrading our search infrastructure from Lucene 2.3.1 to Lucene 3.5. I am in the process of load testing and I could find that Lucene 2.3.1 could index 32,000 docs per second, whereas Lucene 3.5 could index only around 17,000 docs per second. Indeed, both of them use the standard analyzer and the default settings. Is 3.5 slower because it indexes more details and thereby resulting in a faster search? Ours is a log management product and the speed of indexing is highly important. Ok, cutting the long story short, will the slower indexing of 3.5 result in a higher search speed?, if not, what else should I fine tune to improve the indexing speed? -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420
Re: IndexWrite in Lucene/Solr 3.5 is slower?
On Fri, Jun 15, 2012 at 12:20 PM, pravesh suyalprav...@yahoo.com wrote: BTW, Have you changed the MergePolicy MergeScheduler settings also? Since Lucene 3.x/3.5 onwards, there have been new MergePolicy MergeScheduler implementations available, like TieredMergePolicy ConcurrentMergeScheduler. Regards Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/IndexWrite-in-Lucene-Solr-3-5-is-slower-tp3989764p3989768.html Sent from the Solr - User mailing list archive at Nabble.com. Thanks for the reply Pravesh. Yes I initially used the default TieredMergePolicy and later set the merge policy in both the versions to LogByteSizeMergePolicy, in order to maintain congruence. But still Lucene 3.5 lagged behind by 2X approx. -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420
Re: IndexWrite in Lucene/Solr 3.5 is slower?
On Fri, Jun 15, 2012 at 12:50 PM, Ramprakash Ramamoorthy youngestachie...@gmail.com wrote: On Fri, Jun 15, 2012 at 12:20 PM, pravesh suyalprav...@yahoo.com wrote: BTW, Have you changed the MergePolicy MergeScheduler settings also? Since Lucene 3.x/3.5 onwards, there have been new MergePolicy MergeScheduler implementations available, like TieredMergePolicy ConcurrentMergeScheduler. Regards Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/IndexWrite-in-Lucene-Solr-3-5-is-slower-tp3989764p3989768.html Sent from the Solr - User mailing list archive at Nabble.com. Thanks for the reply Pravesh. Yes I initially used the default TieredMergePolicy and later set the merge policy in both the versions to LogByteSizeMergePolicy, in order to maintain congruence. But still Lucene 3.5 lagged behind by 2X approx. -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420 Can someone help me with this please? -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420
Porting from Solr 1.3 to 3.5
I am porting my app from lucene 2.X(solr 1.3) to lucene 3.X(solr 3.5). The following is my issue. This one was valid in 2.X, but 3.5 throws me an error. IndexReader reader = IndexReader.open(/home/path/to/my/dataDir); 2.X accepted a string, but 3.5 strictly wants a Directory object. I find Directory to be abstract and the only way to instantiate it seems a RAMDirectory(). How do I go about this and how do I point my reader to the desired directory? P.S : Our application needs a custom logic this way and hence instead of going with cores, we do it this way. -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420
Re: Porting from Solr 1.3 to 3.5
On Thu, Jun 7, 2012 at 1:18 PM, Ramprakash Ramamoorthy youngestachie...@gmail.com wrote: I am porting my app from lucene 2.X(solr 1.3) to lucene 3.X(solr 3.5). The following is my issue. This one was valid in 2.X, but 3.5 throws me an error. IndexReader reader = IndexReader.open(/home/path/to/my/dataDir); 2.X accepted a string, but 3.5 strictly wants a Directory object. I find Directory to be abstract and the only way to instantiate it seems a RAMDirectory(). How do I go about this and how do I point my reader to the desired directory? P.S : Our application needs a custom logic this way and hence instead of going with cores, we do it this way. -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420 I was able to do it. I just did it this way IndexReader reader = new SimpleFSDirectory(new File(my/desired/path)); Thanks for your time. -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420
Re: Problem with query negation
On Fri, May 18, 2012 at 5:03 PM, Ahmet Arslan iori...@yahoo.com wrote: I am trying the following query and get only zero results (I am supposed to get 10 results according to my dataset) *http://mymachine:8983/solr/select/?q=-(HOSTID:302)* I also tried the below query and got zero results yet again. *http://mymachine:8983/solr/select/?q=NOT(HOSTID:302)* However, I get 10 results(expected) when I put the query this way, *http://mymachine:8983/solr/select/?q=-(HOSTID:302)AND(*:*)* Why is this strange thing happening? Is it a bug in solr or am I missing something? Why don't you just use /solr/select/?q=-HOSTID:302 Tried the same right at start, but never worked :( -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420
Re: Problem with query negation
On Fri, May 18, 2012 at 6:03 PM, Ahmet Arslan iori...@yahoo.com wrote: Why don't you just use /solr/select/?q=-HOSTID:302 Tried the same right at start, but never worked :( q=-HOSTID:302 and q=+*:* -HOSTID:302 should return same result set. Which solr version and query parser are you using? I am using the standard LuceneQParserPlugin and I have a custom request handler. I use solr 3.5 -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420
Re: Problem with query negation
On Fri, May 18, 2012 at 6:20 PM, Ahmet Arslan iori...@yahoo.com wrote: I am using the standard LuceneQParserPlugin and I have a custom request handler. I use solr 3.5 I would test the same query with standard request handler. May be something is your custom request handler? I don't think request handler should be a problem. I have just used the *q *parameter as follows. String q = params.get(CommonParams.Q); IndexSchema schema = req.getSchema(); Query query = new QueryParsing().parseQuery(q, schema); Hope there shouldn't be a problem with the above! debugQuery=on would help too. -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420
Re: Merging two DocSets in solr
On Sun, May 13, 2012 at 4:45 PM, Dmitry Kan dmitry@gmail.com wrote: Are you operating inside the SOLR source code or on the (solrj) client side? SOLR source code! On Fri, May 11, 2012 at 12:46 PM, Ramprakash Ramamoorthy youngestachie...@gmail.com wrote: Dear all, I get two different DocSets from two different searchers. I need to merge them into one and get the facet counts from the merged docSets. How do I do it? Any pointers would be appreciated. -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420 -- Regards, Dmitry Kan -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420
Re: Problem with query negation
On Fri, May 18, 2012 at 7:26 PM, Ahmet Arslan iori...@yahoo.com wrote:
I don't think request handler should be a problem. I
have just used the *q
*parameter as follows.
String q = params.get(CommonParams.Q);
IndexSchema schema = req.getSchema();
Query query = new QueryParsing().parseQuery(q, schema);
Hope there shouldn't be a problem with the above!
Solr converts top level negative query (-field:something) into q=+*:*
-field:something
It seems that you are missing that part.
org.apache.solr.search.QueryUtils
/** Fixes a negative query by adding a MatchAllDocs query clause.
* The query passed in *must* be a negative query.
*/
public static Query fixNegativeQuery(Query q) {
BooleanQuery newBq = (BooleanQuery)q.clone();
newBq.add(new MatchAllDocsQuery(), BooleanClause.Occur.MUST);
return newBq;
}
Oh thats great :)
Thank you very much! You made my day :)
--
With Thanks and Regards,
Ramprakash Ramamoorthy,
Project Trainee,
Zoho Corporation.
+91 9626975420
Merging two DocSets in solr
Dear all, I get two different DocSets from two different searchers. I need to merge them into one and get the facet counts from the merged docSets. How do I do it? Any pointers would be appreciated. -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420
Dynamically pick dataDirs
Dear all, I am using solr for log search. During every search, based on the input request, I will have to search through n index directories dynamically. n may range from 1 to 100. While n being a higher value, firing 100 cores wouldn't be a viable solution. How do I achiever this in solr, in short I would like to have a single core and get results out of multiple index searchers and that implies multiple index readers. This happens in a single machine and I will have to perform a distributed search across m machines like this, while m ranging from 2 to 20. Is there any way to do it graciously? -- With Thanks and Regards, Ramprakash Ramamoorthy, Engineer Trainee, Zoho Corporation. +91 9626975420
Re: Dynamically pick dataDirs
On Wed, May 9, 2012 at 3:26 PM, pravesh suyalprav...@yahoo.com wrote: While n being a higher value, firing 100 cores wouldn't be a viable solution. How do I achiever this in solr, in short I would like to have a single core and get results out of multiple index searchers and that implies multiple index readers. When you'd want to have single core with multiple index directories (which currently is not supported by SOLR), then why can't you have a single merged index within the core. Lucene supports searching through multiple indexes but this hasn't been inherited by the SOLR by design (I mean using MultiSearcher API's for a single core with multiple index directories in it). BTW, how big your index(es) are? Total documents? total size? etc. If each core is small(MBs/ few GBs) then you could merge few of them together. Regards Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/Dynamically-pick-dataDirs-tp3973368p3973682.html Sent from the Solr - User mailing list archive at Nabble.com. Dear Pravesh, Thanks for the response. As I said earlier, I am using this for a log search activity. Every index has around 3,00,000 docs in an average and size spans from 1Gb to 3Gb. Everyday we zip the log indices and in case the user asks for logs from previous dates, then we unzip an index into a particular directory and then allow him to search (this is via lucene, we are moving into solr for adding distributed search functionality). In our product we have the option to search upto 6 months (~180 days), in that case merging 180 indices wouldn't be that easy I guess. -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420
Re: PolySearcher in Solr
On Thu, Apr 19, 2012 at 9:21 PM, Jeevanandam Madanagopal je...@myjeeva.comwrote: Please have a look http://wiki.apache.org/solr/DistributedSearch -Jeevanandam On Apr 19, 2012, at 9:14 PM, Ramprakash Ramamoorthy wrote: Dear all, I came across this while browsing through lucy http://lucy.apache.org/docs/perl/Lucy/Search/PolySearcher.html Does solr have an equivalent of this? My usecase is exactly the same (reading through multiple indices in a single shard and perform a distribution across shards). If not can someone give me a hint? I tried swapping readers for a single searcher, but didn't help. -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420 Dear Jeevanandam, Thanks for the response, but come on, I am aware of it. Try reading my mail again. I will have to read through multiple indices in a single shard, and have a distributed search across all shards. -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420
Convert a SolrDocumentList to DocList
Dear all, Is there any way I can convert a SolrDocumentList to a DocList and set it in the QueryResult object? Or, the workaround adding a SolrDocumentList object to the QueryResult object? -- With Thanks and Regards, Ramprakash Ramamoorthy, Project Trainee, Zoho Corporation. +91 9626975420
Jira 1540
I am using solr to perform a distributed search. I am using version 1.3 to accommodate older indices in the already existing system. I am able to perform a search over a single shard, even faceting and highlighting works. However, when it comes to distributed search, I get an exception 500. The stacktrace is almost similar to the one mentioned in https://issues.apache.org/jira/browse/SOLR-1540 . The issue is also almost the same. Any idea/fix to tackle? -- With Thanks and Regards, Ramprakash Ramamoorthy, SASTRA University.
