Re: How to improve the performance of query with expand query

2015-04-29 Thread Joel Bernstein
as the expand.q, which typically produces smaller result sets. It was also designed mainly to be used with Collapse, where it's main job was to expand the collapsed result set, which would have been limited by the main query. Solr 5, will be much faster for the scenario that you are using. Joel Bernstein

Re: How to improve the performance of query with expand query

2015-04-28 Thread Joel Bernstein
Could you provide a few more details? 1) Version of Lucene/Solr 2) A sample slow query 3) Number of unique values in the collapse field 4) Number of search results before the collapse 5) Number of results fetched in the page 6) Performance numbers for the query Joel Bernstein http

Re: Simple search low speed

2015-04-24 Thread Joel Bernstein
Try breaking down the query to see which part of it is slow. If it turns out to be the range query you may want to look into using an frange postfilter. Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Apr 24, 2015 at 6:50 AM, Norgorn lsunnyd...@mail.ru wrote: Thanks for your reply. Yes

Re: rq breaks wildcard search?

2015-04-22 Thread Joel Bernstein
Just confirmed that wildcard queries work with Re-Ranking following SOLR-6323. Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Apr 22, 2015 at 7:26 PM, Joel Bernstein joels...@gmail.com wrote: This should be resolved in https://issues.apache.org/jira/browse/SOLR-6323 . Solr 4.10.3

Re: rq breaks wildcard search?

2015-04-22 Thread Joel Bernstein
This should be resolved in https://issues.apache.org/jira/browse/SOLR-6323. Solr 4.10.3 Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Apr 15, 2015 at 6:23 PM, Ryan Josal rjo...@gmail.com wrote: Using edismax, supplying a rq= param, like {!rerank ...} is causing

Re: rq breaks wildcard search?

2015-04-22 Thread Joel Bernstein
For your own implementation you'll need to implement the following methods: public Query rewrite(IndexReader reader) throws IOException public void extractTerms(SetTerm terms) You can review the 4.10.3 version of the ReRankQParserPlugin to see how it implements these methods. Joel Bernstein

Re: Collapse and Expand behaviour on result with 1 document.

2015-04-07 Thread Joel Bernstein
release. Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Apr 7, 2015 at 3:27 AM, Derek Poh d...@globalsources.com wrote: Hi Joel Is the number of documents info available when using collapse and expand parameters? I can't seem to find it in the return xml. I know the numFound

Re: Collapse and Expand behaviour on result with 1 document.

2015-04-01 Thread Joel Bernstein
Exactly correct. Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Apr 1, 2015 at 5:44 AM, Derek Poh d...@globalsources.com wrote: Hi Joel Correct me if my understanding is wrong. Using supplier id as the field to collapse on. - If thecollapse group heads inthe main result set has only

Re: Collapse and Expand behaviour on result with 1 document.

2015-03-31 Thread Joel Bernstein
You should be able to use collapse/expand with one result. Does the document in the main result set have group members that aren't being expanded? Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Mar 31, 2015 at 2:00 AM, Derek Poh d...@globalsources.com wrote: If I want to group

Re: Collapse and Expand behaviour on result with 1 document.

2015-03-31 Thread Joel Bernstein
if there is an expanded group. Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Mar 31, 2015 at 7:37 AM, Joel Bernstein joels...@gmail.com wrote: You should be able to use collapse/expand with one result. Does the document in the main result set have group members that aren't being expanded

Re: Collapse and Expand behaviour on result with 1 document.

2015-03-30 Thread Joel Bernstein
The expanded section will only include groups that have expanded documents. So, if the document that in the main result set has no documents to expand, then this is working as expected. Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Mar 30, 2015 at 8:43 PM, Derek Poh d

Re: ExpandComponent not expanding

2015-03-06 Thread Joel Bernstein
The expand component only displays the groups heads when it finds expanded documents in the group. And it only expands for the current page. Are you finding situations where there are group heads on the page, that have child documents that are not being expanded? Joel Bernstein Search Engineer

Re: rankquery usage bug?

2015-02-20 Thread Joel Bernstein
Ryan, This looks like a good jira ticket to me. Joel Bernstein Search Engineer at Heliosearch On Fri, Feb 20, 2015 at 6:40 PM, Ryan Josal rjo...@gmail.com wrote: Hey guys, I put a rq in defaults but I can't figure out how to override it with no rankquery. Looks like one option might

Re: Query Elevation Component only elevates one document in elevateIds list

2015-02-06 Thread Joel Bernstein
Try sending the elevateIds as a comma delimited list. Joel Bernstein Search Engineer at Heliosearch On Fri, Feb 6, 2015 at 2:17 PM, Charles Sanders csand...@redhat.com wrote: Using the Query Elevation Component in solr 4.8 and the elevateIds parameter to force 3 documents to the top

Re: Does DocValues improve Grouping performance ?

2015-01-30 Thread Joel Bernstein
A few questions so we can better understand the scale of grouping you're trying to accomplish: How many distinct groups do you typically have in a search result? How many distinct groups are there in the field you are grouping on? How many results are you trying to group in a query? Joel

Re: How to inject custom response data after results have been sorted

2015-01-24 Thread Joel Bernstein
that because it re-issues the query (and filters) and then collects the expanded groups for the current page. You could also collect the stats for the current page. Joel Bernstein Search Engineer at Heliosearch On Fri, Jan 23, 2015 at 4:19 PM, tedsolr tsm...@sciquest.com wrote: Thank you so much

Re: How to return custom collector info

2015-01-19 Thread Joel Bernstein
You may want to take a look at the AnalyticsQuery: http://heliosearch.org/custom-analytics-engine/ This is an extension to the PostFIlter API that gives you direct access to the ResponseBuilder. Joel Bernstein Search Engineer at Heliosearch On Mon, Jan 19, 2015 at 4:28 PM, tedsolr tsm

Re: How to return custom collector info

2015-01-19 Thread Joel Bernstein
Here is actually the a more useful link for understanding how the AnalyticsQuery works: http://heliosearch.org/solrs-new-analyticsquery-api/ Joel Bernstein Search Engineer at Heliosearch On Mon, Jan 19, 2015 at 4:57 PM, Joel Bernstein joels...@gmail.com wrote: You may want to take a look

Re: Engage custom hit collector for special search processing

2015-01-13 Thread Joel Bernstein
You may also want to take a look at how AnalyticsQueries can be plugged in. This won't show you how to do the implementation but it will show you how you can plugin a custom collector. http://heliosearch.org/solrs-new-analyticsquery-api/ http://heliosearch.org/solrs-mergestrategy/ Joel Bernstein

Re: How to limit the number of result sets of the 'export' handler

2015-01-07 Thread Joel Bernstein
for you. You'll want to be using the 4.10.3 version of export which has an important bug fix in it. Joel Bernstein Search Engineer at Heliosearch On Wed, Jan 7, 2015 at 9:49 AM, Alexandre Rafalovitch arafa...@gmail.com wrote: I believe export is streaming and it avoids building various caches, so

Re: solr export get wrong results

2014-12-30 Thread Joel Bernstein
http://heliosearch.org/streaming-aggregation-for-solrcloud/ Joel Bernstein Search Engineer at Heliosearch On Mon, Dec 29, 2014 at 2:20 AM, Sandy Ding sandy.ding...@gmail.com wrote: Hi, Joel Thanks for your reply. It seems that the weird export results is because that I removed the str

Re: /export - Why need sort criteria (4.10.2)?

2014-12-27 Thread Joel Bernstein
/ You'll want to be using the 4.10.3 version which has a format change and bug fix. If you still run into the NPE, please share your findings. Joel Bernstein Search Engineer at Heliosearch On Wed, Dec 17, 2014 at 2:23 PM, Damien Dykman damien.dyk...@gmail.com wrote: The /export request handler

Re: solr export get wrong results

2014-12-26 Thread Joel Bernstein
Hi Sandy, The export handler should only return documents in JSON format. The results in your second example are in XML for format so something looks to be wrong in the configuration. Can you post what your solrconfig looks like? Joel Joel Bernstein Search Engineer at Heliosearch On Fri, Dec

Re: solr export get wrong results

2014-12-26 Thread Joel Bernstein
the issue. Joel Bernstein Search Engineer at Heliosearch On Fri, Dec 26, 2014 at 1:50 PM, Joel Bernstein joels...@gmail.com wrote: Hi Sandy, The export handler should only return documents in JSON format. The results in your second example are in XML for format so something looks to be wrong

What is the new startup procedure following: SOLR-3619?

2014-12-01 Thread Joel Bernstein
Not having much luck starting Solr in trunk with the new startup scripts. I've scanned the README's but I'm not seeing anything definitive on the how startup works. Here is the steps I took: 1) Updated trunk 2) Ran: ant compile; ant dist; ant example; 3) Ran: bin/solr start 4) This error appears

Re: What is the new startup procedure following: SOLR-3619?

2014-12-01 Thread Joel Bernstein
Thanks Alex. Updating again did the trick. Joel Bernstein Search Engineer at Heliosearch On Mon, Dec 1, 2014 at 12:28 PM, Alexandre Rafalovitch arafa...@gmail.com wrote: On 1 December 2014 at 12:14, Joel Bernstein joels...@gmail.com wrote: SolrConfigRestApi Do it again: https

Re: Duplicate scoring situation in DelegatingCollector

2014-11-21 Thread Joel Bernstein
Sounds like you'll want use the ScoreCachingWrappingScorer. Your DelegatingCollector can wrap the ScoreCachingWrappingScorer around the scorer passed into the setScorer(Scorer) method and pass it to down the collector chain. Joel Bernstein Search Engineer at Heliosearch On Fri, Nov 14, 2014 at 3

Re: Does ReRankQuery support reranking the result of a FuzzyQuery?

2014-11-11 Thread Joel Bernstein
This issue should be resolved in https://issues.apache.org/jira/browse/SOLR-6323. This is committed in trunk, 5x, 4x, and 4_10, but this did not make it into 4.10.2. If you take the version in the 4_10 branch you should be good to go. If a version 4.10.3 is cut, this will be included. Joel

Re: Does ReRankQuery support reranking the result of a FuzzyQuery?

2014-11-11 Thread Joel Bernstein
]/float[@name='id'][.='2.0'], //result/doc[5]/float[@name='id'][.='1.0'] ); Joel Bernstein Search Engineer at Heliosearch On Tue, Nov 11, 2014 at 1:04 PM, Joel Bernstein joels...@gmail.com wrote: This issue should be resolved in https://issues.apache.org/jira/browse/SOLR-6323

Re: Exporting Error in 4.10.1

2014-10-30 Thread Joel Bernstein
sets. This feature includes an entirely new sorting and exporting engine so there are some bugs lurking. I'll be opening a ticket to resolve these bugs shortly. Joel Bernstein Search Engineer at Heliosearch On Thu, Oct 30, 2014 at 9:41 AM, Joseph Obernberger joseph.obernber...@gmail.com wrote

Re: Collapse and Expand Results in Solr 4.10 / Highlighting

2014-10-28 Thread Joel Bernstein
You are correct. Highlighting is working from the DocList, which only includes the collapsed set when using Collapse/Expand. Joel Bernstein Search Engineer at Heliosearch On Tue, Oct 28, 2014 at 9:46 AM, Michael Hagström mhagstr...@brox.de wrote: Hello! I'm testing the »Collapse

Re: Custom Solr Query Post Filter

2014-10-08 Thread Joel Bernstein
The results are being cached in the QueryResultCache most likely. You need to implement equals() and hashCode() on the query object, which is part of the cache key. In your case the creds param must be included in the hashCode and equals logic. Joel Bernstein Search Engineer at Heliosearch

Re: Custom Solr Query Post Filter

2014-10-08 Thread Joel Bernstein
or DocValues. Recent releases of DocValues will likely be the fastest docID-BytesRef lookup. Joel Bernstein Search Engineer at Heliosearch On Wed, Oct 8, 2014 at 2:20 PM, Christopher Gross cogr...@gmail.com wrote: That did the trick! Thanks Joel. -- Chris On Wed, Oct 8, 2014 at 2:05 PM

Re: Does CloudSolrServer hit zookeeper for every request?

2014-10-06 Thread Joel Bernstein
There is a shutdown() method on CloudSolrServer that releases the resources for CloudSolrServer. Joel Bernstein Search Engineer at Heliosearch On Mon, Oct 6, 2014 at 11:39 AM, Jonnakuti, Vijayalatha vijayalatha.jonnak...@disney.com wrote: We are using solr 4.8.0 From: Jonnakuti, Vijay

Re: Export feature issue in Solr 4.10

2014-10-02 Thread Joel Bernstein
Yep getting the same error. Investigating... Joel Bernstein Search Engineer at Heliosearch On Thu, Oct 2, 2014 at 12:59 PM, Ahmed Adel ahmed.a...@badrit.com wrote: Hi All, I'm trying to use Solr 4.10 export feature, but I'm getting an error. Maybe I missed something. Here's the scenario

Re: Export feature issue in Solr 4.10

2014-10-02 Thread Joel Bernstein
no problems running with lazy loading enabled. I'll create jira to fix this issue. Joel Bernstein Search Engineer at Heliosearch On Thu, Oct 2, 2014 at 4:10 PM, Joel Bernstein joels...@gmail.com wrote: Yep getting the same error. Investigating... Joel Bernstein Search Engineer at Heliosearch

Re: Solr and hadoop

2014-09-25 Thread Joel Bernstein
is underway to provide SolrCloud with native streaming aggregation capabilities. Joel Bernstein Search Engineer at Heliosearch On Thu, Sep 25, 2014 at 12:34 PM, Tom Chen tomchen1...@gmail.com wrote: I'm aware of the MapReduceIndexerTool (MRIT). That might be solving the indexing part

Re: Query ReRanking question

2014-09-07 Thread Joel Bernstein
Erick, you've taught me a lot over the past couple of years! Joel Bernstein Search Engineer at Heliosearch On Sun, Sep 7, 2014 at 11:37 AM, Erick Erickson erickerick...@gmail.com wrote: Joel: I find that whenever I say something totally wrong publicly, I remember the correction really

Re: Query ReRanking question

2014-09-07 Thread Joel Bernstein
Oops wrong usage pattern. It should be: 1) Main query is sorted by a field (scores tracked silently in the background). 2) Reranker is reRanking docs based on the score from the main query. Joel Bernstein Search Engineer at Heliosearch On Sun, Sep 7, 2014 at 4:43 PM, Joel Bernstein joels

Re: Query ReRanking question

2014-09-06 Thread Joel Bernstein
to return a float because the query reranker is expecting a score which is a float. So you'll have to devise function query logic that will transform your date to a float. Joel Bernstein Search Engineer at Heliosearch On Fri, Sep 5, 2014 at 7:06 PM, Ravi Solr ravis...@gmail.com wrote

Re: Query ReRanking question

2014-09-06 Thread Joel Bernstein
by *:*, which in theory would have no effect at all. The reRankQuery only uses the reRankQuery to re-rank the results. The sort param will always apply to the main query. Joel Bernstein Search Engineer at Heliosearch On Sat, Sep 6, 2014 at 2:33 PM, Ravi Solr ravis...@gmail.com wrote: Erick

Re: Query ReRanking question

2014-09-06 Thread Joel Bernstein
and see if this changes the result ordering. Joel Bernstein Search Engineer at Heliosearch On Sat, Sep 6, 2014 at 3:58 PM, Joel Bernstein joels...@gmail.com wrote: This folllowing query: http://localhost:8080/solr/select?q=malaysian airline crashrq={!rerank reRankQuery=$rqq reRankDocs=1000

Re: Query ReRanking question

2014-09-04 Thread Joel Bernstein
If you want the main query to be sorted by date then the top N docs reranked by a query, that should work. Try something like this: q=foosort=date+descrq={!rerank reRandDocs=1000 reRankQuery=$myquery}myquery=blah Joel Bernstein Search Engineer at Heliosearch On Thu, Sep 4, 2014 at 4:25 PM

Re: Paging bug in ReRankingQParserPlugin?

2014-08-05 Thread Joel Bernstein
figure out better behavior here that would be great. Joel Bernstein Search Engineer at Heliosearch On Mon, Aug 4, 2014 at 7:56 PM, Adair Kovac adairko...@gmail.com wrote: Looking at this line in the code: // This enusres that reRankDocs = docs needed to satisfy the result set

Re: Paging bug in ReRankingQParserPlugin?

2014-08-05 Thread Joel Bernstein
I updated the docs for now. But I agree this paging issue needs to be handled transparently. Feel free to create a jira issue for this or I can create one when I have time to start looking into it. Joel Bernstein Search Engineer at Heliosearch On Tue, Aug 5, 2014 at 12:04 PM, Adair Kovac

Re: Shuffle results a little

2014-07-25 Thread Joel Bernstein
Query ReRanking is built on the RankQuery API. With the RankQuery API you can build and plugin your own ranking algorithms. Here's a blog describing the RankQuery API: http://heliosearch.org/solrs-new-rankquery-feature/ Joel Bernstein Search Engineer at Heliosearch On Fri, Jul 25, 2014 at 4

Re: Shuffling results

2014-07-24 Thread Joel Bernstein
This is the kind of use case the RankQuery API was created for. It allows you to write your own Lucene ranking collector and plug it in. It's an expert level java API so you'll need to program in Java and understand a lot about how Lucene collectors work, but it's cool stuff to learn. Joel

Re: Shuffling results

2014-07-24 Thread Joel Bernstein
Here's blog describing the RankQuery API: http://heliosearch.org/solrs-new-rankquery-feature/ Joel Bernstein Search Engineer at Heliosearch On Thu, Jul 24, 2014 at 6:22 PM, Joel Bernstein joels...@gmail.com wrote: This is the kind of use case the RankQuery API was created for. It allows you

Re: Question about ReRankQuery

2014-07-23 Thread Joel Bernstein
implementation. Joel Bernstein Search Engineer at Heliosearch On Wed, Jul 23, 2014 at 11:37 AM, Peter Keegan peterlkee...@gmail.com wrote: See http://heliosearch.org/solrs-new-re-ranking-feature/ On Wed, Jul 23, 2014 at 11:27 AM, Erick Erickson erickerick...@gmail.com wrote: I'm having a little

Re: Question about ReRankQuery

2014-07-23 Thread Joel Bernstein
Blog on the RankQuery API http://heliosearch.org/solrs-new-rankquery-feature/ Joel Bernstein Search Engineer at Heliosearch On Wed, Jul 23, 2014 at 3:27 PM, Joel Bernstein joels...@gmail.com wrote: The ReRankingQParserPlugin uses the Lucene QueryRescorer, which only uses the score from

Re: Question about ReRankQuery

2014-07-23 Thread Joel Bernstein
I like the FieldSortedHitQueue idea. If you want to work up a patch for that, it would be great. Joel Bernstein Search Engineer at Heliosearch On Wed, Jul 23, 2014 at 5:17 PM, Peter Keegan peterlkee...@gmail.com wrote: The ReRankingQParserPlugin uses the Lucene QueryRescorer, which only

Re: Performance of indexing using Solr

2014-07-23 Thread Joel Bernstein
It looks you're committing too frequently. If you're explicitly committing from the application you may want to switch to using autoCommits. If you're not committing from the application your autocommit settings are probably too low. Joel Bernstein Search Engineer at Heliosearch On Wed, Jul 23

Re: Inconsistant result's on solr cloud 4.8

2014-07-17 Thread Joel Bernstein
Also, it would be a good idea to verify if the numDocs are in sync between leaders and replicas. You can view this from the admin screens. Joel Bernstein Search Engineer at Heliosearch On Thu, Jul 17, 2014 at 9:53 PM, Erick Erickson erickerick...@gmail.com wrote: Is there any chance

Re: SOLR-6143 Bad facet counts from CollapsingQParserPlugin

2014-07-15 Thread Joel Bernstein
They should be same as long as the same group heads are selected with both queries. The CollapsingQParserPugin simply collapses the result set and then forwards to lower collectors, so the DocSet created should always be for the collapsed set. Joel Bernstein Search Engineer at Heliosearch

Re: Group only top 50 results not All results.

2014-07-12 Thread Joel Bernstein
to the lower collectors. Grouping supports PostFilters so this should work with Grouping or you could use the CollapsingQParserPlugin. Joel Bernstein Search Engineer at Heliosearch On Sat, Jul 12, 2014 at 1:31 PM, Alexandre Rafalovitch arafa...@gmail.com wrote: I don't think either grouping

Re: SOLR-6143 Bad facet counts from CollapsingQParserPlugin

2014-07-11 Thread Joel Bernstein
The CollapsingQParserPlugin currently supports facet counts that match group.truncate. This works great for some use cases. There are use cases though where group.facets counts are preferred. No timetable yet on adding this feature for the CollapsingQParserPlugin. Joel Bernstein Search Engineer

Re: CollapsingQParserPlugin throws Exception when useFilterForSortedQuery=true

2014-06-30 Thread Joel Bernstein
Sure, go ahead create the ticket. I think there is more we can here as well. I suspect we can get the CollapsingQParserPlugin to work with useFilterForSortedQuery=true if scoring is not needed for the collapse. I'll take a closer look at this. Joel Bernstein Search Engineer at Heliosearch

Re: CollapsingQParserPlugin throws Exception when useFilterForSortedQuery=true

2014-06-28 Thread Joel Bernstein
with CollapsingQParsePlugin is probably not possible. So, a nice error message would be a good thing. Joel Bernstein Search Engineer at Heliosearch On Tue, Jun 24, 2014 at 3:31 AM, Umesh Prasad umesh.i...@gmail.com wrote: Hi , Found another bug with CollapsignQParserPlugin. Not a critical one. It throws

Re: Bug in Collapsing QParserPlugin : Sort by 3 or more fields is broken

2014-06-19 Thread Joel Bernstein
two sort critea: cscore(), which returns the score, would be the primary criteria. The recip of field x would be the secondary criteria. Joel Bernstein Search Engineer at Heliosearch On Thu, Jun 19, 2014 at 2:18 AM, Umesh Prasad umesh.i...@gmail.com wrote: Continuing the discussion

Re: Bug in Collapsing QParserPlugin : Sort by 3 or more fields is broken

2014-06-13 Thread Joel Bernstein
Let's open a new ticket. Joel Bernstein Search Engineer at Heliosearch On Fri, Jun 13, 2014 at 8:08 AM, Umesh Prasad umesh.i...@gmail.com wrote: The patch in SOLR-5408 fixes the issue with sorting only for two sort fields. Sorting still breaks when 3 or more sort fields are used. I have

Re: Implementing Hive query in Solr

2014-06-12 Thread Joel Bernstein
You may have to implement this yourself. In Solr 4.9 you'll be able to implement your own analytic functions in java and plug them in using the AnalyticsQuery API. This is a new Solr API for plugging in custom analytics. http://heliosearch.org/solrs-new-analyticsquery-api/ Joel Bernstein Search

Re: Implementing Hive query in Solr

2014-06-12 Thread Joel Bernstein
Yeah, solr-jdbc does look interesting. Has an Apache license as well. Joel Bernstein Search Engineer at Heliosearch On Thu, Jun 12, 2014 at 1:18 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: Hello, I've found https://github.com/kawasima/solr-jdbc recently. Haven't checked it so

Re: Performance/scaling with custom function queries

2014-06-11 Thread Joel Bernstein
In Solr 4.9 there is a feature called RankQueries, that allows you to plugin your own ranking collector. So, if you wanted to write a ranking/sorting collector that used a thread per segment, you could cleanly plug it in. Joel Bernstein Search Engineer at Heliosearch On Wed, Jun 11, 2014 at 9

Re: How Can I modify the DocList and DocSet in solr

2014-06-10 Thread Joel Bernstein
Not sure if this helps but it Solr 4.9, there is a new feature called RankQueries. You can read about it here: http://heliosearch.org/solrs-new-rankquery-feature/. Solr's new ReRankingQParserPlugin is built off of RankQueries. Joel Bernstein Search Engineer at Heliosearch On Tue, Jun 10, 2014

Re: Performance/scaling with custom function queries

2014-06-08 Thread Joel Bernstein
values for documents that match the query, so it won't be a full table scan unless all the docs match the query. Sounds like an interesting project. Please keep us posted. Joel Bernstein Search Engineer at Heliosearch On Sun, Jun 8, 2014 at 6:17 AM, Robert Krüger krue...@lesspain.de wrote: Hi

Re: Code that handles merging results from a distributed query

2014-06-07 Thread Joel Bernstein
The mergeIds method of the QueryComponent does the actual merging of the docs from the shards. Joel Bernstein Search Engineer at Heliosearch On Sun, Jun 8, 2014 at 1:31 AM, Phanindra R phani...@gmail.com wrote: Hi, In Solr in Action book, I read how the distributed queries work. Looks

RE: SOLR-6143 Bad facet counts from CollapsingQParserPlugin

2014-06-06 Thread Joel Bernstein
Reposting this from jira ticket to users list: I'm noticing a very weird bug using the CollapsingQParserPlugin. We tried to use this plugin when we realized that faceting on the groups would take a ridiculous amount of time. To its credit, it works very quickly, however the facet counts that it

Re: SOLR-6143 Bad facet counts from CollapsingQParserPlugin

2014-06-06 Thread Joel Bernstein
The CollapsingQParserPlugin should give you the same facet counts as group.truncate. You're using group.facets, which the CollapsingQParserplugin doesn't yet support. I think this would be an excellent feature, so we could make a jira ticket to add this feature. Joel Bernstein Search Engineer

Re: SOLR-6143 Bad facet counts from CollapsingQParserPlugin

2014-06-06 Thread Joel Bernstein
With tag and exclude you can get the facets counts for the collapsed set and expanded set in the same query. But group.facets is a different count then either of those. group.facets counts each facet bucket once per group. Joel Bernstein Search Engineer at Heliosearch On Fri, Jun 6, 2014 at 1

Re: Strange behaviour when tuning the caches

2014-06-04 Thread Joel Bernstein
the collapsingqparserplugin. Joel Bernstein Search Engineer at Heliosearch On Tue, Jun 3, 2014 at 12:38 PM, Jean-Sebastien Vachon jean-sebastien.vac...@wantedanalytics.com wrote: Yes we are already using it. -Original Message- From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] Sent

Re: (Issue) How improve solr group performance

2014-05-28 Thread Joel Bernstein
that don't have a group you can place a null in the group field and use the expand nullPolicy, which will place each null group record in it's own group. Using the nullPolicy like this will be much more memory efficient then placing a fake group id in the grouping field. Joel Bernstein Search

Re: grouping of multivalued fields

2014-05-21 Thread Joel Bernstein
You may want to investigate the group.func option. This would allow you to plug in your own logic to return the group by key. I don't think there is an existing function that does exactly what you need so you may have to write a custom function. Joel Bernstein Search Engineer at Heliosearch

Re: SOLR 4.8 Collections API Create Collection Fails on Tomcat

2014-05-21 Thread Joel Bernstein
) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64) It looks like the new collection is using the same data dir as an existing collection. This startup param might be the issue: export JAVA_OPTS=$JAVA_OPTS -Dsolr.data.dir=/ebs-data/solr/data at Joel Bernstein Search Engineer at Heliosearch On Wed

Re: Extensibility and code reuse: SOLR vs Lucene

2014-05-20 Thread Joel Bernstein
you'll be able to implement your own MergeStrategy, to merge the results generated by DelegatingCollectors on the shards (SOLR-5973). The pluggable collectors in that ticket are for ranking. The PostFilter delegating collectors are a better place for doing custom analytics. Joel Bernstein

Re: autowarming queries

2014-05-16 Thread Joel Bernstein
Are you talking about static warming queries, which you define as newSearcher and firstSearcher events? If so, you should see all three queries in the log. If you're still having the issue, can you post your warming query configuration? Joel Bernstein Search Engineer at Heliosearch On Wed, May

Re: Sorting problem in Solr due to Lucene Field Cache

2014-05-16 Thread Joel Bernstein
Take a look at Solr's use of DocValues: https://cwiki.apache.org/confluence/display/solr/DocValues. There are docValues options that use less memory then the FieldCache. Joel Bernstein Search Engineer at Heliosearch On Thu, May 15, 2014 at 6:39 AM, Jeongseok Son invictu...@gmail.com wrote

Re: Easises way to insatll solr cloud with tomcat

2014-05-14 Thread Joel Bernstein
Aman, You may want to try the Heliosearch Distribution For Solr. This is a free, pre-built Tomcat/Solr distro. http://heliosearch.com/download.html When you open up the zip file there is an examples.txt file in server/bin that has a couple SolrCloud examples. Joel Joel Bernstein Search

Re: filter capabilities are limited?

2014-04-10 Thread Joel Bernstein
It sounds like you can make it work with the frange qparser plugin: fq={!frange l=0 u=0}sub(field(a),field(b)) Joel Bernstein Search Engineer at Heliosearch On Thu, Apr 10, 2014 at 3:36 PM, Erick Erickson erickerick...@gmail.comwrote: Uhhhm, did you look at function queries at all

Re: CollapsingQParserPlugin returning different result set

2014-03-18 Thread Joel Bernstein
Hi Shamik, I see that you are using distributed search. With the CollapsingQParserPlugin you need to have all the documents that are in the same group on the same shard. Is that the way you have the documents indexed? Joel Joel Bernstein Search Engineer at Heliosearch On Mon, Mar 17, 2014

Re: CollapsingQParserPlugin facet results: fq={!collapse field=fld} vs. group=truegroup.field=fld

2014-03-16 Thread Joel Bernstein
query for specific facets if you want to see facet counts for the un-collapsed result set. Joel Joel Bernstein Search Engineer at Heliosearch On Tue, Mar 11, 2014 at 3:05 PM, tchaffee tchaf...@livingnaturally.comwrote: Should the same exact query using fq={!collapse field=fld} return the same

Re: CollapsingQParserPlugin returning different result set

2014-03-15 Thread Joel Bernstein
post the full debug output for both queries? Joel Joel Bernstein Search Engineer at Heliosearch On Fri, Mar 14, 2014 at 6:22 PM, shamik sham...@gmail.com wrote: Hi, I recently upgraded to 4.7, with the aim of replacing group queries with CollapsingQParserPlugin. As I'm comparing results

Re: CollapsingQParserPlugin is slower than standard Solr field grouping in Solr 4.6.1

2014-02-27 Thread Joel Bernstein
fine using standard grouping. Joel Bernstein Search Engineer at Heliosearch On Wed, Feb 26, 2014 at 2:55 PM, Joe Ho j...@basistech.com wrote: I notice that in Solr 4.6.1 CollapsingQParserPlugin is slower than standard Solr field grouping. I have a Solr index of 1 docs, with a signature

Re: CollapseQParserPlugin problem with ElevateComponent

2014-02-25 Thread Joel Bernstein
is that the elevated document becomes the group head. We can discuss the best way to handle this on the new ticket. Joel Joel Bernstein Search Engineer at Heliosearch On Tue, Feb 25, 2014 at 1:29 PM, dboychuck dboych...@build.com wrote: https://issues.apache.org/jira/browse/SOLR-5773 I am having trouble

Re: Unloading a SolrCloud core in 4.6.0

2014-02-13 Thread Joel Bernstein
are protecting against an NPE. In 4.6 and 4.6.1 the NPE check is not present. So the issue may be resolved in trunk and 4x. Joel Bernstein Search Engineer at Heliosearch On Thu, Feb 13, 2014 at 7:06 PM, Lajos la...@protulae.com wrote: Hi all, I just want to verify that it is no longer possible

Re: Solr perfromance with commitWithin seesm too good to be true. I am afraid I am missing something

2014-02-12 Thread Joel Bernstein
explicitly committing after each document, your commits will happen too frequently, overworking the indexing process. Joel Bernstein Search Engineer at Heliosearch On Wed, Feb 12, 2014 at 9:52 AM, Pisarev, Vitaliy vitaliy.pisa...@hp.comwrote: I am running a very simple performance experiment where I

Re: filtering/faceting by a big list IDs

2014-02-12 Thread Joel Bernstein
cause this can be faster, especially when result sets are relatively small but the index is large. Joel Bernstein Search Engineer at Heliosearch On Wed, Feb 12, 2014 at 2:12 PM, Tri Cao tm...@me.com wrote: Hi all, I am running a Solr application and I would need to implement a feature

Re: handleSelect=true with SolrCloud

2014-02-11 Thread Joel Bernstein
Joel Bernstein Search Engineer at Heliosearch On Tue, Feb 11, 2014 at 1:01 PM, Jeff Wartes jwar...@whitepages.com wrote: Got it in one. Thanks! On 2/11/14, 9:50 AM, Shawn Heisey s...@elyograg.org wrote: On 2/11/2014 10:21 AM, Jeff Wartes wrote: I¹m working on a port of a Solr

Re: Replica not consistent after update request?

2014-01-24 Thread Joel Bernstein
If you're on Solr 4.6 then this is likely the issue: https://issues.apache.org/jira/browse/SOLR-4260. The issue is resolved for Solr 4.6.1 which should be out next week. Joel Bernstein Search Engineer at Heliosearch On Fri, Jan 24, 2014 at 9:52 PM, Nathan Neulinger nn...@neulinger.orgwrote

Re: SolrCloud Result Grouping vs CollapsingQParserPlugin

2014-01-15 Thread Joel Bernstein
, when are you planning to release 4.6.1 ? There is a thread going on the dev list about the 4.6.1 release. You can follow progress at: http://markmail.org/search/?q=%22Lucene+%2F+Solr+4.6.1%22 Joel Bernstein Search Engineer at Heliosearch On Wed, Jan 15, 2014 at 2:35 AM, shamik sham...@gmail.com

Re: SolrCloud Result Grouping vs CollapsingQParserPlugin

2014-01-14 Thread Joel Bernstein
on the same shard. Joel Bernstein Search Engineer at Heliosearch On Tue, Jan 14, 2014 at 6:17 PM, Shamik Bandopadhyay sham...@gmail.comwrote: Hi, I'm planning to upgrade to Solr 4.6 to move from using Result Grouping to CollapsingQParserPlugin. I'm currently using SolrCloud, couple

Re: SolrCloud Result Grouping vs CollapsingQParserPlugin

2014-01-14 Thread Joel Bernstein
Also, there are a number of bugs in the CollapsingQParserPlugin in Solr 4.6 that are resolved in Solr 4.6.1 which should be out soon. Joel Bernstein Search Engineer at Heliosearch On Tue, Jan 14, 2014 at 10:00 PM, Joel Bernstein joels...@gmail.com wrote: Shamik, You still need to keep docs

Re: Questionon CollapsingQParserPlugin

2014-01-14 Thread Joel Bernstein
Something is off but I'm not sure what. A couple of questions. 1) You mention updating the solr.xml. Did you the schema.xml? 2) Did you load only those 4 docs? Joel Joel Bernstein Search Engineer at Heliosearch On Mon, Jan 13, 2014 at 4:21 PM, Shamik Bandopadhyay sham...@gmail.comwrote: Hi

Re: Questionon CollapsingQParserPlugin

2014-01-14 Thread Joel Bernstein
, incubationdate_dt:2006-01-17T00:00:00Z, adskdedup_s:MNOP-QRS, _version_:1457264913752784896}] }} Joel Bernstein Search Engineer at Heliosearch On Tue, Jan 14, 2014 at 10:10 PM, Joel Bernstein joels...@gmail.com wrote: Something is off but I'm not sure what. A couple

Re: Questionon CollapsingQParserPlugin

2014-01-14 Thread Joel Bernstein
, price:0.0, price_c:0,USD, popularity:10, inStock:true, incubationdate_dt:2006-01-17T00:00:00Z, adskdedup_s:MNOP-QRS, _version_:1457264913752784896}] }} Joel Bernstein Search Engineer at Heliosearch On Tue, Jan 14, 2014 at 10:25 PM, Joel Bernstein

Re: Solr Cloud Query Scaling

2014-01-09 Thread Joel Bernstein
You do need to load balance the initial query request across the SolrCloud nodes. Solj's CloudSolrServer and LBHttpSolrServer can perform the load balancing for you in the client. Or you can use a hardware load balancer. Joel Bernstein Search Engineer at Heliosearch On Thu, Jan 9, 2014 at 5:58

Re: Range queries with Grouping is slow?

2014-01-08 Thread Joel Bernstein
Kranti, The range query also looks like a good candidate to be moved to a filter query so it can be cached. Joel Bernstein Search Engineer at Heliosearch On Tue, Jan 7, 2014 at 11:34 PM, Smiley, David W. dsmi...@mitre.org wrote: Kranti, I can't speak to the specific slow-down while

Re: Solr not creating core.properties file

2014-01-08 Thread Joel Bernstein
Craig, Can you describe the steps you are using to create the core? Joel Joel Bernstein Search Engineer at Heliosearch On Wed, Jan 8, 2014 at 2:27 PM, Craig Longman clong...@iconect.com wrote: We're exploring using Solr as a search/faceting engine, and things went quite well during

Re: Solr not creating core.properties file

2014-01-08 Thread Joel Bernstein
having QA take a crack at it that it's not working. All three have the same config, checked pretty thoroughly, although our next step is for me to setup a fourth machine personally and see what happens. -Original Message- From: Joel Bernstein [mailto:joels...@gmail.com] Sent: January 8

Re: Solr not creating core.properties file

2014-01-08 Thread Joel Bernstein
suspect the solr.xml is driving this, so take a look and see how they are different between the server that is working and not working. Joel Joel Bernstein Search Engineer at Heliosearch On Wed, Jan 8, 2014 at 3:13 PM, Craig Longman clong...@iconect.com wrote: On the dev machine, it works

Re: Solr not creating core.properties file

2014-01-08 Thread Joel Bernstein
/4_6_0/solr-core/org/apache/solr/core/CorePropertiesLocator.html . So, I'm betting the machines not writing the core properties file are using the old style solr.xml. Joel Bernstein Search Engineer at Heliosearch On Wed, Jan 8, 2014 at 3:35 PM, Joel Bernstein joels...@gmail.com wrote: Craig

<    2   3   4   5   6   7   8   >