Re: Slow indexing with data import handler
On Sun, Apr 19, 2009 at 7:30 AM, Gustavo A. Lopes galo...@mediacapital.ptwrote: I'm indexing around 1 million documents of one type that requires 4 additional queries for each document + 0,5 M documents that only require 1 query for all. I'm using the data import handler from contrib with SolrWriter modified with allowDups = true (doesn't seem to have made any difference). This doesn't seem to be a that many documents, however, after 21 hours, I have only ~700 k documents of the first type indexed. The size of index is currently 2.1 GB I'm noticing the initial import rate is relatively high, such as all the documents of first type would be indexed in less than 6 hours if it were maintained. When the number of documents already imported rises, the import rate falls significatively. Does anyone have any suggestions on how to speed up full imports? What is the bottleneck? I will probably have to make some changes to schema over the next days that will require new imports. Further to Otis's suggestions -- Do you have autoCommit+autowarming turned on? Maybe that is the cause of the slowdown as the import progresses? -- Regards, Shalin Shekhar Mangar.
Multiple Solr-instance share same solr.home
Hi, Is it possible to have two solr instances share the same solr.home? I've two Solr instances running on the same box and I was wondering if I can configure them to have the same solr.home. I tried it, but looks like the second instance overwrites the first one's value in the solr.xml (I'm using multicore for both instances). This is just for convenience so I don't have to manage multiple solr index directory locations - I can have all the indexes written into the same location and do the clean up from one place itself. If this is not supported then it's not a big deal. Thanks, -vivek
Re: CollapseFilter with the latest Solr in trunk
Ok, here is how I fixed this problem:
public DocListAndSet getDocListAndSet(Query query, ListQuery filterList,
DocSet docSet, Sort lsort, int offset, int len, int flags) throwsIOException {
//DocListAndSet ret = new DocListAndSet();
//getDocListC(ret,query,filterList,docSet,lsort,offset,len, flags |=
GET_DOCSET);
DocSet theFilt = getDocSet(filterList);
if (docSet != null) theFilt = (theFilt != null) ?
theFilt.intersection(docSet) : docSet;
QueryCommand qc = new QueryCommand();
qc.setQuery(query).setFilter(theFilt);
qc.setSort(lsort).setOffset(offset).setLen(len).setFlags(flags |=
GET_DOCSET);
QueryResult result = new QueryResult();
getDocListC(result,qc);
return result.getDocListAndSet();
}
There is also one-off error in CollapseFilter which you can find solution on
Jira.
Cheers,
Cuong
On Sat, Apr 18, 2009 at 4:41 AM, Jeff Newburn jnewb...@zappos.com wrote:
We are currently trying to do the same thing. With the patch unaltered we
can use fq as long as collapsing is turned on. If we just send a normal
document level query with an fq parameter it blows up.
Additionally, it does not appear that the collapse.facet option works at
all.
--
Jeff Newburn
Software Engineer, Zappos.com
jnewb...@zappos.com - 702-943-7562
From: climbingrose climbingr...@gmail.com
Reply-To: solr-user@lucene.apache.org
Date: Fri, 17 Apr 2009 16:53:00 +1000
To: solr-user solr-user@lucene.apache.org
Subject: CollapseFilter with the latest Solr in trunk
Hi all,
Have any one try to use CollapseFilter with the latest version of Solr in
trunk? However, it looks like Solr 1.4 doesn't allow calling
setFilterList()
and setFilter() on one instance of the QueryCommand. I modified the code
in
QueryCommand to allow this:
public QueryCommand setFilterList(Query f) {
// if( filter != null ) {
//throw new IllegalArgumentException( Either filter or
filterList
may be set in the QueryCommand, but not both. );
// }
filterList = null;
if (f != null) {
filterList = new ArrayListQuery(2);
filterList.add(f);
}
return this;
}
However, I still have a problem which prevent query filters from working
when used in conjunction with CollapseFilter. In other words, query
filters
doesn't seem to have any effects on the result set when CollapseFilter is
used.
The other problem is related to OpenBitSet:
java.lang.ArrayIndexOutOfBoundsException: 2183
at org.apache.lucene.util.OpenBitSet.fastSet(OpenBitSet.java:242)
at org.apache.solr.search.CollapseFilter.addDoc(CollapseFilter.java:202)
at
org.apache.solr.search.CollapseFilter.adjacentCollapse(CollapseFilter.java:161
)
at
org.apache.solr.search.CollapseFilter.lt;initgt;(CollapseFilter.java:141)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:2
17)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandle
r.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
va:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303
)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:23
2)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
lterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
n.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:1
07)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processCon
nection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWork
erThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:
684)
at java.lang.Thread.run(Thread.java:619)
I think CollapseFilter is rather an important function in Solr that gets
used quite frequently. Does anyone have a solution for this?
--
Regards,
Cuong Hoang
--
Regards,
Cuong Hoang
Re: Multiple Solr-instance share same solr.home
Vivek - no, unless you want trouble - only 1 writer can write to a specific index at a time. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: vivek sar vivex...@gmail.com To: solr-user@lucene.apache.org Sent: Sunday, April 19, 2009 4:33:00 AM Subject: Multiple Solr-instance share same solr.home Hi, Is it possible to have two solr instances share the same solr.home? I've two Solr instances running on the same box and I was wondering if I can configure them to have the same solr.home. I tried it, but looks like the second instance overwrites the first one's value in the solr.xml (I'm using multicore for both instances). This is just for convenience so I don't have to manage multiple solr index directory locations - I can have all the indexes written into the same location and do the clean up from one place itself. If this is not supported then it's not a big deal. Thanks, -vivek
Re: CollapseFilter with the latest Solr in trunk
Thanks for sharing!
It would be good if you (of Jeff from Zappos or anyone making changes to this)
could put up a new patch for this most-voted-JIRA-issue.
Thanks,
Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
From: climbingrose climbingr...@gmail.com
To: solr-user@lucene.apache.org
Sent: Sunday, April 19, 2009 8:12:11 AM
Subject: Re: CollapseFilter with the latest Solr in trunk
Ok, here is how I fixed this problem:
public DocListAndSet getDocListAndSet(Query query, ListfilterList,
DocSet docSet, Sort lsort, int offset, int len, int flags) throwsIOException {
//DocListAndSet ret = new DocListAndSet();
//getDocListC(ret,query,filterList,docSet,lsort,offset,len, flags |=
GET_DOCSET);
DocSet theFilt = getDocSet(filterList);
if (docSet != null) theFilt = (theFilt != null) ?
theFilt.intersection(docSet) : docSet;
QueryCommand qc = new QueryCommand();
qc.setQuery(query).setFilter(theFilt);
qc.setSort(lsort).setOffset(offset).setLen(len).setFlags(flags |=
GET_DOCSET);
QueryResult result = new QueryResult();
getDocListC(result,qc);
return result.getDocListAndSet();
}
There is also one-off error in CollapseFilter which you can find solution on
Jira.
Cheers,
Cuong
On Sat, Apr 18, 2009 at 4:41 AM, Jeff Newburn wrote:
We are currently trying to do the same thing. With the patch unaltered we
can use fq as long as collapsing is turned on. If we just send a normal
document level query with an fq parameter it blows up.
Additionally, it does not appear that the collapse.facet option works at
all.
--
Jeff Newburn
Software Engineer, Zappos.com
jnewb...@zappos.com - 702-943-7562
From: climbingrose
Reply-To:
Date: Fri, 17 Apr 2009 16:53:00 +1000
To: solr-user
Subject: CollapseFilter with the latest Solr in trunk
Hi all,
Have any one try to use CollapseFilter with the latest version of Solr in
trunk? However, it looks like Solr 1.4 doesn't allow calling
setFilterList()
and setFilter() on one instance of the QueryCommand. I modified the code
in
QueryCommand to allow this:
public QueryCommand setFilterList(Query f) {
// if( filter != null ) {
//throw new IllegalArgumentException( Either filter or
filterList
may be set in the QueryCommand, but not both. );
// }
filterList = null;
if (f != null) {
filterList = new ArrayList(2);
filterList.add(f);
}
return this;
}
However, I still have a problem which prevent query filters from working
when used in conjunction with CollapseFilter. In other words, query
filters
doesn't seem to have any effects on the result set when CollapseFilter is
used.
The other problem is related to OpenBitSet:
java.lang.ArrayIndexOutOfBoundsException: 2183
at org.apache.lucene.util.OpenBitSet.fastSet(OpenBitSet.java:242)
at org.apache.solr.search.CollapseFilter.addDoc(CollapseFilter.java:202)
at
org.apache.solr.search.CollapseFilter.adjacentCollapse(CollapseFilter.java:161
)
at
org.apache.solr.search.CollapseFilter.init(CollapseFilter.java:141)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:2
17)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandle
r.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
va:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303
)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:23
2)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
lterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
n.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:1
07)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processCon
nection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
527)
at
Re: Boosting by facets with standard query
Thanks for the tip. Looks like a neat idea. I have never used the sort feature, so I have to create a new numeric key with values 1 or 2 - value 1 for white_papers/pdfs 2 for others? The problem also is that the facets I need to boost can vary by query. That is, if the query term 'a', boost the facets 'facet1 facet2'. If the query term is 'b', then boost the facets 'facet4 facet5'. Perhaps I can identify the most freqently used boost order, and create as many fields as there are orders. That would be the way right? - ashok Shalin Shekhar Mangar wrote: On Fri, Apr 17, 2009 at 11:32 AM, ashokc ash...@qualcomm.com wrote: What we need is for the white_papers pdfs to be boosted, but if and only if such doucments are valid results to the search term in question. How would I write my above 'q' to accomplish that? Thanks for explaining in detail. Basically, all you want to do is sort the results in the following order: 1. White papers 2. PDFs 3. Others or maybe #1 and #2 are equivalent and can be intermingled. Easiest way to do this is to index a new field whose values (when sorted) give you the desired order. Then you can simply sort on that field and score. -- Regards, Shalin Shekhar Mangar. -- View this message in context: http://www.nabble.com/Boosting-by-facets-with-standard-query-tp23084860p23123288.html Sent from the Solr - User mailing list archive at Nabble.com.
randomizing search results with even display chance
Hi, I'm currently new at Lucene and Solr, but I've managed to get some results so far, and like the project very much. For my current project, I need to randomize the search results, but still keep the sorting by relevance. For example, If 3 results have the same score, they can be ordered 1-2-3, 2-1-3-, 3-2-1, 2-3-1, etc with every new search. And...another thing I'm interested in is how to make sure the different results have equal display times if the scoring is the same. If for example 20 results have the same score, and I only show 5 in my app, I still want result nr 19 to be shown as many times as result 1, because the score is the same. I know a search engine is meant to rank result by score, but I want every result (with the same score) to have the same face time by randomizing the order. Can anyone tell me if these things would be possible, and/or where to start looking? Thanks in advance
Re: Boosting by facets with standard query
On Fri, Apr 17, 2009 at 2:02 AM, ashokc ash...@qualcomm.com wrote: q=(+(content:umts)+OR+(title:umts)^2+OR+(urltext:umts)^2)+AND+(doctype:white_papers)^2+AND+(filetype:pdf)^2 [...] What we need is for the white_papers pdfs to be boosted, but if and only if such doucments are valid results to the search term in question. Make the entire query part mandatory and the doctype boost optional. Assuming that the default query operator is OR... this would look like: +(content:umts title:umts^2 urltext:umts^2) doctype:white_papers^2 filetype:pdf^2 You may also want to look at the dismax query parser, which can make constructing queries like this a lot easier. defType=dismax q=umts qf=content title^2 urltext^2 bq=doctype:white_papers^2 filetype:pdf^2 -Yonik http://www.lucidimagination.com
Re: Multiple Solr-instance share same solr.home
Both Solr instances will be writing to separate indexes, but can they share the same solr.home? So, here is what I want, 1) solr.home = solr/multicore 2) There is a single solr.xml under multicore directory 3) Each instance would use the same solr.xml, which will have entries for multiple cores 4) Each instance will write to different core at a time - so one index will be written by only one writer at a time. not sure if this is a supported configuration. Thanks. -vivek On Sun, Apr 19, 2009 at 5:55 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Vivek - no, unless you want trouble - only 1 writer can write to a specific index at a time. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: vivek sar vivex...@gmail.com To: solr-user@lucene.apache.org Sent: Sunday, April 19, 2009 4:33:00 AM Subject: Multiple Solr-instance share same solr.home Hi, Is it possible to have two solr instances share the same solr.home? I've two Solr instances running on the same box and I was wondering if I can configure them to have the same solr.home. I tried it, but looks like the second instance overwrites the first one's value in the solr.xml (I'm using multicore for both instances). This is just for convenience so I don't have to manage multiple solr index directory locations - I can have all the indexes written into the same location and do the clean up from one place itself. If this is not supported then it's not a big deal. Thanks, -vivek
Re: Multiple Solr-instance share same solr.home
as long as you make sure there are never two applications writing to the same index, you *should* be ok. But tread carefully... On Apr 19, 2009, at 3:28 PM, vivek sar wrote: Both Solr instances will be writing to separate indexes, but can they share the same solr.home? So, here is what I want, 1) solr.home = solr/multicore 2) There is a single solr.xml under multicore directory 3) Each instance would use the same solr.xml, which will have entries for multiple cores 4) Each instance will write to different core at a time - so one index will be written by only one writer at a time. not sure if this is a supported configuration. Thanks. -vivek On Sun, Apr 19, 2009 at 5:55 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Vivek - no, unless you want trouble - only 1 writer can write to a specific index at a time. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: vivek sar vivex...@gmail.com To: solr-user@lucene.apache.org Sent: Sunday, April 19, 2009 4:33:00 AM Subject: Multiple Solr-instance share same solr.home Hi, Is it possible to have two solr instances share the same solr.home? I've two Solr instances running on the same box and I was wondering if I can configure them to have the same solr.home. I tried it, but looks like the second instance overwrites the first one's value in the solr.xml (I'm using multicore for both instances). This is just for convenience so I don't have to manage multiple solr index directory locations - I can have all the indexes written into the same location and do the clean up from one place itself. If this is not supported then it's not a big deal. Thanks, -vivek
Re: Advice on moving from 1.3 to 1.4-dev or trunk?
When you say Test ... Are you suggesting there is a test suite I should run, or do just do my own testing? your own testing... If you use a 'nightly' the unit tests all pass. BUT if you are not running from a standard release, there is may be things that are not totally flushed out, or configurations that have not been tried yet. For a release build lots of effort is made to make sure all lose ends are tied up. ryan
Can we provide context dependent faceted navigation from SOLR search results
Assuming a solr search returns 10 listing items as below 1) 4 digital cameras 2) 4 LCD televisions 3) 2 clothing items If we navigate to /electronics we want solr to show us facets specific to 8 electronics items (e.g brand, price). If we navigate to /electronics/cameras we want solr to show us facets specific to 4 camera items (e.g mega-pixels, screens-size, brand, price). If we navigate to /electronics/televisions we want to see different facets and their counts specific to TV items. If we navigate to /clothing we want to obtain totally different facets and their counts. I am not sure if we can think of this as Hierarchical Facet Navigation system or not. From the UI perspective , we can think of /electronics/cameras as Hierarchical classification. But how about electronics/cameras/canon vs electronics/canon/camera. In this case both navigation should show the same result set no matter which facet is selected first. My question is with the current solr implementation can we provide context dependent faceted navigation from SOLR search results? Thank you. Thanh Doan
Re: CollapseFilter with the latest Solr in trunk
I wish it would be planned for 1.4 :))
--- Den sön 2009-04-19 skrev Otis Gospodnetic otis_gospodne...@yahoo.com:
Från: Otis Gospodnetic otis_gospodne...@yahoo.com
Ämne: Re: CollapseFilter with the latest Solr in trunk
Till: solr-user@lucene.apache.org
Datum: söndag 19 april 2009 15.06
Thanks for sharing!
It would be good if you (of Jeff from Zappos or anyone
making changes to this) could put up a new patch for this
most-voted-JIRA-issue.
Thanks,
Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
From: climbingrose climbingr...@gmail.com
To: solr-user@lucene.apache.org
Sent: Sunday, April 19, 2009 8:12:11 AM
Subject: Re: CollapseFilter with the latest Solr in
trunk
Ok, here is how I fixed this problem:
public DocListAndSet
getDocListAndSet(Query query, ListfilterList,
DocSet docSet, Sort lsort, int offset, int len, int
flags) throwsIOException {
//DocListAndSet ret = new
DocListAndSet();
//getDocListC(ret,query,filterList,docSet,lsort,offset,len,
flags |=
GET_DOCSET);
DocSet theFilt = getDocSet(filterList);
if (docSet != null) theFilt = (theFilt != null) ?
theFilt.intersection(docSet) : docSet;
QueryCommand qc = new
QueryCommand();
qc.setQuery(query).setFilter(theFilt);
qc.setSort(lsort).setOffset(offset).setLen(len).setFlags(flags
|=
GET_DOCSET);
QueryResult result = new
QueryResult();
getDocListC(result,qc);
return
result.getDocListAndSet();
}
There is also one-off error in CollapseFilter which
you can find solution on
Jira.
Cheers,
Cuong
On Sat, Apr 18, 2009 at 4:41 AM, Jeff Newburn wrote:
We are currently trying to do the same
thing. With the patch unaltered we
can use fq as long as collapsing is turned
on. If we just send a normal
document level query with an fq parameter it
blows up.
Additionally, it does not appear that the
collapse.facet option works at
all.
--
Jeff Newburn
Software Engineer, Zappos.com
jnewb...@zappos.com
- 702-943-7562
From: climbingrose
Reply-To:
Date: Fri, 17 Apr 2009 16:53:00 +1000
To: solr-user
Subject: CollapseFilter with the latest Solr
in trunk
Hi all,
Have any one try to use CollapseFilter with
the latest version of Solr in
trunk? However, it looks like Solr 1.4
doesn't allow calling
setFilterList()
and setFilter() on one instance of the
QueryCommand. I modified the code
in
QueryCommand to allow this:
public QueryCommand
setFilterList(Query f) {
// if( filter != null )
{
// throw new
IllegalArgumentException( Either filter or
filterList
may be set in the QueryCommand, but not
both. );
// }
filterList =
null;
if (f !=
null) {
filterList = new ArrayList(2);
filterList.add(f);
}
return this;
}
However, I still have a problem which
prevent query filters from working
when used in conjunction with
CollapseFilter. In other words, query
filters
doesn't seem to have any effects on the
result set when CollapseFilter is
used.
The other problem is related to OpenBitSet:
java.lang.ArrayIndexOutOfBoundsException:
2183
at
org.apache.lucene.util.OpenBitSet.fastSet(OpenBitSet.java:242)
at
org.apache.solr.search.CollapseFilter.addDoc(CollapseFilter.java:202)
at
org.apache.solr.search.CollapseFilter.adjacentCollapse(CollapseFilter.java:161
)
at
org.apache.solr.search.CollapseFilter.init(CollapseFilter.java:141)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:2
17)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandle
r.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
va:131)
at
org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303
)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:23
2)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
lterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
n.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
Re: CollapseFilter with the latest Solr in trunk
Once somebody really makes it work, I'm sure it will be released!
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
From: Antonio Eggberg antonio_eggb...@yahoo.se
To: solr-user@lucene.apache.org
Sent: Sunday, April 19, 2009 9:21:20 PM
Subject: Re: CollapseFilter with the latest Solr in trunk
I wish it would be planned for 1.4 :))
--- Den sön 2009-04-19 skrev Otis Gospodnetic :
Från: Otis Gospodnetic
Ämne: Re: CollapseFilter with the latest Solr in trunk
Till: solr-user@lucene.apache.org
Datum: söndag 19 april 2009 15.06
Thanks for sharing!
It would be good if you (of Jeff from Zappos or anyone
making changes to this) could put up a new patch for this
most-voted-JIRA-issue.
Thanks,
Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
From: climbingrose
To: solr-user@lucene.apache.org
Sent: Sunday, April 19, 2009 8:12:11 AM
Subject: Re: CollapseFilter with the latest Solr in
trunk
Ok, here is how I fixed this problem:
public DocListAndSet
getDocListAndSet(Query query, ListfilterList,
DocSet docSet, Sort lsort, int offset, int len, int
flags) throwsIOException {
//DocListAndSet ret = new
DocListAndSet();
//getDocListC(ret,query,filterList,docSet,lsort,offset,len,
flags |=
GET_DOCSET);
DocSet theFilt = getDocSet(filterList);
if (docSet != null) theFilt = (theFilt != null) ?
theFilt.intersection(docSet) : docSet;
QueryCommand qc = new
QueryCommand();
qc.setQuery(query).setFilter(theFilt);
qc.setSort(lsort).setOffset(offset).setLen(len).setFlags(flags
|=
GET_DOCSET);
QueryResult result = new
QueryResult();
getDocListC(result,qc);
return
result.getDocListAndSet();
}
There is also one-off error in CollapseFilter which
you can find solution on
Jira.
Cheers,
Cuong
On Sat, Apr 18, 2009 at 4:41 AM, Jeff Newburn wrote:
We are currently trying to do the same
thing. With the patch unaltered we
can use fq as long as collapsing is turned
on. If we just send a normal
document level query with an fq parameter it
blows up.
Additionally, it does not appear that the
collapse.facet option works at
all.
--
Jeff Newburn
Software Engineer, Zappos.com
jnewb...@zappos.com
- 702-943-7562
From: climbingrose
Reply-To:
Date: Fri, 17 Apr 2009 16:53:00 +1000
To: solr-user
Subject: CollapseFilter with the latest Solr
in trunk
Hi all,
Have any one try to use CollapseFilter with
the latest version of Solr in
trunk? However, it looks like Solr 1.4
doesn't allow calling
setFilterList()
and setFilter() on one instance of the
QueryCommand. I modified the code
in
QueryCommand to allow this:
public QueryCommand
setFilterList(Query f) {
// if( filter != null )
{
//throw new
IllegalArgumentException( Either filter or
filterList
may be set in the QueryCommand, but not
both. );
// }
filterList =
null;
if (f !=
null) {
filterList = new ArrayList(2);
filterList.add(f);
}
return this;
}
However, I still have a problem which
prevent query filters from working
when used in conjunction with
CollapseFilter. In other words, query
filters
doesn't seem to have any effects on the
result set when CollapseFilter is
used.
The other problem is related to OpenBitSet:
java.lang.ArrayIndexOutOfBoundsException:
2183
at
org.apache.lucene.util.OpenBitSet.fastSet(OpenBitSet.java:242)
at
org.apache.solr.search.CollapseFilter.addDoc(CollapseFilter.java:202)
at
org.apache.solr.search.CollapseFilter.adjacentCollapse(CollapseFilter.java:161
)
at
org.apache.solr.search.CollapseFilter.(CollapseFilter.java:141)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:2
17)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandle
r.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
va:131)
at
org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303
)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:23
2)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
lterChain.java:202)
at
Re: Multiple Solr-instance share same solr.home
I think this should work, but it surely sounds unusual... which always makes one wonder why. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: vivek sar vivex...@gmail.com To: solr-user@lucene.apache.org Sent: Sunday, April 19, 2009 3:28:35 PM Subject: Re: Multiple Solr-instance share same solr.home Both Solr instances will be writing to separate indexes, but can they share the same solr.home? So, here is what I want, 1) solr.home = solr/multicore 2) There is a single solr.xml under multicore directory 3) Each instance would use the same solr.xml, which will have entries for multiple cores 4) Each instance will write to different core at a time - so one index will be written by only one writer at a time. not sure if this is a supported configuration. Thanks. -vivek On Sun, Apr 19, 2009 at 5:55 AM, Otis Gospodnetic wrote: Vivek - no, unless you want trouble - only 1 writer can write to a specific index at a time. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: vivek sar To: solr-user@lucene.apache.org Sent: Sunday, April 19, 2009 4:33:00 AM Subject: Multiple Solr-instance share same solr.home Hi, Is it possible to have two solr instances share the same solr.home? I've two Solr instances running on the same box and I was wondering if I can configure them to have the same solr.home. I tried it, but looks like the second instance overwrites the first one's value in the solr.xml (I'm using multicore for both instances). This is just for convenience so I don't have to manage multiple solr index directory locations - I can have all the indexes written into the same location and do the clean up from one place itself. If this is not supported then it's not a big deal. Thanks, -vivek
Re: Advice on moving from 1.3 to 1.4-dev or trunk?
There may be APIs which are introduced after 1.3 release and are not yet crystallized , may change by the time release happens. As a rule of thumb any feature in the CHANGES.TXT post 1.3 is possible to change . But as long as you know what you are using you should be OK. It is not quite possible to ensure that everything works perfectly . but, what you can do is, what ever features you use , test them thoroughly . If you face issues report them back --Noble On Fri, Apr 17, 2009 at 12:28 PM, ristretto.rb ristretto...@gmail.com wrote: I have built the trunk code as of Revision: 765826 and tried !tag=/!ex= which is what I need to work. And IT WORKS! That's great. Now, is it unwise to release 1.4 into production for this feature (based on my explanation below)? thanks gene -- Forwarded message -- From: ristretto.rb ristretto...@gmail.com Date: Fri, Apr 17, 2009 at 11:16 AM Subject: Advice on moving from 1.3 to 1.4-dev or trunk? To: solr-user@lucene.apache.org Hello, I'm using solr 1.3 with solr.py. We have a basic schema.xml, nothing custom or out of the ordinary. I need the following the feature from http://svn.apache.org/repos/asf/lucene/solr/trunk/CHANGES.txt SOLR-911: Add support for multi-select faceting by allowing filters to be tagged and facet commands to exclude certain filters. This patch also added the ability to change the output key for facets in the response, and optimized distributed faceting refinement by lowering parsing overhead and by making requests and responses smaller. Since this requires 1.4, looks like I have to upgrade (or roll my own solution that this feature provides.) I'm looking for a bit of advice. I have looked through the bugs here http://issues.apache.org/jira/browse/SOLR/fixforversion/12313351 1. I would need to get the source for 1.4 and build it, right? No release yet, eh? 2. Any one using 1.4 in production without issue; is this wise? Or should I wait? 3. Will I need to make changes to my schema.xml to support my current field set under 1.4? 4. Do I need to reindex all my data? thanks gene -- --Noble Paul
