RE: how to sampling search result

2016-09-28 Thread Yongtao Liu
, Yongtao Liu wrote: > Mikhail, > > Thanks for your reply. > > Random field is based on index time. > We want to do sampling based on search result. > > Like if the random field has value 1 - 100. > And the query touched documents may all in range 90 - 100. > So random

RE: how to remove duplicate from search result

2016-09-27 Thread Yongtao Liu
Shamik, Thanks a lot. Collapsing query parser solve the issue. Thanks, Yongtao -Original Message- From: shamik [mailto:sham...@gmail.com] Sent: Tuesday, September 27, 2016 3:09 PM To: solr-user@lucene.apache.org Subject: RE: how to remove duplicate from search result Did you take a look

RE: how to remove duplicate from search result

2016-09-27 Thread Yongtao Liu
David, Thanks for your reply. Group cannot solve the issue. We also need run facet and stats based on search result. With group, facet and stats result still count duplicate. Thanks, Yongtao -Original Message- From: David Santamauro [mailto:david.santama...@gmail.com] Sent: Tuesday, Sep

RE: how to sampling search result

2016-09-27 Thread Yongtao Liu
Mikhail, Thanks for your reply. Random field is based on index time. We want to do sampling based on search result. Like if the random field has value 1 - 100. And the query touched documents may all in range 90 - 100. So random field will not help. Is it possible we can sampling based on searc

remove user defined duplicate from search result

2016-09-26 Thread Yongtao Liu
Hi, I am try to remove user defined duplicate from search result. like below documents match the query. when query return, I try to remove doc3 from result since it has duplicate guid with doc1. Id (uniqueKey) guid doc1 G1 doc2 G2 doc3 G1 To do this, I generate exclude list based guid

RE: remove user defined duplicate from search result

2016-09-26 Thread Yongtao Liu
Sorry, the table is missing. Update below email with table. -Original Message- From: Yongtao Liu [mailto:y...@commvault.com] Sent: Monday, September 26, 2016 10:47 AM To: 'solr-user@lucene.apache.org' Subject: remove user defined duplicate from search result Hi, I am try to r

RE: memory usage keep increase

2011-11-17 Thread Yongtao Liu
to:erickerick...@gmail.com] Sent: Tuesday, November 15, 2011 8:37 AM To: solr-user@lucene.apache.org Subject: Re: memory usage keep increase I'm pretty sure not. The words "virtual memory address space" is important here, that's not physical memory... Best Erick On Mon, Nov 14,

memory usage keep increase

2011-11-14 Thread Yongtao Liu
Hi all, I saw one issue is ram usage keep increase when we run query. After look in the code, looks like Lucene use MMapDirectory to map index file to ram. According to http://lucene.apache.org/java/3_1_0/api/core/org/apache/lucene/store/MMapDirectory.html comments, it will use lot of memory.

Re: FW: MMapDirectory failed to map a 23G compound index segment

2011-09-21 Thread Yongtao Liu
, and there are lot of query which cause wide index file access. Then, the machine has no available memory. The system change to very slow. What i did is change lucene code to disable MMapDirectory. On Wed, Sep 21, 2011 at 1:26 PM, Yongtao Liu wrote: > > > -Original Message- > F