Re: [ANNOUNCE] Solr wiki editing change

AndrzejBialecki to this group. Thank you! -- Best regards, Andrzej Bialecki http://www.sigram.com, blog http://www.sigram.com/blog ___.,___,___,___,_._. __ [___||.__|__/|__||\/|: Information Retrieval, System Integration ___|||__||..\|..||..|: Contact: info at sigram dot com

Re: What is the docs number in Solr explain query results for fieldnorm?

be only used to co-ordinate parts of the query that matched the same document number. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System

Re: is there any practice to load index into RAM to accelerate solr performance?

://www.siam.org/proceedings/alenex/2008/alx08_01transierf.pdf http://research.google.com/pubs/archive/37365.pdf -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix

Re: Solr Lucene Index Version

there. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: Solr Lucene Index Version

LUCENE-3622. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: Can I delete the stored value?

LUCENE-1812 for another practical application of this concept. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com

Re: Feed index with analyzer output

/browse/SOLR-1535 -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: Performance loss - querying more than 64 cores (randomly)

dramatically (and the performance will drop then). Modern OS-es try to keep as much data in memory as possible, so the memory usage itself is not that informative - but check what are the pagein/pageout rates when you start hitting the 32 vs 64 cores. -- Best regards, Andrzej Bialecki

Re: Performance loss - querying more than 64 cores (randomly)

that excess of memory, but it won't be available for OS-level disk IO. Therefore reducing the heap size may actually increase your performance. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic

Re: Lucid Works

://www.lucidimagination.com/forum/ . -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot

Re: Lucid Works

around) then 3.1 is a safer bet. If you need a dozen or so new exciting features (e.g. results grouping) or top performance, or if you need LucidWorks with Click and other goodies, then use 4.x and be prepared for an occasional full reindex. -- Best regards, Andrzej Bialecki

Re: Lucid Works

to transparently convert indexes from one 4.x to another 4.x format, but they are not there yet. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration

Re: Lucid Works

a contrib/patch that was applied? At the moment it's proprietary. I will have a talk at the Lucene Revolution conference that describes the Click tools in detail. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information

Re: Detecting an empty index during start-up

. For now it's better to pass openNew=false and be prepared to get a null. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http

Re: a bug of solr distributed search

distributed RPC for every query... To summarize, I would qualify your statement with: ...if the composition of your shards is drastically different. Otherwise the cost of using global IDF is not worth it, IMHO. -- Best regards, Andrzej Bialecki

Re: a bug of solr distributed search

On 2010-10-25 13:37, Toke Eskildsen wrote: On Mon, 2010-10-25 at 11:50 +0200, Andrzej Bialecki wrote: * there is an exact solution to this problem, namely to make two distributed calls instead of one (first call to collect per-shard IDFs for given query terms, second call to submit a query

Re: Different analyzers for dfferent documents in different languages?

of a tokenizing chain that could use a language detector to create different fields (or tokenize differently) based on this decision. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web

Re: anyone use hadoop+solr?

hitting the trunk (right, Mark? ;) ), so medium-term I think this is your best bet. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http

SolrCloud distributed indexing (Re: anyone use hadoop+solr?)

convenience components that will make it easier. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info

Re: How to retrieve the full corpus

a spellchecker in another core (instead of using the current sub-index hack). -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http

Re: SolrCloud distributed indexing (Re: anyone use hadoop+solr?)

MOD won't do ;) so I think it would be good to hide this strategy behind an interface/abstract class. It costs nothing, and gives you flexibility in how you implement this mapping. -- Best regards, Andrzej Bialecki

Re: SolrCloud distributed indexing (Re: anyone use hadoop+solr?)

On 2010-09-06 22:03, Dennis Gearon wrote: What is a 'simple MOD'? md5(docId) % numShards -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System

Re: Analyser depending on field's value

, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: Auto-suggest internal terms

an awful lot of RAM... see SOLR-1316 for some measurements. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com

Re: Auto-suggest internal terms

On 2010-06-03 13:38, Michael Kuhlmann wrote: Am 03.06.2010 13:02, schrieb Andrzej Bialecki: ..., and deploy this index in a separate JVM (to benefit from other CPUs than the one that runs your Solr core) Every known webserver ist multithreaded by default, so putting different Solr

Re: Importing large datasets

statements that can then be fetched separately. You may want to write your own multithreaded client to index. SOLR-1301 is also an option if you are familiar with Hadoop ... -- Best regards, Andrzej Bialecki

Re: Importing large datasets

On 2010-06-02 13:12, Grant Ingersoll wrote: On Jun 2, 2010, at 6:53 AM, Andrzej Bialecki wrote: On 2010-06-02 12:42, Grant Ingersoll wrote: On Jun 1, 2010, at 9:54 PM, Blargy wrote: We have around 5 million items in our index and each item has a description located on a separate

Re: Autosuggest

On 2010-05-15 02:46, Blargy wrote: Thanks for your help and especially your analyzer.. probably saved me a full-import or two :) Also, take a look at this issue: https://issues.apache.org/jira/browse/SOLR-1316 -- Best regards, Andrzej Bialecki

Re: SOLR-1316 How To Implement this autosuggest component ???

to assign different priorities to suggestions (other than a simple IDF based priority), or have many city names consisting of multiple tokens, then use SOLR-1316. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information

Re: SOLR-1316 How To Implement this autosuggest component ???

equivalent to the TermsComponent; or from a list of frequent queries - but you need to build that list yourself). -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web

Re: SOLR-1316 How To Implement this autosuggest component ???

to the frequency of terms/phrases in the query logs ... TermsComponent and EdgeNGrams, while simple to use, suffer from both issues. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web

Re: wikipedia and teaching kids search engines

contrib/ is a quick and perhaps acceptable solution ... -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com

Re: multiple binary documents into a single solr document - Vignette/OpenText integration

an AddUpdateCommand to the update processor. You can obtain both the update processor and SolrCell instance from req.getCore(). -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web

Re: Features not present in Solr

. Could you perhaps elaborate a bit on this functionality? Your description sounds intriguing - it reminds me of ParallelReader, but I'm probably completely wrong ... -- Best regards, Andrzej Bialecki

Re: SOLR-1316 How To Implement this autosuggest component ???

that shows fragments of XML config files. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info

Re: Update Index : Updating Specific Fields

consists of 4 fields, F1, F2, F3, F4 Now I want to update the value of field F2, so if I send the update xml to SOLR, can it keep the old field values for F1,F3,F4 and update the new value specified for F2? Best Regards, Kranti K K Parisa -- Best regards, Andrzej Bialecki

Re: If you could have one feature in Solr...

). Kullback-Leibler divergence? -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: term frequency vector access?

be too costly. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: Can Solr be forced to return all field tags for a document even if the field is empty?l

out from the response... You can also implement a SearchComponent that post-processes results and based on the schema if a field is missing then it adds an empty node to the result. -- Best regards, Andrzej Bialecki

Re: How to Split Index file.

On 2010-01-10 01:55, Lance Norskog wrote: Make two copies of the index. In each copy, delete the records you do not want. Optimize. ... which is essentially what the MultiPassIndexSplitter does, only it avoids the initial copy (by deleting in the source index). -- Best regards, Andrzej

Re: restore space between words by spell checker

to solving this is to index compound words, i.e. when producing a spellchecker dictionary add a record tommyhitfiger with a field that points to tommy hitfiger. Details vary depending on what spellchecking impl. you use. -- Best regards, Andrzej Bialecki

Re: Index Splitter

, etc. The cost for this flexibility is that it needs to read index files multiple times (hence multi-pass). -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded

Re: how to get the autocomplete feature in solr 1.4?

-1316, there are patches there that implement such component using prefix trees. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration

Re: leading and trailing wildcard query

term appropriately. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: Solr Cell on web-based files?

regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: QTime always a multiple of 50ms ?

: [idx1] webapp=/solr path=/update/ params={} status=0 QTime=104 INFO: [idx1] webapp=/solr path=/update/ params={} status=0 QTime=52 ... Is this a known issue ? It may be an issue with System.currentTimeMillis() resolution on some platforms (e.g. Windows)? -- Best regards, Andrzej Bialecki

Re: Is negative boost possible?

Yonik Seeley wrote: On Mon, Oct 12, 2009 at 12:03 PM, Andrzej Bialecki a...@getopt.org wrote: Solr never discarded non-positive hits, and now Lucene 2.9 no longer does either. Hmm ... The code that I pasted in my previous email uses Searcher.search(Query, int), which in turn uses search(Query

Re: Is negative boost possible?

(a:b in 0), product of: 1.0 = tf(termFreq(a:b)=1) 0.30685282 = idf(docFreq=1, numDocs=1) 0.5 = fieldNorm(field=a, doc=0) bsh % -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic

Re: Is negative boost possible?

Yonik Seeley wrote: On Mon, Oct 12, 2009 at 5:58 AM, Andrzej Bialecki a...@getopt.org wrote: BTW, standard Collectors collect only results with positive scores, so if you want to collect results with negative scores as well then you need to use a custom Collector. Solr never discarded non

Re: Passing request to another handler

Shalin Shekhar Mangar wrote: On Fri, Oct 9, 2009 at 10:53 PM, Andrzej Bialecki a...@getopt.org wrote: Hi, What's the canonical way to pass an update request to another handler? I'm implementing a handler that has to dispatch its result to different update handlers based on its internal

Passing request to another handler

dependent on deployment paths defined in solrconfig.xml. Using SolrCore.getRequestHandlers(handler.class) often returns the LazyRequestHandlerWrapper, from which it's not possible to retrieve the wrapped instance of the handler .. -- Best regards, Andrzej Bialecki

Re: Where to place ReversedWildcardFilterFactory in Chain

regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: Number of terms in a SOLR field

- just get IndexReader.terms() enumeration and traverse it. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com

Re: Number of terms in a SOLR field

fields and term counts per field. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: Adding data from nutch to a Solr index

way involves writing a SearchComponent that does the latter part of that process on the Solr side.) -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix

Re: How to get a stack trace

a heap dump on OOM (it's a JVM flag) and then use a tool like HAT to find largest objects and references to them. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web

Re: Language Detection for Analysis?

/apache/nutch/analysis/lang/LanguageIdentifier.html -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact

Re: Solr Search probem w/ phrase searches, text type, w/ escaped characters

. Phrase queries that you can construct using QueryParser can't match two tokens separated by a hole, unless you set a slop value 0. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web

Re: Redhat vs FreeBSD vs other unix flavors

Otis Gospodnetic wrote: You should be fine on either Linux or FreeBSD (or any other UNIX flavour). Running on Solaris would probably give you access to goodness like dtrace, but you can live without it. There's dtrace on FreeBSD, too. -- Best regards, Andrzej Bialecki

Re: Integrating Solr and Nutch

to reindex your segments using the solrindex command, and change the searcher configuration. See nutch-default.xml for details. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web

Re: Please help me integrate Nutch with Solr

a couple days - please monitor this issue, and when it's done just download the patched code. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System

Re: [VOTE] Community Logo Preferences

/apache_solr_c_blue.jpg -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: TextProfileSigature using deduplication

in the proceedings of SIGIR-08, which presents an interesting and relatively simple algorithm that yields excellent results. Who has some spare CPU cycles to implement this? ;) http://ilpubs.stanford.edu:8090/860/ -- Best regards, Andrzej Bialecki

Re: TextProfileSigature using deduplication

) - this should work in your case. Ultimately, what you are probably looking for is a shingle-based algorithm, but it's relatively costly and requires multiple passes. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information

Re: maxFieldLength

the length of posting lists, which leads to increased memory/CPU consumption during decoding and traversing of the lists. Also, the overall increased number of positions will have an impact on the index size. -- Best regards, Andrzej Bialecki

Re: Adding bias to Distributed search feature?

straightforward, and relieves your from the need to de-duplicate your collection. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http

Re: Extending Solr with custom filter

(or Morfologik) for Polish stemming and lemmatization. It provides a superset of Stempel features, namely in addition to the algorithmic stemming it provides a dictionary-based stemming, and these two methods nicely complement each other. -- Best regards, Andrzej Bialecki

Re: Lucene-based Distributed Index Leveraging Hadoop

manager that knows the latest versions of each shard among the whole active set - or that clients discover this dynamically by querying the shard servers every now and then. -- Best regards, Andrzej Bialecki

Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)

). -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: multilingual list of stopwords

the correct stopword list. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com

Re: solr, snippets and stored field in nutch...

for which Nutch architecture is optimized. -- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info