Re: Problem using db-data-config.xml
On Thu, Jun 11, 2009 at 2:41 AM, jayakeerthi s mail2keer...@gmail.comwrote: As displayed above str name=*Total Requests made to DataSource**3739*/str * * str name=*Total Rows Fetched**4135*/str * * str name=*Total Documents Processed**1402*/str are differing The request to the datasource is increasing ..and the documents processed is less than the rows fetchedPlease advise If I am missing something here. You many entities, some of them nested within others. The rows is the count of number of rows retrieved for all entities but only the root-level entities create documents. -- Regards, Shalin Shekhar Mangar.
How to copyFeild to reverse string in another field
Hi How can I use copyField to copy field that contains normal text to another field and that field contains as reversed text? Best regards, Lee
Re: How to support stemming and case folding for english content mixed with non-english content?
Note: I request Solr users to go through this mail and let me thier ideas.
Thanks Yonik, you rightly pointed it out. That clearly says that the way I'm
trying to mimic the default behaviour of Solr indexing/searching in Lucene
is wrong, right?.
I downloaded the latest version of solr nightly on may20[at that time I was
using Solr, now switched to Lucene]. I hope the issue must have been fixed
with that version.Anyway I'm going to download the latest nightly build
today and try it out. I hope using the nightly build instead of getting the
src from latest trunk is more or less same[provided I donwload the latest
nightly build, right?]as I don't know much about getting/compiling the src
from solr trunk. Do let me know if I've to use the trunk anyway, in that
case I'm ready to spend time to get that done.
BTW, Yonik, as per the basic Solr schema.xml file, the analyzers/filters
used by default are these ones, correct me if I'm wrong,
this is the code snip that mentions the filters used for indexing in Solr
fieldType name=text class=solr.TextField positionIncrementGap=100
analyzer type=index
tokenizer class=solr.WhitespaceTokenizerFactory/
!-- in this example, we will only use synonyms at query time
filter class=solr.SynonymFilterFactory
synonyms=index_synonyms.txt ignoreCase=true expand=false/
--
!-- Case insensitive stop word removal.
enablePositionIncrements=true ensures that a 'gap' is left to
allow for accurate phrase queries.
--
filter class=solr.StopFilterFactory
ignoreCase=true
words=stopwords.txt
enablePositionIncrements=true
/
filter class=solr.WordDelimiterFilterFactory
generateWordParts=1 generateNumberParts=1 catenateWords=1
catenateNumbers=1 catenateAll=0 splitOnCaseChange=1/
filter class=solr.LowerCaseFilterFactory/
filter class=solr.EnglishPorterFilterFactory
protected=protwords.txt/
filter class=solr.RemoveDuplicatesTokenFilterFactory/
/analyzer
and this is the part used for Solr querying,
analyzer type=query
tokenizer class=solr.WhitespaceTokenizerFactory/
filter class=solr.SynonymFilterFactory synonyms=synonyms.txt
ignoreCase=true expand=true/
filter class=solr.StopFilterFactory ignoreCase=true
words=stopwords.txt/
filter class=solr.WordDelimiterFilterFactory
generateWordParts=1 generateNumberParts=1 catenateWords=0
catenateNumbers=0 catenateAll=0 splitOnCaseChange=1/
filter class=solr.LowerCaseFilterFactory/
filter class=solr.EnglishPorterFilterFactory
protected=protwords.txt/
filter class=solr.RemoveDuplicatesTokenFilterFactory/
/analyzer
/fieldType
To summarize the names are like this,
Indexing:
1. solr.WhitespaceTokenizerFactory -- tokenizer and the followings filters
as is clear from the name itself
2. solr.SynonymFilterFactory
3. solr.StopFilterFactory
4. solr.WordDelimiterFilterFactory (with the options as,
generateWordParts=1 generateNumberParts=1 catenateWords=1
catenateNumbers=1 catenateAll=0 splitOnCaseChange=1)
5. solr.LowerCaseFilterFactory
6. solr.EnglishPorterFilterFactory
7. solr.RemoveDuplicatesTokenFilterFactory
Querying:
1. solr.WhitespaceTokenizerFactory
2. solr.SynonymFilterFactory
3. solr.StopFilterFactory
4. solr.WordDelimiterFilterFactory( options are: generateWordParts=1
generateNumberParts=1 catenateWords=0 catenateNumbers=0
catenateAll=0 splitOnCaseChange=1)
5. solr.LowerCaseFilterFactory
6. solr.EnglishPorterFilterFactory
7. solr.RemoveDuplicatesTokenFilterFactory
Now the filters/analyzers I used that tried to mimic the above behavior of
Solr [in Lucene] is as show below.
I pulled out the whitespacedelimiterfilter from Solr and my custom analyzer
for indexing is like this,
/**
* Analyzer for Indian language.
*/
public class IndicAnalyzerIndex extends Analyzer {
public TokenStream tokenStream(String fieldName, Reader reader) {
TokenStream ts = new WhitespaceTokenizer(reader);
ts = new WordDelimiterFilter(ts, 1, 1, 1, 1, 0); // I tried using
...(ts, 1, 1, 1, 1, 0, 1) 7 params, but no constructor found for that, I
didn't try to modify the code to add this feature though, then used this
with 6 params, that uses the constructor for which the last option for
splitOnCaseChange is set to 1 so we're doing the same thing even in this
way...
ts = new StopFilter(ts, StopAnalyzer.ENGLISH_STOP_WORDS);
ts = new LowerCaseFilter(ts);
ts = new PorterStemFilter(ts);
return ts;
}
}
and for querying this is teh code
/**
* Analyzer for Indian language.
*/
public class IndicAnalyzerQuery extends Analyzer {
public TokenStream tokenStream(String fieldName, Reader reader) {
TokenStream ts = new WhitespaceTokenizer(reader);
ts = new WordDelimiterFilter(ts, 1, 1, 0, 0, 0);
ts = new StopFilter(ts,
Re: How to copyFeild to reverse string in another field
Hmm, you want to do that because sorl doesn't allow* and ? as the first character? On Jun 11, 2009, at 9:02 AM, chem leakhina wrote: Hi How can I use copyField to copy field that contains normal text to another field and that field contains as reversed text? Best regards, Lee -- Reza Safari LUKKIEN Copernicuslaan 15 6716 BM Ede The Netherlands - http://www.lukkien.com t: +31 (0) 318 698000 This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
How to use offset and limit in Solr
Hi How can I use offset or limit in Solr? Could you give some examples? Best regards, LEE
Re: How to use offset and limit in Solr
Use start and rows instead. See the common query parameters http://wiki.apache.org/solr/CommonQueryParameters - Markus Jelsma Buyways B.V. Tel. 050-3118123 Technisch ArchitectFriesestraatweg 215c Fax. 050-3118124 http://www.buyways.nl 9743 AD GroningenKvK 01074105 On Thu, 2009-06-11 at 15:23 +0700, chem leakhina wrote: Hi How can I use offset or limit in Solr? Could you give some examples? Best regards, LEE
Re: How to use offset and limit in Solr
Query Parameters: start=yourOffsetValuerows=yourLimitValue If you use a SolrJ client: SolrQuery query = new SolrQuery(); query.setQuery(yourQuery); query.setRows(yourLimitValue); query.setStart(yourOffsetValue); Cheers Avlesh On Thu, Jun 11, 2009 at 1:53 PM, chem leakhina chem.leakh...@gmail.comwrote: Hi How can I use offset or limit in Solr? Could you give some examples? Best regards, LEE
Search Phrase Wildcard?
Hi all, I have my document like this: doc nameSolr web service/name /doc Is there any ways that I can search like startswith: So* We* : found Sol*: found We*: not found Cheers, Samnang
Re: Customizing results
revas schrieb: What is GNU gettext and how this can be used in a multilanguage scenario? It'a an internationalization technology, so it is well suited to the tasks of internationalizing and localizing applications. http://www.gnu.org/software/gettext/manual/ http://www.gnu.org/software/gettext/manual/html_node/Why.html In your case, it might mean that the client is equipped with the language packages it needs and uses the name returned by Solr (likely the English term) to look up the translation by means of Gettext. But it certainly depends very much on your particular setup. It might be overkill for your particular situation. Michael Ludwig
Re: Build Failed
Mukerjee, Neiloy (Neil) schrieb: When running ant example to do an example configuration, I get the following message: BUILD FAILED /home/stagger2/Solr/apache-solr-1.3.0/common-build.xml:149: Compile failed; see the compiler error output for details. I've tried reading through the files in question, but I can't seem to find the issue. Any suggestions? Run: ant -verbose Michael Ludwig
Re: How to use offset and limit in Solr
Thanks On Thu, Jun 11, 2009 at 3:29 PM, Markus Jelsma - Buyways B.V. mar...@buyways.nl wrote: Use start and rows instead. See the common query parameters http://wiki.apache.org/solr/CommonQueryParameters - Markus Jelsma Buyways B.V. Tel. 050-3118123 Technisch ArchitectFriesestraatweg 215c Fax. 050-3118124 http://www.buyways.nl 9743 AD GroningenKvK 01074105 On Thu, 2009-06-11 at 15:23 +0700, chem leakhina wrote: Hi How can I use offset or limit in Solr? Could you give some examples? Best regards, LEE
Re: dismax parsing applied to specific fields
Nick Jenkin schrieb: Hi I was wondering if there is a way of applying dismax parsing to specific fields, where there are multiple fields being searched - all with different query values e.g. author:(tolkien) AND title:(the lord of the rings) would be something like: dismax(author, tolkien) AND dismax(title, the lord of the rings) I guess this can be thought of having two separate dismax configurations, one searching author and one searching title - and the intersection of the results is returned. http://wiki.apache.org/solr/DisMaxRequestHandler This says that the DisMaxRequestHandler is simply the standard request handler with the default query parser set to the DisMax Query Parser. So maybe you could program your own CustomDisMaxRequestHandler that reuses the DisMax query parser (and probably other components) to achieve what you want. Michael Ludwig
Re: Search Phrase Wildcard?
Solr does not support wildcards in phrase queries, yet. Cheers, Aleks On Thu, 11 Jun 2009 11:48:13 +0200, Samnang Chhun samnang.ch...@gmail.com wrote: Hi all, I have my document like this: doc nameSolr web service/name /doc Is there any ways that I can search like startswith: So* We* : found Sol*: found We*: not found Cheers, Samnang -- Aleksander M. Stensby Lead software developer and system architect Integrasco A/S www.integrasco.no http://twitter.com/Integrasco Please consider the environment before printing all or any of this e-mail
Re: Search Phrase Wildcard?
Infact, Lucene does not support that. Lucene supports single and multiple character wildcard searches within single terms (*not within phrase queries*). Taken from http://lucene.apache.org/java/2_3_2/queryparsersyntax.html#Wildcard%20Searches Cheers Avlesh On Thu, Jun 11, 2009 at 4:32 PM, Aleksander M. Stensby aleksander.sten...@integrasco.no wrote: Solr does not support wildcards in phrase queries, yet. Cheers, Aleks On Thu, 11 Jun 2009 11:48:13 +0200, Samnang Chhun samnang.ch...@gmail.com wrote: Hi all, I have my document like this: doc nameSolr web service/name /doc Is there any ways that I can search like startswith: So* We* : found Sol*: found We*: not found Cheers, Samnang -- Aleksander M. Stensby Lead software developer and system architect Integrasco A/S www.integrasco.no http://twitter.com/Integrasco Please consider the environment before printing all or any of this e-mail
Re: Search Phrase Wildcard?
Well yes:) Since Solr do infact support the entire lucene query parser syntax:) - Aleks On Thu, 11 Jun 2009 13:57:23 +0200, Avlesh Singh avl...@gmail.com wrote: Infact, Lucene does not support that. Lucene supports single and multiple character wildcard searches within single terms (*not within phrase queries*). Taken from http://lucene.apache.org/java/2_3_2/queryparsersyntax.html#Wildcard%20Searches Cheers Avlesh On Thu, Jun 11, 2009 at 4:32 PM, Aleksander M. Stensby aleksander.sten...@integrasco.no wrote: Solr does not support wildcards in phrase queries, yet. Cheers, Aleks On Thu, 11 Jun 2009 11:48:13 +0200, Samnang Chhun samnang.ch...@gmail.com wrote: Hi all, I have my document like this: doc nameSolr web service/name /doc Is there any ways that I can search like startswith: So* We* : found Sol*: found We*: not found Cheers, Samnang -- Aleksander M. Stensby Lead software developer and system architect Integrasco A/S www.integrasco.no http://twitter.com/Integrasco Please consider the environment before printing all or any of this e-mail -- Aleksander M. Stensby Lead software developer and system architect Integrasco A/S www.integrasco.no http://twitter.com/Integrasco Please consider the environment before printing all or any of this e-mail
Re: Faceting on text fields
FYI. I did a direct integration with Carrot2 with Solrj with a separate Ajax call from UI for top 100 hits to clusters terms in the two text fields. It gots comparable performance to other facets in terms of response time. In terms of algorithms, their listed two Lingo and STC which I don't reconize. But I think at least one of them might have used SVD (http://en.wikipedia.org/wiki/Singular_value_decomposition). -Yao Otis Gospodnetic wrote: I'd call it related (their application in search encourages exploration), but also distinct enough to never mix them up. I think your assessment below is correct, although I'm not familiar with the details of Carrot2 any more (was once), so I can't tell you exactly which algo is used under the hood. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Michael Ludwig m...@as-guides.com To: solr-user@lucene.apache.org Sent: Wednesday, June 10, 2009 9:41:54 AM Subject: Re: Faceting on text fields Otis Gospodnetic schrieb: Solr can already cluster top N hits using Carrot2: http://wiki.apache.org/solr/ClusteringComponent Would it be fair to say that clustering as detailed on the page you're referring to is a kind of dynamic faceting? The faceting not being done based on distinct values of certain fields, but on the presence (and frequency) of terms in one field? The main difference seems to be that with faceting, grouping criteria (facets) are known beforehand, while with clustering, grouping criteria (the significant terms which create clusters - the cluster keys) have yet to be determined. Is that a correct assessment? Michael Ludwig -- View this message in context: http://www.nabble.com/Faceting-on-text-fields-tp23872891p23980124.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Build Failed
Running ant -verbose still doesn't allow me to run an example configuration. I get the same error from ant example after getting the following from ant -verbose: Apache Ant version 1.7.0 compiled on August 29 2007 Buildfile: build.xml Detected Java version: 1.6 in: /usr/lib/jvm/java-1.5.0-gcj-4.2-1.5.0.0/jre Detected OS: Linux parsing buildfile /home/stagger2/Solr/apache-solr-1.3.0/build.xml with URI = file:/home/stagger2/Solr/apache-solr-1.3.0/build.xml Project base dir set to: /home/stagger2/Solr/apache-solr-1.3.0 [antlib:org.apache.tools.ant] Could not load definitions from resource org/apache/tools/ant/antlib.xml. It could not be found. Importing file /home/stagger2/Solr/apache-solr-1.3.0/common-build.xml from /home/stagger2/Solr/apache-solr-1.3.0/build.xml parsing buildfile /home/stagger2/Solr/apache-solr-1.3.0/common-build.xml with URI = file:/home/stagger2/Solr/apache-solr-1.3.0/common-build.xml [property] Loading /home/stagger2/Solr/apache-solr-1.3.0/build.properties [property] Unable to find property file: /home/stagger2/Solr/apache-solr-1.3.0/build.properties Override ignored for property DSTAMP Override ignored for property TSTAMP [available] Unable to load class org.apache.maven.artifact.ant.Pom to set property maven.ant.tasks.present [available] Unable to load class com.cenqua.clover.tasks.CloverReportTask to set property clover.present [macrodef] creating macro solr-javac [macrodef] creating macro solr-jar [macrodef] creating macro solr-checksum [macrodef] creating macro contrib-crawl [macrodef] creating macro invoke-javadoc Property pom.xml has not been set Property jar.file has not been set [macrodef] creating macro m2-deploy [available] Unable to load class com.cenqua.clover.tasks.CloverReportTask to set property clover.present Build sequence for target(s) `usage' is [usage] Complete build sequence is [usage, common.make-manifest, init-forrest-entities, compile-common, compile, compile-solrj-core, compile-solrj, compileTests, junit, test-core, build-contrib, test-contrib, test, make-manifest, dist-jar, dist-solrj, dist-contrib, dist-war, dist, example, javadoc-core, javadoc-common, javadoc-contrib, javadoc-solrj, javadoc-all, javadoc, dist-src, dist-javadoc, create-package, nightly, clover.setup, common.dist-contrib, clean-contrib, clean, svn-up, build-site, maven.ant.tasks-check, generate-maven-artifacts, package, prepare-release, clean-dest, common.test-contrib, -taskdef, stub-factories, rat-sources, test-reports, clover.info, clover.check, clover, generate-clover-reports, common.maven.ant.tasks-check, common.build-contrib, common.clean-contrib, ] usage: [echo] Welcome to the Solr project! [echo] Use 'ant example' to create a runnable example configuration. [echo] And for developers: [echo] Use 'ant clean' to clean compiled files. [echo] Use 'ant compile' to compile the source code. [echo] Use 'ant dist' to build the project WAR and JAR files. [echo] Use 'ant generate-maven-artifacts' to generate maven artifacts. [echo] Use 'ant package' to generate zip, tgz, and maven artifacts for distribution. [echo] Use 'ant test' to run unit tests. BUILD SUCCESSFUL Total time: 1 second -Original Message- From: Michael Ludwig [mailto:m...@as-guides.com] Sent: Thursday, June 11, 2009 6:05 AM To: solr-user@lucene.apache.org Subject: Re: Build Failed Mukerjee, Neiloy (Neil) schrieb: When running ant example to do an example configuration, I get the following message: BUILD FAILED /home/stagger2/Solr/apache-solr-1.3.0/common-build.xml:149: Compile failed; see the compiler error output for details. I've tried reading through the files in question, but I can't seem to find the issue. Any suggestions? Run: ant -verbose Michael Ludwig
Re: Search Phrase Wildcard?
You might be interested in this Lucene issue: https://issues.apache.org/jira/browse/LUCENE-1486 Aleksander M. Stensby wrote: Well yes:) Since Solr do infact support the entire lucene query parser syntax:) - Aleks On Thu, 11 Jun 2009 13:57:23 +0200, Avlesh Singh avl...@gmail.com wrote: Infact, Lucene does not support that. Lucene supports single and multiple character wildcard searches within single terms (*not within phrase queries*). Taken from http://lucene.apache.org/java/2_3_2/queryparsersyntax.html#Wildcard%20Searches Cheers Avlesh On Thu, Jun 11, 2009 at 4:32 PM, Aleksander M. Stensby aleksander.sten...@integrasco.no wrote: Solr does not support wildcards in phrase queries, yet. Cheers, Aleks On Thu, 11 Jun 2009 11:48:13 +0200, Samnang Chhun samnang.ch...@gmail.com wrote: Hi all, I have my document like this: doc nameSolr web service/name /doc Is there any ways that I can search like startswith: So* We* : found Sol*: found We*: not found Cheers, Samnang -- Aleksander M. Stensby Lead software developer and system architect Integrasco A/S www.integrasco.no http://twitter.com/Integrasco Please consider the environment before printing all or any of this e-mail -- - Mark http://www.lucidimagination.com
Re: Build Failed
On Thu, Jun 11, 2009 at 5:49 PM, Mukerjee, Neiloy (Neil) neil.muker...@alcatel-lucent.com wrote: Running ant -verbose still doesn't allow me to run an example configuration. I get the same error from ant example after getting the following from ant -verbose: Apache Ant version 1.7.0 compiled on August 29 2007 Buildfile: build.xml Detected Java version: 1.6 in: /usr/lib/jvm/java-1.5.0-gcj-4.2-1.5.0.0/jre You are using GCJ. Switch to Sun JDK. -- Regards, Shalin Shekhar Mangar.
Re: solr in distributed mode
Hi, i went through the document: http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr i have a couple of questions: 1. In the document its been mentioned that There will be a 'master' server for each shard and then 1-n 'slaves' that are replicated from the master. how is the replication process done? suppose i have 2 machines nodeA and nodeB I edited scripts.config in solr/conf of both nodeA and nodeB to point to the master (i.e. nodeA). i) is it the right approach for setting up master/slave configuration? ii) to start the master/slave config, should i execute start.jar from both the nodes? or just from the master node? iii) are indexes automatically replicated when you insert/update it in the master.. or do we have to run a script for that? iv) how do i know if replication process is sucessfully carried out. v) suppose the master goes down. i do i perform a node failover.. for example make one of the slaves as master without disrupting my application? 2. It has also been mentioned that: With distribution and replication, none of the master shards know about each other. You index to each master, the index is replicated to each slave, and then searches are distributed across the slaves, using one slave from each master/slave shard. i) Are slaves used only for index replications? i mean can't i have indexes distributed across slaves so that when i perform a search, it searches across all slaves? ii) since none of the shards have any information about one another, if i update/delete the document based on term, how does the index gets updated across all shards? or do we have to merge, update/delete and then distribute it across shards? Regards, Rakahi In a distributed configuration, one server 'shard' will get a query request and then search itself, as well as the other shards in the configuration, and return the combined results from each shard. On Wed, Jun 10, 2009 at 11:23 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Hello, All of this is covered on the Wiki, search for: distributed search Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Rakhi Khatwani rkhatw...@gmail.com To: solr-user@lucene.apache.org Cc: ninad.r...@germinait.com; ranjit.n...@germinait.com; saurabh.maha...@germinait.com Sent: Tuesday, June 9, 2009 4:55:55 AM Subject: solr in distributed mode Hi, I was looking for ways in which we can use solr in distributed mode. is there anyways we can use solr indexes across machines or by using Hadoop Distributed File System? Its has been mentioned in the wiki that When an index becomes too large to fit on a single system, or when a single query takes too long to execute, an index can be split into multiple shards, and Solr can query and merge results across those shards. what i understand is that shards are a partition. are shards on the same machine or can it be on different machines?? do we have to manually split the indexes to store in different shards. do you have an example or some tutorial which demonstrates distributed index searching/ storing using shards? Regards, Raakhi
Re: Build Failed
Mukerjee, Neiloy (Neil) schrieb: Running ant -verbose still doesn't allow me to run an example configuration. I get the same error from ant example after getting the following from ant -verbose: Build sequence for target(s) `usage' is [usage] usage: [echo] Welcome to the Solr project! [echo] Use 'ant example' to create a runnable example configuration. [echo] And for developers: [echo] Use 'ant clean' to clean compiled files. [echo] Use 'ant compile' to compile the source code. [echo] Use 'ant dist' to build the project WAR and JAR files. [echo] Use 'ant generate-maven-artifacts' to generate maven artifacts. [echo] Use 'ant package' to generate zip, tgz, and maven artifacts for distribution. [echo] Use 'ant test' to run unit tests. BUILD SUCCESSFUL You might want to read up on Ant usage in the Ant User Manual, a copy of which should be part of your installation, or can be found on the web. Quick overview: ant -help When I wrote ant -verbose, I meant ant -verbose your-target, so: ant -verbose example Michael Ludwig
Re: Solr update performance decrease after a while
Thank for your answer, I will have a look and try to find a solution. Cheers -- View this message in context: http://www.nabble.com/Solr-update-performance-decrease-after-a-while-tp23945947p23980681.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr update performance decrease after a while
2009/6/9 Vincent Pérès vincent.pe...@gmail.com: We are indexing approximately 500 documents per day. My benchmark says an update is done in 0.7 sec just after Solr has been started. But it quickly decrease to 2.2 secs per update ! Per document added, or for the whole update + commit? I have just been focused on the Schema until now, and didn't changed many stuffs in the solrconfig file. Maybe you have some tips which could help me to be more linear ? Auto-warming (filling of caches based on previous queries) is enabled by default and happens with each commit. You may want to disable this (set autowarmCount=0 on the caches in solrconfig.xml). -Yonik http://www.lucidimagination.com
Re: Faceting on text fields
BTW, Carrot2 has a very impressive Clustering Workbench (based on eclipse) that has built-in integration with Solr. If you have a Solr service running, it is a just a matter of point the workbench to it. The clustering results and visualization are amazing. (http://project.carrot2.org/download.html). Yao Ge wrote: FYI. I did a direct integration with Carrot2 with Solrj with a separate Ajax call from UI for top 100 hits to clusters terms in the two text fields. It gots comparable performance to other facets in terms of response time. In terms of algorithms, their listed two Lingo and STC which I don't reconize. But I think at least one of them might have used SVD (http://en.wikipedia.org/wiki/Singular_value_decomposition). -Yao Otis Gospodnetic wrote: I'd call it related (their application in search encourages exploration), but also distinct enough to never mix them up. I think your assessment below is correct, although I'm not familiar with the details of Carrot2 any more (was once), so I can't tell you exactly which algo is used under the hood. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Michael Ludwig m...@as-guides.com To: solr-user@lucene.apache.org Sent: Wednesday, June 10, 2009 9:41:54 AM Subject: Re: Faceting on text fields Otis Gospodnetic schrieb: Solr can already cluster top N hits using Carrot2: http://wiki.apache.org/solr/ClusteringComponent Would it be fair to say that clustering as detailed on the page you're referring to is a kind of dynamic faceting? The faceting not being done based on distinct values of certain fields, but on the presence (and frequency) of terms in one field? The main difference seems to be that with faceting, grouping criteria (facets) are known beforehand, while with clustering, grouping criteria (the significant terms which create clusters - the cluster keys) have yet to be determined. Is that a correct assessment? Michael Ludwig -- View this message in context: http://www.nabble.com/Faceting-on-text-fields-tp23872891p23980959.html Sent from the Solr - User mailing list archive at Nabble.com.
DataImportHandler backwards compatibility
Hi, I'm in the process of implementing a DataImportHandler config for Solr 1.3 and I've hit across the Postgresql/JDBC Out Of Memory problem. Whilst the solution is documented on the wiki FAQ page: http://wiki.apache.org/solr/DataImportHandlerFaq it appears that the JDBC driver parameters were implemented in DataImportHandler post the 1.3 release. I was wondering if it would be safe to take a nightly build of just the DataImportHandler contrib and run it against a Solr 1.3 installation? Many thanks, kevin
Re: DataImportHandler backwards compatibility
On Thu, Jun 11, 2009 at 6:42 PM, Kevin Lloyd kll...@lulu.com wrote: I'm in the process of implementing a DataImportHandler config for Solr 1.3 and I've hit across the Postgresql/JDBC Out Of Memory problem. Whilst the solution is documented on the wiki FAQ page: http://wiki.apache.org/solr/DataImportHandlerFaq it appears that the JDBC driver parameters were implemented in DataImportHandler post the 1.3 release. Yes, those parameters are new in 1.4 (we should note that on the wiki). I was wondering if it would be safe to take a nightly build of just the DataImportHandler contrib and run it against a Solr 1.3 installation? Solr 1.4 has a rollback command which 1.3 did not have. So, you'd need to hack the DataImportHandler code to remove references to RollBackCommand. You can use the 1.4 dih jar with 1.3 if you comment out the code in SolrWriter.rollback method, remove the import of RollbackUpdateCommand and recompile. -- Regards, Shalin Shekhar Mangar.
Re: Faceting on text fields
Yao Ge schrieb: BTW, Carrot2 has a very impressive Clustering Workbench (based on eclipse) that has built-in integration with Solr. If you have a Solr service running, it is a just a matter of point the workbench to it. The clustering results and visualization are amazing. (http://project.carrot2.org/download.html). A new world opens up for me ... Thanks for pointing out how cool this is! Hint for other newcomers: Open the View Menu to configure the details of how you perform your search, e.g. your Solr URL in case it differs from the default, or your summary field, which is what gets used to analyze the data in order to determine clusters, if I understand correctly. Michael Ludwig
Getting details from delete
Hello, Is there any way to get the number of deleted records from a delete request? I'm sending: deletequerytype_i:(2 OR 3) AND creation_time_rl:[0 TO 124426080]/query/delete And getting: ?xml version=1.0 encoding=UTF-8? response lst name=responseHeaderint name=status0/intint name=QTime2/int/lst /response This is Solr 1.3. -- Jacob Elder
Re: Upgrading 1.2.0 to 1.3.0 solr
I disagree with waiting that month. At this point, most of the kinks in
the upgrade from 1.2 to 1.3 have been worked out. Waiting for 1.4 to
come out risks you becoming a guinea pig for the upgrade procedure.
Plus, if any show-stoppers come along delaying 1.4, you delay
implementation of your auto-complete function. When 1.4 comes out, if
it has any features you feel compel an upgrade, you can begin another
round of testing and migration, but don't upgrade a production system
just for the sake of being bleeding edge.
Ryan T. Grange, IT Manager
DollarDays International, Inc.
rgra...@dollardays.com (480)922-8155 x106
Otis Gospodnetic wrote:
Francis,
If you can wait another month or so, you could skip 1.3.0, and jump to 1.4
which will be released soon.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
From: Francis Yakin fya...@liquid.com
To: solr-user@lucene.apache.org solr-user@lucene.apache.org
Sent: Wednesday, June 10, 2009 1:17:25 AM
Subject: Upgrading 1.2.0 to 1.3.0 solr
I am in process to upgrade our solr 1.2.0 to solr 1.3.0
Our solr 1.2.0 now is working fine, we just want to upgrade it cause we have an
application that requires some function from 1.3.0( we call it autocomplete).
Currently our config files on 1.2.0 are as follow:
Solrconfig.xml
Schema.xml ( we wrote this in house)
Index_synonyms.txt ( we also modified and wrote this in house)
Scripts.conf
Protwords.txt
Stopwords.txt
Synonyms.txt
I understand on 1.3.0 , it has new solrconfig.xml .
My questions are:
1) what config files that I can reuse from 1.2.0 for 1.3.0
can I use the same schema.xml
2) Solrconfig.xml, can I use the 1.2.0 version or I have to stick with 1.3.0
If I need to stick with 1.3.0, what that I need to change.
As of right I am testing it in my sandbox, so it doesn't work.
Please advice, if you have any docs for upgrading 1.2.0 to 1.3.0 let me know.
Thanks in advance
Francis
Note: I attached my solrconfigand schema.xml in this email
-Inline Attachment Follows-
{edited out by Ryan for brevity}
correct? impossible to filter / facet on ExternalFileField
in our design some often changing fields would benefit from being defined as ExternalFileFields, so we can index them more often than the rest. However we need to filter and facet on them. I don't think that this currently is possible with ExternalFileField but just want to make sure. if not possible, is it on the roadmap? Thanks, Britske -- View this message in context: http://www.nabble.com/correct--impossible-to-filter---facet-on-ExternalFileField-tp23985106p23985106.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: correct? impossible to filter / facet on ExternalFileField
On Thu, Jun 11, 2009 at 12:53 PM, Britskegbr...@gmail.com wrote: in our design some often changing fields would benefit from being defined as ExternalFileFields, so we can index them more often than the rest. However we need to filter and facet on them. I don't think that this currently is possible with ExternalFileField but just want to make sure. if not possible, is it on the roadmap? You're right, it's not currently possible. We don't have any official roadmap, but we've thought about updateable fields (the other way to accomplish this sort of thing) and faceting/searching on an external source like ExternalFileField. -Yonik http://www.lucidimagination.com
RE: Upgrading 1.2.0 to 1.3.0 solr
DO you have experience to upgrade from 1.2.0 to 1.3.0?
In other words, do you have any suggestions or best if you have any docs or
instructions for doing this.
I appreciate if you can help me.
Thanks
Francis
-Original Message-
From: Ryan Grange [mailto:rgra...@dollardays.com]
Sent: Thursday, June 11, 2009 8:39 AM
To: solr-user@lucene.apache.org
Subject: Re: Upgrading 1.2.0 to 1.3.0 solr
I disagree with waiting that month. At this point, most of the kinks in the
upgrade from 1.2 to 1.3 have been worked out. Waiting for 1.4 to come out
risks you becoming a guinea pig for the upgrade procedure.
Plus, if any show-stoppers come along delaying 1.4, you delay implementation of
your auto-complete function. When 1.4 comes out, if it has any features you
feel compel an upgrade, you can begin another round of testing and migration,
but don't upgrade a production system just for the sake of being bleeding edge.
Ryan T. Grange, IT Manager
DollarDays International, Inc.
rgra...@dollardays.com (480)922-8155 x106
Otis Gospodnetic wrote:
Francis,
If you can wait another month or so, you could skip 1.3.0, and jump to 1.4
which will be released soon.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
From: Francis Yakin fya...@liquid.com
To: solr-user@lucene.apache.org solr-user@lucene.apache.org
Sent: Wednesday, June 10, 2009 1:17:25 AM
Subject: Upgrading 1.2.0 to 1.3.0 solr
I am in process to upgrade our solr 1.2.0 to solr 1.3.0
Our solr 1.2.0 now is working fine, we just want to upgrade it cause we have
an application that requires some function from 1.3.0( we call it
autocomplete).
Currently our config files on 1.2.0 are as follow:
Solrconfig.xml
Schema.xml ( we wrote this in house)
Index_synonyms.txt ( we also modified and wrote this in house)
Scripts.conf Protwords.txt Stopwords.txt Synonyms.txt
I understand on 1.3.0 , it has new solrconfig.xml .
My questions are:
1) what config files that I can reuse from 1.2.0 for 1.3.0
can I use the same schema.xml
2) Solrconfig.xml, can I use the 1.2.0 version or I have to stick with 1.3.0
If I need to stick with 1.3.0, what that I need to change.
As of right I am testing it in my sandbox, so it doesn't work.
Please advice, if you have any docs for upgrading 1.2.0 to 1.3.0 let me know.
Thanks in advance
Francis
Note: I attached my solrconfigand schema.xml in this email
-Inline Attachment Follows-
{edited out by Ryan for brevity}
localSolr + collapse
Hi, I am using localSolr with collapse. Query response gives two result sets one generated by localSolr based on geo distance and other by collapsing fields. How can I get combined result set? SolrConfig.xml requestHandler name=geo class=org.apache.solr.handler.component.SearchHandler lst name=defaults str name=echoParamsexplicit/str float name=tie0.01/float /lst lst name=invariants str name=spellcheck.collatetrue/str str name=spellchecktrue/str str name=facettrue/str str name=facet.mincount1/str /lst arr name=components strcollapse/str strlocalsolr/str strfacet/str strdebug/str /arr /requestHandler I am using solr 1.4 nightly build with collapse patch. Localsolr version is 1.5 Query: http://localhost:8080/solr/select?q=TVqt=geocollapse=truecollapse.fie ld=resource_idlat=41.883784long=-87.637668radius=100 Regards Chandra
how to get to highlitghting results using solrJ
first time I'm using highlighting and results work ok.
Im using it for an auto-suggest function. For reference I used the
following query:
http://localhost:8983/solr/autocompleteCore/select?fl=name_display,importance,score,hlidwt=xmlq={!boost
b=log(importance)}(prefix1:or OR prefix2:or)hl=truehl.fl=prefix1
However, when using solrJ I can't get to the actual highlighted results,
i.e:
QueryResponse.getHighlighting() shows me a map as follows:
{2-1-57010={}, 2-7-8481={}, } which I can't use because the result is
empty.(?)
but debugging I see a field:
QueryResponse._highlightingInfo with contents:
{1-4167147={prefix1=[emOrl/emando Verenigde Staten]},}
which is exactly what I need.
However there is no (public) method:
QueryRepsonse.getHighlightingInfo() !
what am I missing?
thanks,
Britske
--
View this message in context:
http://www.nabble.com/how-to-get-to-highlitghting-results-using-solrJ-tp23986063p23986063.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to get to highlitghting results using solrJ
the query contained some experimenting code. The correct one is:
http://localhost:8983/solr/autocompleteCore/select?fl=name_display,importance,scorewt=xmlq={!boost
b=log(importance)}(prefix1:or OR prefix2:or)hl=truehl.fl=prefix1
Moreover, Is there a way to simply add the result of highlighting to the
fl-parameter, so I can just read the annotated name (including highlighting)
instead of the normal name. (analogously as you can apply 'score' to fl.)
To me, this would seem like the perfect way to get the highlighted result
without having to supply additional code in a client. You would only need to
refer to the annotated field name...
Britske wrote:
first time I'm using highlighting and results work ok.
Im using it for an auto-suggest function. For reference I used the
following query:
http://localhost:8983/solr/autocompleteCore/select?fl=name_display,importance,score,hlidwt=xmlq={!boost
b=log(importance)}(prefix1:or OR prefix2:or)hl=truehl.fl=prefix1
However, when using solrJ I can't get to the actual highlighted results,
i.e:
QueryResponse.getHighlighting() shows me a map as follows:
{2-1-57010={}, 2-7-8481={}, } which I can't use because the result is
empty.(?)
but debugging I see a field:
QueryResponse._highlightingInfo with contents:
{1-4167147={prefix1=[emOrl/emando Verenigde Staten]},}
which is exactly what I need.
However there is no (public) method:
QueryRepsonse.getHighlightingInfo() !
what am I missing?
thanks,
Britske
--
View this message in context:
http://www.nabble.com/how-to-get-to-highlitghting-results-using-solrJ-tp23986063p23986127.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Filter fq with OR operator
Adding my vote for this feature. Another possible way to do this would be to introduce the ofq parameter .. so all of the fq's and AND'ed and the ofq's would be OR'd I've got some really complicated fq's that aren't going to cache well, introducing OR'd filters would allow me to use simpler filters and I expect performance would be a lot better Thanks -- View this message in context: http://www.nabble.com/Query-Filter-fq-with-OR-operator-tp23895837p23986336.html Sent from the Solr - User mailing list archive at Nabble.com.
can Trie fields be stored?
Looking at the new examples of solr.TrieField http://svn.apache.org/repos/asf/lucene/solr/trunk/example/solr/conf/schema.xml I see that all have indexed=true stored=false in the field tpye definition. Does this mean that yo cannot ever store a value for one of these fields? I.e. if I want to do a range query and also return the values, I need to store the values in a separate field? Thanks, Peter -- Peter M. Wolanin, Ph.D. Momentum Specialist, Acquia. Inc. peter.wola...@acquia.com
Re: Faceting on text fields
Hi all, We are thinking of using the carrot clustering too. But we saw that carrot maybe can only cluster up to 1000 search snippets. Does anyone know how can we cluster snippets that is much more than that ? (maybe in the million range?) And what is the difference between mahout and carrot? Thank! Jeffrey On Thu, Jun 11, 2009 at 9:47 PM, Michael Ludwig m...@as-guides.com wrote: Yao Ge schrieb: BTW, Carrot2 has a very impressive Clustering Workbench (based on eclipse) that has built-in integration with Solr. If you have a Solr service running, it is a just a matter of point the workbench to it. The clustering results and visualization are amazing. (http://project.carrot2.org/download.html). A new world opens up for me ... Thanks for pointing out how cool this is! Hint for other newcomers: Open the View Menu to configure the details of how you perform your search, e.g. your Solr URL in case it differs from the default, or your summary field, which is what gets used to analyze the data in order to determine clusters, if I understand correctly. Michael Ludwig
change data dir location
I want to change data dir location and not use solrConfig.xml. So on windows I removed dataDir xonfiguration from solrConfig.xml and tried setting 'solr.data.dir' system property to my location. But it looks like solr is not picking up this property on windows. Anybody experienced the same??? Please share... Thanks, Ashish -- View this message in context: http://www.nabble.com/change-data-dir-location-tp23992946p23992946.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Faceting on text fields
Jeffrey, Are you looking to cluster a whole corpus of documents of just the search results? If it's the latter, use Carrot2. If it's the former, look at Mahout. Clustering top 1M matching documents doesn't really make sense. Usually top 100-200 is sufficient. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Jeffrey Tiong jeffrey.ti...@gmail.com To: solr-user@lucene.apache.org Sent: Friday, June 12, 2009 12:44:55 AM Subject: Re: Faceting on text fields Hi all, We are thinking of using the carrot clustering too. But we saw that carrot maybe can only cluster up to 1000 search snippets. Does anyone know how can we cluster snippets that is much more than that ? (maybe in the million range?) And what is the difference between mahout and carrot? Thank! Jeffrey On Thu, Jun 11, 2009 at 9:47 PM, Michael Ludwig wrote: Yao Ge schrieb: BTW, Carrot2 has a very impressive Clustering Workbench (based on eclipse) that has built-in integration with Solr. If you have a Solr service running, it is a just a matter of point the workbench to it. The clustering results and visualization are amazing. (http://project.carrot2.org/download.html). A new world opens up for me ... Thanks for pointing out how cool this is! Hint for other newcomers: Open the View Menu to configure the details of how you perform your search, e.g. your Solr URL in case it differs from the default, or your summary field, which is what gets used to analyze the data in order to determine clusters, if I understand correctly. Michael Ludwig
Re: change data dir location
set the dataDir value in solrconfig.xml to what you like On Fri, Jun 12, 2009 at 10:38 AM, Ashish Pashish.ping...@gmail.com wrote: I want to change data dir location and not use solrConfig.xml. So on windows I removed dataDir xonfiguration from solrConfig.xml and tried setting 'solr.data.dir' system property to my location. But it looks like solr is not picking up this property on windows. Anybody experienced the same??? Please share... Thanks, Ashish -- View this message in context: http://www.nabble.com/change-data-dir-location-tp23992946p23992946.html Sent from the Solr - User mailing list archive at Nabble.com. -- - Noble Paul | Principal Engineer| AOL | http://aol.com
Re: change data dir location
I don't want to set in solrConfig.xml. I want solr to take from my config file or from system property. Thanks, Ashish Noble Paul നോബിള് नोब्ळ्-2 wrote: set the dataDir value in solrconfig.xml to what you like On Fri, Jun 12, 2009 at 10:38 AM, Ashish Pashish.ping...@gmail.com wrote: I want to change data dir location and not use solrConfig.xml. So on windows I removed dataDir xonfiguration from solrConfig.xml and tried setting 'solr.data.dir' system property to my location. But it looks like solr is not picking up this property on windows. Anybody experienced the same??? Please share... Thanks, Ashish -- View this message in context: http://www.nabble.com/change-data-dir-location-tp23992946p23992946.html Sent from the Solr - User mailing list archive at Nabble.com. -- - Noble Paul | Principal Engineer| AOL | http://aol.com -- View this message in context: http://www.nabble.com/change-data-dir-location-tp23992946p23993210.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Faceting on text fields
Thanks Otis! Do you know under what circumstances or application should we cluster the whole corpus of documents vs just the search results? Jeffrey On Fri, Jun 12, 2009 at 1:39 PM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Jeffrey, Are you looking to cluster a whole corpus of documents of just the search results? If it's the latter, use Carrot2. If it's the former, look at Mahout. Clustering top 1M matching documents doesn't really make sense. Usually top 100-200 is sufficient. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Jeffrey Tiong jeffrey.ti...@gmail.com To: solr-user@lucene.apache.org Sent: Friday, June 12, 2009 12:44:55 AM Subject: Re: Faceting on text fields Hi all, We are thinking of using the carrot clustering too. But we saw that carrot maybe can only cluster up to 1000 search snippets. Does anyone know how can we cluster snippets that is much more than that ? (maybe in the million range?) And what is the difference between mahout and carrot? Thank! Jeffrey On Thu, Jun 11, 2009 at 9:47 PM, Michael Ludwig wrote: Yao Ge schrieb: BTW, Carrot2 has a very impressive Clustering Workbench (based on eclipse) that has built-in integration with Solr. If you have a Solr service running, it is a just a matter of point the workbench to it. The clustering results and visualization are amazing. (http://project.carrot2.org/download.html). A new world opens up for me ... Thanks for pointing out how cool this is! Hint for other newcomers: Open the View Menu to configure the details of how you perform your search, e.g. your Solr URL in case it differs from the default, or your summary field, which is what gets used to analyze the data in order to determine clusters, if I understand correctly. Michael Ludwig
Re: Getting details from delete
I guess not. u will have to make a query to get the no:of docs matching the query probably this can be an enhancement to Solr On Thu, Jun 11, 2009 at 8:16 PM, Jacob Elderjel...@locamoda.com wrote: Hello, Is there any way to get the number of deleted records from a delete request? I'm sending: deletequerytype_i:(2 OR 3) AND creation_time_rl:[0 TO 124426080]/query/delete And getting: ?xml version=1.0 encoding=UTF-8? response lst name=responseHeaderint name=status0/intint name=QTime2/int/lst /response This is Solr 1.3. -- Jacob Elder -- - Noble Paul | Principal Engineer| AOL | http://aol.com
Re: change data dir location
that is fine. you can use any variable in solrconfig.xml
as dataDir${the.solr.data.dir}/dataDir}
set the value of the.solr.data.dir
as
-Dthe.solr.data.dir=%SOME_WINDOWS_PROPERTY%
On Fri, Jun 12, 2009 at 11:16 AM, Ashish Pashish.ping...@gmail.com wrote:
I don't want to set in solrConfig.xml. I want solr to take from my config
file or from system property.
Thanks,
Ashish
Noble Paul നോബിള് नोब्ळ्-2 wrote:
set the dataDir value in solrconfig.xml to what you like
On Fri, Jun 12, 2009 at 10:38 AM, Ashish Pashish.ping...@gmail.com
wrote:
I want to change data dir location and not use solrConfig.xml.
So on windows I removed dataDir xonfiguration from solrConfig.xml and
tried
setting 'solr.data.dir' system property to my location.
But it looks like solr is not picking up this property on windows.
Anybody experienced the same???
Please share...
Thanks,
Ashish
--
View this message in context:
http://www.nabble.com/change-data-dir-location-tp23992946p23992946.html
Sent from the Solr - User mailing list archive at Nabble.com.
--
-
Noble Paul | Principal Engineer| AOL | http://aol.com
--
View this message in context:
http://www.nabble.com/change-data-dir-location-tp23992946p23993210.html
Sent from the Solr - User mailing list archive at Nabble.com.
--
-
Noble Paul | Principal Engineer| AOL | http://aol.com
