It is indeed possible. Just need to use a different syntax. As far as I know,
the facet parameters need to be local parameters, like this...
facet.range={!key=date_decade facet.range.start=1600-01-01T00:00:00Z
facet.range.end=2000-01-01T00:00:00Z
I have the following problem: I have many (let's say hundreds of millions) of
documents in an existing distributed index that have a field with a variety of
values. Two of these values are dog and puppy. I have decided that I want
to reclassify these to just all be dog.
I do queries on this
This should do what you want:
String fq = Field1 + \ +
org.apache.solr.client.solrj.util.ClientUtils.escapeQueryChars(value) + \;
-Michael
-Original Message-
From: tedsolr [mailto:tsm...@sciquest.com]
Sent: Monday, October 06, 2014 10:49 AM
To: solr-user@lucene.apache.org
Subject: Re:
It could be due to the minimum timer resolution on Windows. Do a search for
windows 15ms and you'll find a lot of information about it. Though, I'm not
sure which versions of Windows and/or Java have that problem. You could test it
out by timing things other than Solr and see if they also take
When you call addFacetField, the parameter you pass it should just be the
fieldName. The fieldValue shouldn't come into play at all (unless I'm
misunderstanding what you're trying to do).
If you ever do need to escape a value for a query, you can use
Done. https://issues.apache.org/jira/browse/SOLR-6360
-Michael
-Original Message-
From: Chris Hostetter [mailto:hossman_luc...@fucit.org]
Sent: Wednesday, August 06, 2014 7:55 PM
To: solr-user@lucene.apache.org
Subject: Re: Content-Charset header in HttpSolrServer
: I was reviewing
Are the latLong_0_coordinate and latLong_1_coordinate fields populated using
copyField? If so, this sounds like it could be
https://issues.apache.org/jira/browse/SOLR-3502.
-Michael
-Original Message-
From: Franco Giacosa [mailto:fgiac...@gmail.com]
Sent: Monday, August 04, 2014 9:05
I was reviewing the httpclient code in HttpSolrServer and noticed that it sets
a Content-Charset header. As far as I know this is not a real header and is
not necessary. Anyone know a reason for this to be there? I'm guessing this was
just a mistake when converting from httpclient3 to
On Tue, Jul 22, 2014 at 6:50 AM, Michael Ryan mr...@moreover.com wrote:
Is it possible to use DocValues on an existing index without first
re-indexing?
-Michael
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
http://www.griddynamics.com
mkhlud...@griddynamics.com
Is it possible to use DocValues on an existing index without first re-indexing?
-Michael
I suggest doing this in two queries. In the first query, retrieve the unique
ids of the top 50 documents. In the second query, just query for those ids
(e.g., q=ids:(2 13 55 62 81)), and add the facet parameters on that query.
-Michael
-Original Message-
From: Aaron Gibbons
any questions.
LUCENE-5470 and LUCENE-5504 would move multiterm analysis farther down and make
it available to all parsers that use QueryParserBase, including the
ComplexPhraseQueryParser.
Best,
Tim
-Original Message-
From: Michael Ryan [mailto:mr...@moreover.com]
Sent: Sunday
In LUCENE-5472, Lucene was changed to throw an error if a term is too long,
rather than just logging a message. I have fields with terms that are too long,
but I don't care - I just want to ignore them and move on.
The recommended solution in the docs is to use LengthFilterFactory, but this
script
update processor.
Can you tell us more about the nature of your data? I mean, sometimes analyzer
filters strip or fold accented characters anyway, so count of characters versus
UTF-8 bytes may be a non-problem.
-- Jack Krupansky
-Original Message-
From: Michael Ryan
Sent
I've been using a modified version of the complex phrase query parser patch
from https://issues.apache.org/jira/browse/SOLR-1604 in Solr 3.6, and I'm
currently upgrading to 4.9, which has this built-in.
I'm having trouble with using accents in wildcard queries, support for which
was added in
Well for CEST, which is 2 hours ahead, I would think you could just do...
datefield:[* TO NOW/MONTH-2HOURS]
That would give you everything up to 2014-04-30 22:00:00 GMT, which is
2014-05-01 00:00:00 CEST.
Always always always store the correct value.
-Michael
-Original Message-
From:
Is there any significant difference in query speed when retrieving the score
pseudo-field? E.g., does...
q=foosort=date+descfl=*,score
...take longer to run than...
q=foosort=date+descfl=*
I know there's different code paths in Solr depending on whether the score is
needed or not, but not
Unfortunately the timeAllowed parameter doesn't apply to the part of the
processing that makes wildcard queries so slow. It only applies to a later part
of the processing when the matching documents are being collected. There's some
discussion in the original ticket that implemented this
My gut instinct is that your heap size is way too high. Try decreasing it to
like 5-10G. I know you say it uses more than that, but that just seems bizarre
unless you're doing something like faceting and/or sorting on every field.
-Michael
-Original Message-
From: Patrick O'Lone
Sounds like https://issues.apache.org/jira/browse/LUCENE-3821 (issue seems to
be fixed but still shows as open).
-Michael
-Original Message-
From: Arcadius Ahouansou [mailto:arcad...@menelic.com]
Sent: Sunday, September 22, 2013 11:15 PM
To: solr-user
Subject: Interesting edismax/qs
This is a known bug in that JDK version. Upgrade to a newer version of JDK 7
(any build within the last two years or so should be fine). If that's not
possible for you, you can add -XX:-UseLoopPredicate as a command line option to
java to work around this.
-Michael
-Original Message-
However, the Solr instance we direct our client query to is consuming
significantly more RAM (10GB) and is still failing after a few queries when
it runs out of heap space. This is presumably due to the role it plays,
aggregating the results from each shard.
That seems quite odd... What
This is interesting... How are you measuring the heap size?
-Michael
-Original Message-
From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de]
Sent: Monday, July 29, 2013 5:34 AM
To: solr-user@lucene.apache.org
Subject: swap and GC
Something interesting I have noticed today, after
I'm 99% sure that the deleted docs will indeed use up space in the field cache,
at least until the segments that those documents are in are merged - that is
what an optimize will do. Of course, these segments will automatically be
merged eventually, but it might take days for this to happen,
I have some custom code that uses the top-level FieldCache (e.g.,
FieldCache.DEFAULT.getLongs(reader, foobar, false)). I'd like to redesign
this to use the per-segment FieldCaches so that re-opening a Searcher is
fast(er). In most cases, I've got a docId and I want to get the value for a
To enforce uniqueness, Solr needs to be able to search on the id to see if it
is currently in the index.
-Michael
-Original Message-
From: Mysurf Mail [mailto:stammail...@gmail.com]
Sent: Monday, June 24, 2013 11:52 AM
To: solr-user@lucene.apache.org
Subject: why does the uniqueKey has
Restarting Solr won't clear the disk cache. When I'm doing perf testing, I'll
sometimes run this on the server before each test to clear out the disk cache:
echo 1 /proc/sys/vm/drop_caches
-Michael
-Original Message-
From: Learner [mailto:bbar...@gmail.com]
Sent: Friday, June 21,
Sounds like this could be https://issues.apache.org/jira/browse/SOLR-2976.
-Michael
-Original Message-
From: vinothkumar raman [mailto:vinothkr.k...@gmail.com]
Sent: Monday, April 22, 2013 5:54 AM
To: solr-user@lucene.apache.org; solr-...@lucene.apache.org
Subject: Stats facet on
I've investigated this in the past. The worst case is 2*indexSize additional
disk space (3*indexSize total) during an optimize.
In our system, we use LogByteSizeMergePolicy, and used to have a mergeFactor of
10. We would see the worst case happen when there were exactly 20 segments (or
some
Large facet.limit values cause a very large amount of form data to be sent to
the shards, though I'm not sure why this would cause a NullPointerException.
Perhaps the web server you are using is truncating the data instead of
returning a form too large error, which is somehow causing an NPE.
Yes, this is a distributed search thing. In a distributed search, it will first
make a somewhat normal facet request to all of the shards, get back the facet
values, then make a second request in order to get the full counts of the facet
values - this second request contains a list of facet
What are the values of the start and rows parameters you are using? When you
say the controller shard takes a long time, how long is it taking - 100ms, 1s,
10s...?
-Michael
-Original Message-
From: qungg [mailto:qzheng1...@gmail.com]
Sent: Tuesday, March 26, 2013 11:17 AM
To:
Depending on your use case and the particulars of your system, a previous post
I made about using a FieldCache in SolrIndexSearcher for id retrieval (see
http://osdir.com/ml/solr-user.lucene.apache.org/2013-01/msg01574.html) may help
you. In your case, it might not be the merging process on the
I was wondering if anyone is aware of an existing Jira for this bug...
_query_:\a b\~2
...is parsed as...
PhraseQuery(someField:a b)
...instead of the expected...
PhraseQuery(someField:a b~2)
_query_:\a b\~2
...is parsed as...
PhraseQuery(someField:a b~2)
_query_:\a b\~2~3
...is parsed as...
I don't have anything to add besides saying this is awesome. Great analysis.
-Michael
I think the order needs to be in lowercase. Try asc instead of ASC.
-Michael
-Original Message-
From: PeterKerk [mailto:vettepa...@hotmail.com]
Sent: Wednesday, February 13, 2013 7:30 PM
To: solr-user@lucene.apache.org
Subject: Can't determine Sort Order: 'prijs ASC', pos=5
On this
Assuming that createdDate is a DateField in your schema.xml, the object
returned by SolrJ will be a Date object (though you will need to cast it to a
Date).
-Michael
I'm pretty sure the local params have to be at the very start of the query. But
you should be able to do this with nested queries. Try this...
fq=_query_:{!tag=d0feea8}category:\5\ OR otherField:\otherValue\ AND
type:DOCUMENT
-Michael
-Original Message-
From: Karol Sikora
From a performance point of view, I can't imagine it mattering. In our setup,
we have a dedicated Solr server that is not a shard that takes incoming
requests (we call it the coordinator). This server is very lightweight and
practically has no load at all.
My gut feeling is that having a
Following up from a post I made back in 2011...
I am a user of Solr 3.2 and I make use of the distributed search capabilities
of Solr using
a fairly simple architecture of a coordinator + some shards.
Correct me if I am wrong: In a standard distributed search with
QueryComponent, the
Are you able to see any evidence that some of the 500k docs are being added
twice? Check the maxDocs on the Solr admin page. I vaguely recall there being
some issue with docs in SolrCloud being added multiple times (which under the
covers is really add, delete, add). I think that could cause
(This is based on my knowledge of 3.6 - not sure if this has changed in 4.0)
You are using rows=3, which requires retrieving 3 documents from disk.
In a non-distributed search, the QTime will not include the time it takes to
retrieve these documents, but in a distributed search, it
If you have the same documents -- with the same uniqueKey -- across multiple
shards, the count will not be what you expect. You'll need to ensure that each
document exists only on a single shard.
-Michael
-Original Message-
From: Jean-Sebastien Vachon
I'd guess that the patch simply doesn't implement it for distributed searches.
The code for distributed facets is quite a bit more complicated, and I don't
see it touched in this patch.
-Michael
-Original Message-
From: jmozah [mailto:jmo...@gmail.com]
Sent: Tuesday, January 08, 2013
From my own experience, the timestamp seems to be logged at the start of the
garbage collection.
-Michael
We see these EofExceptions in our system occasionally. I believe they occur
when our SolrJ client times out and closes the connection, before Jetty returns
the response.
-Michael
-Original Message-
From: Shawn Heisey [mailto:s...@elyograg.org]
Sent: Thursday, January 03, 2013 10:07 AM
Perhaps if there are a lot more ties on one end vs the other?
Or of the values being sorted on aren't that random? Do they naturally
increase like a timestamp?
It's a unique id field. The id is a simple sequential id, so docs with a lower
doc id will naturally also have a lower id.
I think
Has anyone ever attempted to highlight a field that is not stored in Solr? We
have been considering not storing fields in Solr, but still would like to use
Solr's built-in highlighting. On first glance, it looks like it would be
fairly simply to modify DefaultSolrHighlighter to get the stored
We have a longstanding issue with failed to respond errors in Solr when our
coordinator is querying our Solr shards.
To elaborate further... we're using the built-in distributed capabilities of
Solr 3.6, and using Jetty as our server. Occasionally, we will have a query
fail due to an error
As you add nodes, the average response time of the slowest node will likely
increase. For example, consider an extreme case where you have something like 1
million nodes - you're practically guaranteed that one of them is going to be
doing something like a stop-the-world garbage collection. So
When I upgraded from 3.2 to 3.6, I found that an optimize - all other variables
being the same - took about twice as long. Eventually I was able to track this
down to the new default of MMapDirectory. By changing back to NIOFSDirectory, I
was able to get the optimize time back down to what it
Yeah, the situation is kind of a pain right now. In
https://issues.apache.org/jira/browse/SOLR-2438, it was enabled by default and
there is no way to disable without patching SolrQueryParser. There's also the
edismax parser which doesn't have a setting for this, which I've made a jira
for at
I'd recommend not optimizing every hour. Are you seeing a significant
performance increase from optimizing this frequently?
-Michael
This should do it:
facet=truefacet.query=yourDateField:([* TO
NOW/DAY-1MILLI])facet.query=yourDateField:([NOW/DAY TO *])
-Michael
-Original Message-
From: Paul [mailto:p...@nines.org]
Sent: Thursday, October 18, 2012 5:28 PM
To: solr-user@lucene.apache.org
Subject: facet by in the past
Easiest way I know of without parsing any of the index files is to take the
size of the fdx file in bytes and divide by 8. This will give you the exact
number of documents before 4.0, and a close approximation in 4.0.
Though, the fdx file might not be on disk if you haven't committed.
-Michael
We have a maven project to build a war containing everything from the Solr war,
plus some of our own code. Here's the relevant stuff from our pom.xml:
packagingwar/packaging
dependencies
dependency
groupIdorg.apache.solr/groupId
A few questions...
1) Do you only see these spikes when running JMeter? I.e., do you ever see a
spike when you manually run a query?
2) How are you measuring the response time? In my experience there are three
different ways to measure query speed. Usually all of them will be
approximately
Try changing the tokenizer2 SynonymFilterFactory filter to this:
filter class=solr.SynonymFilterFactory synonyms=synonyms.txt
ignoreCase=false expand=true
tokenizerFactory=solr.KeywordTokenizerFactory/
By default, it seems that it uses WhitespaceTokenizer.
-Michael
I'd guess that this is because SnowballPorterFilterFactory does not implement
MultiTermAwareComponent. Not sure, though.
-Michael
In case anyone tries to do this... If you facet on a TrieField and change the
precisionStep to 0, you'll need to re-index. Changing precisionStep to 0
changes the prefix returned by TrieField.getMainValuePrefix(FieldType), which
then causes facets with a value of 0 to be returned.
-Michael
Is it safe to change the precisionStep for a TrieField without doing a re-index?
Specifically, I want to change a field from this:
fieldType name=tlong class=solr.TrieLongField precisionStep=8
omitNorms=true positionIncrementGap=0/
to this:
fieldType name=long class=solr.TrieLongField
Not really - it changes what tokens are indexed for them numbers and
range queries won't work correctly.
Sorting (FieldCache), function queries, etc, would still work, and
exact match queries would still work.
Thanks. So it is just range queries that won't work correctly? That's okay for
my
It looks like the first format was removed in 3.6 as part of
https://issues.apache.org/jira/browse/SOLR-1052. The second format works in all
3.x versions.
-Michael
-Original Message-
From: Peter Wolanin [mailto:peter.wola...@acquia.com]
Sent: Friday, April 13, 2012 12:32 PM
To:
I'm curious, why can't you do a master/slave setup?
It's just not all that useful for this particular application. Indexing new
docs and merging segments - which as I understand is the main strength of
having a write-only master - is a relatively small part of our app. What really
is
Unless you have warming happening, there should
only be a single searcher open at any given time.
So it seems to me that maxWarmingSearchers
should give you what you need.
What I'm seeing is that if a query takes a very long time to run, and runs
across the duration of multiple commits (I
Is there a way to limit the number of searchers that can be open at a given
time? I know there is a maxWarmingSearchers configuration that limits the
number of warming searchers, but that's not quite what I'm looking for...
Ideally, when I commit, I want there to only be one searcher open
https://issues.apache.org/jira/browse/SOLR-2548 may be of interest to you.
-Michael
This should be fine. From my experience, changing a field from stored=false
to stored=true and vice versa is generally safe to do and has no unexpected
behavior.
-Michael
Try putting the HTMLStripCharFilterFactory before the StandardTokenizerFactory
instead of after it. I vaguely recall being burned by something like this
before.
-Michael
Can this be the reason why it is working automatically although there are no
reversed tokens being stored and even without the
ReversedWildcardFilterFactory being set, solr automatically is allowing
leading wild card search?
Yes, that's correct. See
How about having a single-valued field named firstDestination that has the
first destination in the list, and then your query could be something like
'destination:Buenos Aires firstDestination:Buenos Aires'. Docs that match
both should have a higher score and thus will be listed first.
I was wondering... how does the TrieField precisionStep value affect the speed
of non-range queries and sorting?
I'm assuming that int (precisionStep=0) is no slower than tint
(precisionStep=8) for these - is that correct? tint is just faster for range
queries?
Is int any faster than tint
I had a similar requirement in my project, where a user might ask for up to
3000 results. What I did was change SolrIndexSearcher.doc(int, Set) to retrieve
the unique key from the field cache instead of retrieving it as a stored field
from disk. This resulted in a massive speed improvement for
According to http://lucene.apache.org/java/3_4_0/fileformats.html, the
FNMVersion changed from -2 to -3 in Lucene 3.4. Is it possible that the new
master is actually running 3.4, and the new slave is running 3.2? (This is just
a wild guess.)
-Michael
The problem I have is that at search time, I have faceting turned on for
this field and therefore, I get the four facets canadian, imperial,
bank, and commerce, which all refer to the same record.
How can I go about searching for any word contained in the company name but
then return the
I have some fields I facet on that are TextFields but have just a single token.
The fieldType looks like this:
fieldType name=myStringFieldType class=solr.TextField indexed=true
stored=false omitNorms=true sortMissingLast=true
positionIncrementGap=100
analyzer
tokenizer
Another thing to note is that QTime does not include the time it takes to
retrieve the stored documents to include in the response. So if you're using a
high rows value in your query, QTime may be much smaller than the actual time
Solr spends generating the response.
Try adding rows=1 to your
I am trying to highlight FieldA when a user searches on either FieldA or FieldB,
but I do not want to highlight FieldA when a user searches on FieldC.
To explain further: I have a field named content and a field named
contentCS. The content field is a stored text field that uses
I was wondering if anyone has any ideas for making UnInvertedField.uninvert()
faster, or other alternatives for generating facets quickly.
The vast majority of the CPU time for our Solr instances is spent generating
UnInvertedFields after each commit. Here's an example of one of our slower
I think the problem is that the mergePolicy config needs to be inside of the
indexDefaults config, rather than after it as your have.
-Michael
Has anyone used the Flexible Query Parser
(https://issues.apache.org/jira/browse/LUCENE-1567) in Solr? I'm just starting
to look at it for the first time and was wondering if it is something that can
be dropped into Solr fairly easily, or if more extensive changes are needed. I
thought
Are you using facet.mincount in the query?
-Michael
yep - facet.mincount=1
Yeah, I've ran into this same issue, though I never looked too closely into it.
What is happening is that the facet.mincount parameter is removed when the
query is made to the shards, so each shard is returning about 3 facet
values, most of them with a count of 0. I
I have recently upgraded from Solr 1.4 to Solr 3.2. In Solr 1.4 only 3
files (one .cfs two segments) file were made in *index/* directory.
(after
doing optimize).
Now, in Solr 3.2, the optimize seems not be working. My final number of
files in *index/* directory are in 7-8 in number. Can
10,000,000 document index
Internal Document id is 32 bit unsigned int
Max Memory Used by a single cache slot in the filter cache = 32 bits x
10,000,000 docs = 320,000,000 bits or 38 MB
I think it depends on where exactly the result set was generated. I believe the
result set will usually be
I'm using Solr 3.2 with a mergeFactor of 10 and no merge policy configured,
thus using the default LogByteSizeMergePolicy. Before I do an optimize,
typically the largest segment will be about 90% of the total index size.
When I do an optimize, the total disk space required is usually about 2x
One simple way of doing this is maybe to write a wrapper for TermQuery
that only returns docs with a Term Frequency X as far as I
understand the question those terms don't have to be within a certain
window right?
Correct. Terms can be anywhere in the document. I figured term frequencies
Is there a way to specify in a query that a term must match at least X times in
a document, where X is some value greater than 1?
For example, I want to only get documents that contain the word dog three
times. I've thought that using a proximity query with an arbitrary large
distance value
Are you using the same analyzer for both type=query and type=index? Can you
show us the fieldType from your schema?
-Michael
Nope. The 'text' field will just have the 'titulo' contents. To have both, you
would have to do something like this:
copyField source=title dest=titulo/
copyField source=title dest=text/
copyField source=titulo dest=text/
-Michael
I think this is because ) is treated as a token delimiter. So (foo)bar is
treated the same as (foo) bar (that is, bar is treated as a separate word).
So (foo)* is really parsed as (foo) * and thus the * is treated as the
start of a new word.
-Michael
You should be fine - no need to re-index your data.
Adding and removing fields is generally safe to do without a re-index. Changing
a field (its type, analyzers, etc) requires more caution and generally does
require a re-index.
-Michael
SolrDocumentList docs = queryResponse.getResults();
long totalMatches = docs.getNumFound();
-Michael
You could try adding a new int field (like typeSort) that has the desired
sort values. So when adding a document with type:car, also add typeSort:1; when
adding type:van, also add typeSort:2; etc. Then you could do sort=typeSort
asc to get them in your desired order.
I think this is also
I am a user of Solr 3.2 and I make use of the distributed search capabilities
of Solr using a fairly simple architecture of a coordinator + some shards.
Correct me if I am wrong: In a standard distributed search with
QueryComponent, the first query sent to the shards asks for fl=myUniqueKey or
Is it possible to use omitTermFreqAndPositions=true in a fieldType
declaration that uses class=solr.TextField? I've tried doing this and it does
not seem to work (i.e., the prx file size does not change). Using it in a
field declaration does work, but I'd rather set it in the fieldType so I
96 matches
Mail list logo
