For a similar idea, check:
https://issues.apache.org/jira/browse/SOLR-906
This opens a single stream and writes all documents to that. It could
easily be extended to have multiple threads draining the same Queue
On Dec 9, 2008, at 4:02 AM, Noble Paul നോബിള്
नोब्ळ् wrote:
I guess this
I guess this is the best idea . Let us have a new BatchHttpSolrServer
which can help achieve this
--Noble
On Thu, Dec 4, 2008 at 7:14 PM, Yonik Seeley [EMAIL PROTECTED] wrote:
On Thu, Dec 4, 2008 at 8:39 AM, Mark Miller [EMAIL PROTECTED] wrote:
Kick off some indexing more than once - eg, post a
:
: Not sure how that would work (unless you didn't want responses), but
: I've thought about it from the SolrJ side - something you could
: quickly add documents to and it would manage a number of threads under
: the covers to maximize throughput. Not sure what would be the best
: for error
be parallelized and your
hardware supports it, it can certainly make a big difference... it
did in my case. Both CPUs were cooking during my parallel indexing
runs.
Erik
--
View this message in context:
http://www.nabble.com/solr-performance-tp9055437p20833421.html
Sent
Kick off some indexing more than once - eg, post a folder of docs, and
while thats working, post another.
I've been thinking about a multi threaded UpdateProcessor as well - that
could be interesting.
- Mark
sunnyfr wrote:
Hi,
I was reading this post and I wondering how can I parallelize
CPUs were cooking during my parallel indexing
runs.
Erik
--
View this message in context:
http://www.nabble.com/solr-performance-tp9055437p20833662.html
Sent from the Solr - User mailing list archive at Nabble.com.
On Thu, Dec 4, 2008 at 8:39 AM, Mark Miller [EMAIL PROTECTED] wrote:
Kick off some indexing more than once - eg, post a folder of docs, and while
thats working, post another.
I've been thinking about a multi threaded UpdateProcessor as well - that
could be interesting.
Not sure how that
Yonik Seeley wrote:
Not sure what would be the best
for error handling though - perhaps just polling (allow user to ask
for failed or successful operations).
Thats how I've handled similar situations in the past. Your submitting a
batch of data to be processed, and if your so inclined to
: Mike Klaas [mailto:[EMAIL PROTECTED]
Sent: Monday, November 03, 2008 4:25 PM
To: solr-user@lucene.apache.org
Subject: Re: SOLR Performance
If you never execute any queries, a gig should be more than enough.
Of course, I've never played around with a .8 billion doc corpus on one
machine
Subject: SOLR Performance
We've moved past this issue by reducing date precision -- thanks to
all for the help. Now we're at another problem.
There is relatively constant updating of the index -- new log entries
are pumped in from several applications continuously. Obviously, new
entries do
configuration
comes in handy. Commits to the Master don't slow down queries on the
Slave.
-Todd
-Original Message-
From: Alok Dhir [mailto:[EMAIL PROTECTED]
Sent: Monday, November 03, 2008 1:47 PM
To: solr-user@lucene.apache.org
Subject: SOLR Performance
We've moved past this issue by reducing
Message-
From: Alok Dhir [mailto:[EMAIL PROTECTED]
Sent: Monday, November 03, 2008 1:47 PM
To: solr-user@lucene.apache.org
Subject: SOLR Performance
We've moved past this issue by reducing date precision -- thanks to
all for the help. Now we're at another problem.
There is relatively
-user@lucene.apache.org
Subject: SOLR Performance
We've moved past this issue by reducing date precision -- thanks to
all for the help. Now we're at another problem.
There is relatively constant updating of the index -- new log
entries
are pumped in from several applications continuously
Sent: Monday, November 3, 2008 5:16:27 PM
Subject: Re: SOLR Performance
in terms of RAM -- how to size that on the indexer?
---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
[EMAIL PROTECTED]
On Nov 3, 2008, at 4:07 PM, Walter Underwood wrote
]
Sent: Monday, November 03, 2008 4:25 PM
To: solr-user@lucene.apache.org
Subject: Re: SOLR Performance
If you never execute any queries, a gig should be more than enough.
Of course, I've never played around with a .8 billion doc corpus on one
machine.
-Mike
On 3-Nov-08, at 2:16 PM, Alok Dhir
Maybe i missed it, but skimming this thread i haven't seen any indication
of how your configured the various caches in solrconfig.xml ... or any
indication of what kinds of cache hit/miss/expullsion stats you see from
stats.jsp after running any tests.
considering you're doing faceting on
: considering you're doing faceting on quite a few fields, the filterCache
: is somewhat important.
Sorry ... i overlooked the bit where QueryComponent was taking 6.x seconds
... in general knowing what the cache hit rates are looking like is
crucial to understanding the performance, but as
searching but might reduce relevancy.
- Keep the index as lean as possible
thanks,
Rajiv
--
View this message in context:
http://www.nabble.com/Need-help-with-Solr-Performance-tp19881808p19881808.html
Sent from the Solr - User mailing list archive at Nabble.com.
I don't think you can search a 15 million doc index with any kind of
query complexity beyond a low freq query term in under .25 seconds
unless its a cached hit (in which case it still might not *quite* make
it under .25 every time either I'd think). Would love to be proven wrong
though g You
?
thanks,
Rajiv
--
View this message in context:
http://www.nabble.com/Need-help-with-Solr-Performance-tp19881808p19881808.html
Sent from the Solr - User mailing list archive at Nabble.com.
--
Grant Ingersoll
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java
- I've setup autowarming queries
- Haven't warmed sort fields because I'm not doing any sorting
- Not using any solid state drives
- Using filters instead of queries for filtering.
thanks,
Rajiv
--
View this message in context:
http://www.nabble.com/Need-help-with-Solr-Performance
.
thanks,
Rajiv
--
View this message in context:
http://www.nabble.com/Need-help-with-Solr-Performance-tp19881808p19881808.htm
l
Sent from the Solr - User mailing list archive at Nabble.com.
--
Grant Ingersoll
Lucene Helpful Hints:
http://wiki.apache.org/lucene
of queries for filtering.
thanks,
Rajiv
--
View this message in context:
http://www.nabble.com/Need-help-with-Solr-Performance-tp19881808p19881808.html
Sent from the Solr - User mailing list archive at Nabble.com.
--
Grant Ingersoll
Lucene Helpful Hints:
http
On Oct 8, 2008, at 3:30 PM, Rajiv2 wrote:
what is your actual query?
Are you doing faceting / highlighting / or anything else?
I am doing faceting on 5 fields, no highlighting or anything else,
debugging
is also off. A basic query that I'm doing using dismax is 'cleaning
services' over
faster.
--
View this message in context:
http://www.nabble.com/Need-help-with-Solr-Performance-tp19881808p19886504.html
Sent from the Solr - User mailing list archive at Nabble.com.
://www.nabble.com/Need-help-with-Solr-Performance-tp19881808p19886504.html
Sent from the Solr - User mailing list archive at Nabble.com.
=org.apache.solr.handler.component.HighlightComponent
double name=time0.0/double
/lst
-
lst name=org.apache.solr.handler.component.DebugComponent
double name=time270.0/double
/lst
/lst
/lst
/lst
--
View this message in context:
http://www.nabble.com/Need-help-with-Solr-Performance-tp19881808p19888155.html
Sent
-
lst name=process
double name=time6727.0/double
-
lst name=org.apache.solr.handler.component.QueryComponent
double name=time6457.0/double
/lst
-
lst name=org.apache.solr.handler.component.FacetComponent
double name=time0.0/double
/lst
-
So I take it, this is with faceting turned off...
what
--
View this message in context:
http://www.nabble.com/Need-help-with-Solr-Performance-tp19881808p19888604.html
Sent from the Solr - User mailing list archive at Nabble.com.
On Oct 8, 2008, at 6:11 PM, Rajiv2 wrote:
w/ faceting qtime is around +200ms.
if your target time is 250, this will need some work... but lets
ignore that for now...
qtime for a standard query on the default search field is less than
100ms.
Usually around 60ms.
qtime for id:
Hi,
We want to update the index based on TIB listener, whenever database changes
happens we want to update my index instantly this may happen very frequently
for number of records.
Could anyone please tell me how would be the performance for these scenarios?
Question related linguistic
Hi,
- Original Message
From: mahendra mahendra [EMAIL PROTECTED]
To: solr-user@lucene.apache.org
Sent: Friday, September 26, 2008 3:52:57 PM
Subject: Solr performance for Instance updates
Hi,
We want to update the index based on TIB listener, whenever database changes
to
increase, the old doc is physically not going to delete.
Any Idea on this..
Thanks Regards,
Mahendra
--- On Sat, 9/27/08, Otis Gospodnetic [EMAIL PROTECTED] wrote:
From: Otis Gospodnetic [EMAIL PROTECTED]
Subject: Re: Solr performance for Instance updates
To: solr-user@lucene.apache.org
Date
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
From: mahendra mahendra [EMAIL PROTECTED]
To: solr-user@lucene.apache.org
Sent: Friday, September 26, 2008 6:30:27 PM
Subject: Re: Solr performance for Instance updates
Hi,
Instantly I want to update
On Jun 19, 2008, at 6:28 PM, Yonik Seeley wrote:
2. I use acts_as_solr and by default they only make post
requests, even
for /select. With that setup the response time for most queries,
simple or
complex ones, were ranging from 150ms to 600ms, with an average of
250ms. I
changed the
On Fri, Jun 20, 2008 at 8:32 AM, Erik Hatcher [EMAIL PROTECTED]
wrote:
On Jun 19, 2008, at 6:28 PM, Yonik Seeley wrote:
2. I use acts_as_solr and by default they only make post requests, even
for /select. With that setup the response time for most queries, simple
or
complex ones, were
Hi,
I've been using solr for a little without worrying too much about how it
works but now it's becoming a bottleneck in my application. I have a couple
issues with it:
1. My index always gets slower and slower when commiting/optimizing for some
obscure reason. It goes from 1 second with a new
://www.nabble.com/Improve-Solr-Performance-tp17664065p17719927.html
Sent from the Solr - User mailing list archive at Nabble.com.
Some of these cache values are too large and will drastically slow
some things down (like commiting new changes to the index) or may
cause you to run out of memory over time. I would revert the cache
params back to what they were in the example solrconfig.xml
Then focus on requirements: are your
hello every body
I want to imporve solr performance at time search, so i have read about
SolrPerformanceFactor so I tried to increase the size of my caches but the
QTime hasn't changed a lot. this is how I am quering I ask to solr to do a
search and to return me 200 result the first time so I
solr performance at time search, so i have read about
SolrPerformanceFactor so I tried to increase the size of my caches but the
QTime hasn't changed a lot. this is how I am quering I ask to solr to do a
search and to return me 200 result the first time so I display them then I
ask for the next
I am just wondering, because having 200 fields seems like too much (for
me),
I want to know if people actually have such kind of schemas and how well
they perform.
On Thu, Apr 24, 2008 at 5:10 PM, Grant Ingersoll [EMAIL PROTECTED]
wrote:
Are you actually seeing performance problems or just
That is well within the boundaries of what Solr/Lucene can handle.
But, of course, it depends on what you're doing with those fields
too. Putting 200 fields into a dismax qf specification, for example,
would surely be bad for performance :) But querying on only a
handful of fields or
Message
From: Erik Hatcher [EMAIL PROTECTED]
To: solr-user@lucene.apache.org
Sent: Friday, April 25, 2008 8:23:27 AM
Subject: Re: solr performance for documents with hundreds of fields
That is well within the boundaries of what Solr/Lucene can handle.
But, of course, it depends on what
Hi,
I wanted to know what would be the performance of SOLR for the following
scenario:
the documents contain say 200 fields with
say 100 of the fields (containing numbers)
and rest containing short strings of 40-50 character length.
the sparseness of the data can be assumed to be as
Are you actually seeing performance problems or just wondering if
there will be a performance problem?
-Grant
On Apr 24, 2008, at 7:08 AM, Umar Shah wrote:
Hi,
I wanted to know what would be the performance of SOLR for the
following
scenario:
the documents contain say 200 fields with
do you have a stack trace around the Lucene clone() stuff?
-Grant
On Feb 7, 2008, at 9:56 PM, Fuad Efendi wrote:
Question:
Why constant updates slow down SOLR performance even if I am not
executing
Commit? I just noticed this... Thead dump shows something Lucene ...
Clone
the Lucene clone() stuff?
-Grant
On Feb 7, 2008, at 9:56 PM, Fuad Efendi wrote:
Question:
Why constant updates slow down SOLR performance even if I am not
executing
Commit? I just noticed this... Thead dump shows something
Lucene ...
Clone(), and significant CPU usage. I did
Question:
Why constant updates slow down SOLR performance even if I am not executing
Commit? I just noticed this... Thead dump shows something Lucene ...
Clone(), and significant CPU usage. I did about 5 mlns updates via HTTP
XML, single document at a time, without commit, and performance went
On 4/20/07, Michael Thessel [EMAIL PROTECTED] wrote:
Hey Erik,
thanks for the fast reply. Yes this could be possible. I currently got
solr running for the indexing of a forum with 100k users. It could
definitely be possible that two commits overlap. But I need to commit
all changes because the
Mike Klaas wrote:
On 4/20/07, Michael Thessel [EMAIL PROTECTED] wrote:
Hey Erik,
thanks for the fast reply. Yes this could be possible. I currently got
solr running for the indexing of a forum with 100k users. It could
definitely be possible that two commits overlap. But I need to commit
all
Michael Thessel wrote:
Mike Klaas wrote:
On 4/20/07, Michael Thessel [EMAIL PROTECTED] wrote:
Hey Erik,
thanks for the fast reply. Yes this could be possible. I currently got
solr running for the indexing of a forum with 100k users. It could
definitely be possible that two commits overlap.
Hello,
in my logs I get from time to time this message:
INFO: PERFORMANCE WARNING: Overlapping onDeckSearchers=2
What does this mean? What can I do to avoid this?
Cheers,
Michael
On Apr 19, 2007, at 7:47 PM, Michael Thessel wrote:
in my logs I get from time to time this message:
INFO: PERFORMANCE WARNING: Overlapping onDeckSearchers=2
What does this mean? What can I do to avoid this?
I think you have issued multiple commits (or optimizes) that hadn't
fully
Thanks for all who replied.
my number 1000 was per minute, not second!
I can't read! :-p
couple of times today at around 158 documents / sec.
This is not bad at all. How about search performance?
How many concurrent queries have people been having?
What does the response time look like?
On 2/21/07, Jack L [EMAIL PROTECTED] wrote:
Thanks to the others that clarified. I run my indexers in
parallel... but a single instance of Solr (which in turn handles
requests in parallel as well).
Do you feel if multi-threaded posting is helpful?
I suppose when solr does indexing, it's
On Feb 21, 2007, at 4:25 PM, Jack L wrote:
couple of times today at around 158 documents / sec.
This is not bad at all. How about search performance?
How many concurrent queries have people been having?
What does the response time look like?
I'm the only user :) What I've done is a
You could build your index using Lucene directly and then point a
Solr instance at it once its built. My suspicion is that the
overhead of forming a document as an XML string and posting to Solr
via HTTP won't be that much different than indexing with Lucene
directly.
My largest Solr
. Februar 2007 11:43
An: solr-user@lucene.apache.org
Betreff: Re: solr performance
You could build your index using Lucene directly and then point a
Solr instance at it once its built. My suspicion is that the
overhead of forming a document as an XML string and posting to Solr
via HTTP won't
-Ursprüngliche Nachricht-
Von: Erik Hatcher [mailto:[EMAIL PROTECTED]
Gesendet: Dienstag, 20. Februar 2007 11:43
An: solr-user@lucene.apache.org
Betreff: Re: solr performance
You could build your index using Lucene directly and then point a
Solr instance at it once its built. My
Thanks to all who replied. It's encouraging :)
The numbers vary quite a bit though, from 13 docs/s (Burkamp)
to 250 docs/s (Walter) to 1000 docs/s I understand the results also depend
on the doc size and hardware.
I have a question for Erik: you mentioned single threaded indexer
(below). I'm not
: The numbers vary quite a bit though, from 13 docs/s (Burkamp)
: to 250 docs/s (Walter) to 1000 docs/s I understand the results also depend
: on the doc size and hardware.
It also depends a lot on how much analysis you do of each field ... and
that doesn't even begin to get totheissue of what
Try running your submits while watching a CPU load meter.
Do this on a multi-CPU machine.
If all CPUs are busy, you are running as fast as possible.
If one CPU is busy (around 50% usage on a dual-CPU system),
parallel submits might help.
If no CPU is 100% busy, the bottleneck is probably disk
On Feb 20, 2007, at 1:46 PM, Jack L wrote:
The numbers vary quite a bit though, from 13 docs/s (Burkamp)
to 250 docs/s (Walter) to 1000 docs/s I understand the results also
depend
on the doc size and hardware.
my number 1000 was per minute, not second! however, i've done a few
runs
401 - 464 of 464 matches
Mail list logo
