Re: PHP/Solr library
I see the same error from my php/curl script when my request times out. I believe you can up your timeouts both in php/curl and in your solr configs. On Wed, Jan 4, 2012 at 3:15 PM, Brian Lamb wrote: > Hi Param, > > That's the method I'm switching over from. It seems that script works > inefficiently with my set up as the data is spread out over multiple > tables. I've considered creating a simple solr MySQL table just to maintain > the solr data but I wanted to try out this PHP extension first. > > But thanks for the suggestion! > > Brian Lamb > > On Wed, Jan 4, 2012 at 2:58 PM, Sethi, Parampreet < > parampreet.se...@teamaol.com> wrote: > > > Hi Brian, > > > > Not exactly solution to your problem. But it may help, you can run Solr > > directly on top of your database, if your schema is simple manipulation > of > > the database fields. This way you only need to update the database and > > solr index will be automatically updated with the latest data. I am using > > this in production and it's working pretty neatly. > > > > Here are few helpful links: > > http://wiki.apache.org/solr/DataImportHandler > > http://www.params.me/2011/03/configure-apache-solr-14-with-mysql.html > > > > -param > > > > On 1/4/12 2:50 PM, "Brian Lamb" wrote: > > > > >Hi all, > > > > > >I've been exploring http://www.php.net/manual/en/book.solr.php as a way > > to > > >maintain my index. I already have a PHP script that I use to update a > > >database so I was hoping to be able to update the database at the same > > >time > > >I am updating the index. > > > > > >However, I've been getting the following error when trying to run > > >$solr_client->commit(); > > > > > >Unsuccessful update request. Response Code 0. (null) > > > > > >I've tried looking to see why I'm getting the error but I cannot find a > > >reasonable explanation. My guess is that it is because my index is > rather > > >large (22 million records) and thus it is timing out or something like > > >that > > >but I cannot confirm that that is the case nor do I know how to fix it > > >even > > >if it were. > > > > > >Any help here would be greatly appreciated. > > > > > >Thanks, > > > > > >Brian Lamb > > > > >
Re: Painfully slow transfer speed from Solr
Thanks for the reply Shawn. The solr server currently has 8gb of ram and the total size of the dataDir is around 30gb. I start solr and give the java heap up to 4gb of ram, so that leaves 4gb for the OS, there are no other running services on the box. So from what you are saying, we are way under on the amount of ram we would ideally have. Just trying to get a better understanding of this.Wouldn't the indexes not being in the disk cache make the queries themselves slow as well (high qTime), not just fetching the results? We currently store all the fields that we index, my reasoning behind that is that debugging results we get from solr w/o being able to see what is stored in solr would be near impossible (in my head anyhow..). Generally our original source (mysql) and solr are consistent, but we've had cases where some updates have been missed for one reason or another. So my options are: reduce index sizes, increase ram on the server, increase disk speed (SSD drives)? Thanks Stephen On Mon, Nov 21, 2011 at 11:33 PM, Shawn Heisey wrote: > On 11/21/2011 8:45 PM, Stephen Powis wrote: > >> I'm running Solr 1.4.1 with Jetty. When I make requests against solr that >> have a large response (~1mb of data) I'm getting super slow transfer times >> back to the client, I'm hoping you guys can help shed some light on this >> issue for me. >> >> Some more information about my setup: >> - The qTime header in the response generally is very small, under 1 sec (< >> 1000ms). >> - The client making the request is on a 1000mb LAN with the solr server, >> yet the transfer speed is only between 16k and 30k per sec. >> - If I make the same request against localhost on the solr server, I see >> the same slow speeds. SCP and other transfer between the client and >> server >> are all quick. I'd like to think these tests eliminate any kind of >> network >> pipe problem between the two servers. >> - If I make the same query repeatedly, sometimes it will send the response >> very quickly (6mb/sec and faster) >> - While testing this, load on the box was basically at idle. >> >> So I guess I'm hoping someone can help me understand whats going on here, >> and why I'm seeing this behavior, and perhaps a possible solution? >> >> What exactly does qTime measure? I assume it is the time it takes to >> process the request and fetch the resulting rows. It obviously does not >> include the transfer time back to the client, but does it include pulling >> the data from the index? Is solr slow to pull the data from the index and >> drop it into the network pipe? >> > > Your bottleneck is probably disk I/O and a lack of OS disk cache. How big > is your index, how much RAM do you have, and how much RAM is used by > processes, especially the Java heap? QTime measures the amount of time > that Solr spent finding the document IDs. It does not include time spent > retrieving the requested fields or sending it to the client. > > Solr is designed to work best when the entire index fits into the OS disk > cache, which is free memory that is not assigned to the OS or other > processes. Limiting the number of fields that Solr indexes (for searching) > and stores (for data retrieval) keeps index size down, so you can fit more > of it in the disk cache. When the index data is in RAM, Solr is very very > fast. If it has to go out to the disk to search or retrieve, it is very > slow. > > You should only index the fields absolutely required to get good search > results, and you should only store the fields required to display a grid of > search results. When displaying full details for an individual item, go to > the the original data source using the identifier returned in the search > results. In typical search applications, you only need full details for a > small subset of the results returned by a search, so don't retrieve > megabytes of information that will never be used. > > Thanks, > Shawn > >
Painfully slow transfer speed from Solr
I'm running Solr 1.4.1 with Jetty. When I make requests against solr that have a large response (~1mb of data) I'm getting super slow transfer times back to the client, I'm hoping you guys can help shed some light on this issue for me. Some more information about my setup: - The qTime header in the response generally is very small, under 1 sec ( < 1000ms). - The client making the request is on a 1000mb LAN with the solr server, yet the transfer speed is only between 16k and 30k per sec. - If I make the same request against localhost on the solr server, I see the same slow speeds. SCP and other transfer between the client and server are all quick. I'd like to think these tests eliminate any kind of network pipe problem between the two servers. - If I make the same query repeatedly, sometimes it will send the response very quickly (6mb/sec and faster) - While testing this, load on the box was basically at idle. So I guess I'm hoping someone can help me understand whats going on here, and why I'm seeing this behavior, and perhaps a possible solution? What exactly does qTime measure? I assume it is the time it takes to process the request and fetch the resulting rows. It obviously does not include the transfer time back to the client, but does it include pulling the data from the index? Is solr slow to pull the data from the index and drop it into the network pipe? Thanks for any help! Stephen
Re: Post size limit to Solr?
Thanks! On Mon, Jan 10, 2011 at 9:27 PM, Ahmet Arslan wrote: > > Is there a max POST size limit when > > sending documents over to Solrs update > > handler to be indexed? Right now I've self imposed a > > limit of sending a max > > of 50 docs per request to solr in my PHP code..and that > > seems to work fine. > > I was just curious as to if there was a limit somewhere at > > which Solr will > > complain? > > I think this is related to servlet container. > Default maxPostSize for tamcat is 2 megabytes. > http://tomcat.apache.org/tomcat-5.5-doc/config/http.html > > > >
Post size limit to Solr?
Is there a max POST size limit when sending documents over to Solrs update handler to be indexed? Right now I've self imposed a limit of sending a max of 50 docs per request to solr in my PHP code..and that seems to work fine. I was just curious as to if there was a limit somewhere at which Solr will complain? Thanks Stephen
Re: Problem escaping question marks
Looking at the JIRA issue, looks like there's been a new patch related to this. This is good news! We've re-written a portion of our web app to use Solr instead of mysql. This part of our app allows clients to construct rules to match data within their account, and automatically apply actions to those matched data points. So far our testing and then rollout has been smooth, until we encountered the above rule/query. I guess I assumed since these metacharacters were escaped that they would be parsed correctly under any type of query. What is the likelihood of this being included in the next release/bug fix version of Solr? Are there docs available online with basic information about rolling our own build of Solr that includes this patch? I appreciate your help! Thanks! Stephen On Thu, Nov 4, 2010 at 9:26 AM, Robert Muir wrote: > On Thu, Nov 4, 2010 at 1:44 AM, Stephen Powis > wrote: > > I want to return any first name with a Question Mark in it > > Query: first_name: *\?* > > > > There is no way to escape the metacharacters * or ? for a wildcard > query (regardless of queryparser, even if you write your own). > See https://issues.apache.org/jira/browse/LUCENE-588 > > Its something we could fix, but in all honesty it seems one reason it > isn't fixed is because the bug is so old, yet there hasn't really been > any indication of demand for such a thing... >
Problem escaping question marks
I'm having difficulty properly escaping ? in my search queries. It seems as tho it matches any character. Some info, a simplified schema and query to explain the issue I'm having. I'm currently running solr1.4.1 Schema: I want to return any first name with a Question Mark in it Query: first_name: *\?* Returns all documents with any character in it. Can anyone lend a hand? Thanks! Stephen
