Re: Connection Established but waiting for response for a long time.
Set name=ThreadPool New class=org.eclipse.jetty.util.thread.QueuedThreadPool Set name=minThreads10/Set Set name=maxThreads1/Set Set name=detailedDumpfalse/Set /New /Set Call name=addConnector Arg New class=org.eclipse.jetty.server.bio.SocketConnector Set name=hostSystemProperty name=jetty.host //Set Set name=portSystemProperty name=jetty.port default=8983//Set Set name=maxIdleTime5000/Set Set name=requestHeaderSize65536/Set Set name=lowResourceMaxIdleTime1500/Set Set name=statsOnfalse/Set /New /Arg /Call Everything is default expect for Set name=maxIdleTime5000/Set Set name=requestHeaderSize65536/Set Thanks, Qun -- View this message in context: http://lucene.472066.n3.nabble.com/Connection-Established-but-waiting-for-response-for-a-long-time-tp4088587p4088874.html Sent from the Solr - User mailing list archive at Nabble.com.
Connection Established but waiting for response for a long time.
Hi, I'm runing solr 4.0 but using legacy distributed search set up. I set the shards parameter for search, but indexing into each solr shards directly. The problem I have been experiencing is building connection with solr shards. If I run a query, by using wget, to get number of records from each individual shards (50 of them) sequentially, the request will hang at some shards (seems random). The wget log will say the connection is established but waiting for response. At that point I thought that the Solr shard might be under high load, but the strange behavior happens when I send another request to the send shard (using wget again) from another thread, the response comes back, and will trigger something in Solr to send back response for the first request I have sent before. This also happens in my daily indexing. If I send an commit, it will some times hangs. However, if I send another commit to the same shard, both commit will come back fine. I'm running Solr on stock jetty server, and sometime back my boss told me to set the maxIdleTime to 5000 for indexing purpose. I'm not sure if this have anything to do with the strange behavior that I'm seeing right now. Please help me resolve this issue. Thanks, Qun -- View this message in context: http://lucene.472066.n3.nabble.com/Connection-Established-but-waiting-for-response-for-a-long-time-tp4088587.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: ConcurrentUpdateSolrServer hanging
Hi, BlockUntilFinish block indefinitely sometimes. But if I send a commit from another thread to the instance, the concurrentUpdateServer unblock and send the rest of the documents and commit. So the squence look like this: 1. adding documents as usual... 2. finish adding documents... 3. block untill finished... block forever (i try to block before commit, call this commit 1) 4. from other thread, send a commit (lets call this commit 2) 5. magically unblocked... and flushed out the rest of the documents... 6. commit 1... 7. commit 2 ... The order of commit in 6 and 7 is observed in solr log. Thanks, Qun -- View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrServer-hanging-tp4073620p4074366.html Sent from the Solr - User mailing list archive at Nabble.com.
ConcurrentUpdateSolrServer hanging
Hi, I'm using concurrentUpdateSolrServer to do my incremental indexing nightly. I have 50 shards to index into, about 10,000 documents each night. I start one concurrentUpdateSolrServer on each shards and start to send documents. The queue size for concurrentUpdateSolrServer is 100, and 4 threads. At the end of the import, i will send commit using the same concurrentUpdateSolrServer. The problem is some of the concurrentUpdateSolrServer is not sending the commit to the shards and the import task hangs for a couple hours. So I looked at the log and find out that the shards received about 1000 document couple hours later following with a commit. Is there anything methods I can call to flush out documents before I send the commit? Or are there any existing issue related to concurrentUpdateSolrServer related to this? Thanks, Qun -- View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrServer-hanging-tp4073620.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: ConcurrentUpdateSolrServer hanging
Hi Michael, I realized that I might have to use blockUntilFinished before commit, but do I have to use shutdown as well?? Thanks, Qun -- View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrServer-hanging-tp4073620p4073651.html Sent from the Solr - User mailing list archive at Nabble.com.
Slow performance on distributed search
Hi, I have 40 shards running on 48 core machine with 256GB RAM (The data is about 40 GB). I am using legacy distributed method as setup. So I have one additional shard with no data. Queries would go to this shard and the shard would merge result from the rest of the 40 shards. From the log, I see that query time for each shard is almost instant, but the controller shard take a long time to merge the result. I tried to cache the results to controller shards like I do in each individual shard, but it wouldn't not cache, probably because the controller shard contains no documents. I can tell that merging result is the bottle neck here, but I couldn't find a way to fix it. Please let me know if you guys have any suggestion. Thanks in advance! Best Regards, Qun -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-performance-on-distributed-search-tp4051434.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Slow performance on distributed search
Thank you for reply. Queries do need to go to all 40 shards, though 40 shards is not a final number, my setup can be changed if search time decreases. Im using 40 shards because we have a tool that can index faster with more shards. -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-performance-on-distributed-search-tp4051434p4051439.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Slow performance on distributed search
for start=100,000row=10. event though each individual shard take only 10ms to query, the merging process done by controller would take about a minutes. By looking at logs, each shard is giving the controller shard 100,010 rows of data, and because there are 40 shards in total, the controller is getting 100,010*40 rows of data, therefore merging is taking a long time. I have not tried solr cloud, does any one know the performance of query large start row on solr cloud? -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-performance-on-distributed-search-tp4051434p4051492.html Sent from the Solr - User mailing list archive at Nabble.com.
