Re: distributed search is significantly slower than direct search
https://issues.apache.org/jira/browse/SOLR-5478 There it goes On Mon, Nov 18, 2013 at 5:44 PM, Manuel Le Normand manuel.lenorm...@gmail.com wrote: Sure, I am out of office till end of week. I reply after i upload the patch
Re: distributed search is significantly slower than direct search
Manuel, that sounds very interesting. Would you be willing to contribute this back to the community? On Mon, Nov 18, 2013 at 9:53 AM, Manuel Le Normand manuel.lenorm...@gmail.com wrote: In order to accelerate the BinaryResponseWriter.write we extended this writer class to implement the docid to id tranformation by docValues (on memory) with no need to access stored field for id reading nor lazy loading of fields that also has a cost. That should improve read rate as docValues are sequential and should avoid disk IO. This docValues implementation is accessed during both query stages (as mentioned above) in case you ask for id's only, or only once, during the distributed search stage, in case you intend asking for stored fields different than id. We just started testing it for performance. I would love hearing any oppinions or test performances for this implementation Manu -- Regards, Shalin Shekhar Mangar.
Re: distributed search is significantly slower than direct search
Hi Thanks very much for your answers :) Manuel, if you have a patch I will be glad to test it's performance Yuval On Mon, Nov 18, 2013 at 10:49 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: Manuel, that sounds very interesting. Would you be willing to contribute this back to the community? On Mon, Nov 18, 2013 at 9:53 AM, Manuel Le Normand manuel.lenorm...@gmail.com wrote: In order to accelerate the BinaryResponseWriter.write we extended this writer class to implement the docid to id tranformation by docValues (on memory) with no need to access stored field for id reading nor lazy loading of fields that also has a cost. That should improve read rate as docValues are sequential and should avoid disk IO. This docValues implementation is accessed during both query stages (as mentioned above) in case you ask for id's only, or only once, during the distributed search stage, in case you intend asking for stored fields different than id. We just started testing it for performance. I would love hearing any oppinions or test performances for this implementation Manu -- Regards, Shalin Shekhar Mangar.
Re: distributed search is significantly slower than direct search
Sure, I am out of office till end of week. I reply after i upload the patch
Re: distributed search is significantly slower than direct search
Hi, I isolated the case Installed on a new machine (2 x Xeon E5410 2.33GHz) I have an environment with 12Gb of memory. I assigned 6gb of memory to Solr and I’m not running any other memory consuming process so no memory issues should arise. Removed all indexes apart from two: emptyCore – empty – used for routing core1 – holds the stored data – has ~750,000 docs and size of 400Mb Again this is a single machine that holds both indexes. The query http://localhost:8210/solr/emptyCore/select?rows=5000q=*:*shards=127.0.0.1:8210/solr/core1wt=jsonQTime takes ~3 seconds and direct query http://localhost:8210/solr/core1/select?rows=5000q=*:*wt=json Qtime takes ~15 ms - a magnitude difference. I ran the long query several times and got an improvement of about a sec (33%) but that’s it. I need to better understand why this is happening. I tried looking at Solr code and debugging the issue but with no success. The one thing I did notice is that the getFirstMatch method which receives the doc id, searches the term dict and returns the internal id takes most of the time for some reason. I am pretty stuck and would appreciate any ideas My only solution for the moment is to bypass the distributed query, implement code in my own app that directly queries the relevant cores and handles the sorting etc.. Thanks On Sat, Nov 16, 2013 at 2:39 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: Did you say what the memory profile of your machine is? How much memory, and how large are the shards? This is just a random guess, but it might be that if you are memory-constrained, there is a lot of thrashing caused by paging (swapping?) in and out the sharded indexes while a single index can be scanned linearly, even if it does need to be paged in. -Mike On 11/14/2013 8:10 AM, Elran Dvir wrote: Hi, We tried returning just the id field and got exactly the same performance. Our system is distributed but all shards are in a single machine so network issues are not a factor. The code we found where Solr is spending its time is on the shard and not on the routing core, again all shards are local. We investigated the getFirstMatch() method and noticed that the MultiTermEnum.reset (inside MultiTerm.iterator) and MultiTerm.seekExact take 99% of the time. Inside these methods, the call to BlockTreeTermsReader$ FieldReader$SegmentTermsEnum$Frame.loadBlock takes most of the time. Out of the 7 seconds run these methods take ~5 and BinaryResponseWriter.write takes the rest(~ 2 seconds). We tried increasing cache sizes and got hits, but it only improved the query time by a second (~6), so no major effect. We are not indexing during our tests. The performance is similar. (How do we measure doc size? Is it important due to the fact that the performance is the same when returning only id field?) We still don't completely understand why the query takes this much longer although the cores are on the same machine. Is there a way to improve the performance (code, configuration, query)? -Original Message- From: idokis...@gmail.com [mailto:idokis...@gmail.com] On Behalf Of Manuel Le Normand Sent: Thursday, November 14, 2013 1:30 AM To: solr-user@lucene.apache.org Subject: Re: distributed search is significantly slower than direct search It's surprising such a query takes a long time, I would assume that after trying consistently q=*:* you should be getting cache hits and times should be faster. Try see in the adminUI how do your query/doc cache perform. Moreover, the query in itself is just asking the first 5000 docs that were indexed (returing the first [docid]), so seems all this time is wasted on transfer. Out of these 7 secs how much is spent on the above method? What do you return by default? How big is every doc you display in your results? Might be the matter that both collections work on the same ressources. Try elaborating your use-case. Anyway, it seems like you just made a test to see what will be the performance hit in a distributed environment so I'll try to explain some things we encountered in our benchmarks, with a case that has at least the similarity of the num of docs fetched. We reclaim 2000 docs every query, running over 40 shards. This means every shard is actually transfering to our frontend 2000 docs every document-match request (the first you were referring to). Even if lazily loaded, reading 2000 id's (on 40 servers) and lazy loading the fields is a tough job. Waiting for the slowest shard to respond, then sorting the docs and reloading (lazy or not) the top 2000 docs might take a long time. Our times are 4-8 secs, but do it's not possible comparing cases. We've done few steps that improved it along the way, steps that led to others. These were our starters: 1. Profile these queries from different servers and solr instances, try putting your finger what collection is working hard and why. Check if you're
Re: distributed search is significantly slower than direct search
Hi Yuval, quick question. You say that your code has 750k docs and around 400mb? Is this some kind of test dataset and you expect it to grow significantly? For an index of this size, I wouldn't use distributed search, single shard should be fine. Tomás On Sun, Nov 17, 2013 at 6:50 AM, Yuval Dotan yuvaldo...@gmail.com wrote: Hi, I isolated the case Installed on a new machine (2 x Xeon E5410 2.33GHz) I have an environment with 12Gb of memory. I assigned 6gb of memory to Solr and I’m not running any other memory consuming process so no memory issues should arise. Removed all indexes apart from two: emptyCore – empty – used for routing core1 – holds the stored data – has ~750,000 docs and size of 400Mb Again this is a single machine that holds both indexes. The query http://localhost:8210/solr/emptyCore/select?rows=5000q=*:*shards=127.0.0.1:8210/solr/core1wt=jsonQTime takes ~3 seconds and direct query http://localhost:8210/solr/core1/select?rows=5000q=*:*wt=json Qtime takes ~15 ms - a magnitude difference. I ran the long query several times and got an improvement of about a sec (33%) but that’s it. I need to better understand why this is happening. I tried looking at Solr code and debugging the issue but with no success. The one thing I did notice is that the getFirstMatch method which receives the doc id, searches the term dict and returns the internal id takes most of the time for some reason. I am pretty stuck and would appreciate any ideas My only solution for the moment is to bypass the distributed query, implement code in my own app that directly queries the relevant cores and handles the sorting etc.. Thanks On Sat, Nov 16, 2013 at 2:39 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: Did you say what the memory profile of your machine is? How much memory, and how large are the shards? This is just a random guess, but it might be that if you are memory-constrained, there is a lot of thrashing caused by paging (swapping?) in and out the sharded indexes while a single index can be scanned linearly, even if it does need to be paged in. -Mike On 11/14/2013 8:10 AM, Elran Dvir wrote: Hi, We tried returning just the id field and got exactly the same performance. Our system is distributed but all shards are in a single machine so network issues are not a factor. The code we found where Solr is spending its time is on the shard and not on the routing core, again all shards are local. We investigated the getFirstMatch() method and noticed that the MultiTermEnum.reset (inside MultiTerm.iterator) and MultiTerm.seekExact take 99% of the time. Inside these methods, the call to BlockTreeTermsReader$ FieldReader$SegmentTermsEnum$Frame.loadBlock takes most of the time. Out of the 7 seconds run these methods take ~5 and BinaryResponseWriter.write takes the rest(~ 2 seconds). We tried increasing cache sizes and got hits, but it only improved the query time by a second (~6), so no major effect. We are not indexing during our tests. The performance is similar. (How do we measure doc size? Is it important due to the fact that the performance is the same when returning only id field?) We still don't completely understand why the query takes this much longer although the cores are on the same machine. Is there a way to improve the performance (code, configuration, query)? -Original Message- From: idokis...@gmail.com [mailto:idokis...@gmail.com] On Behalf Of Manuel Le Normand Sent: Thursday, November 14, 2013 1:30 AM To: solr-user@lucene.apache.org Subject: Re: distributed search is significantly slower than direct search It's surprising such a query takes a long time, I would assume that after trying consistently q=*:* you should be getting cache hits and times should be faster. Try see in the adminUI how do your query/doc cache perform. Moreover, the query in itself is just asking the first 5000 docs that were indexed (returing the first [docid]), so seems all this time is wasted on transfer. Out of these 7 secs how much is spent on the above method? What do you return by default? How big is every doc you display in your results? Might be the matter that both collections work on the same ressources. Try elaborating your use-case. Anyway, it seems like you just made a test to see what will be the performance hit in a distributed environment so I'll try to explain some things we encountered in our benchmarks, with a case that has at least the similarity of the num of docs fetched. We reclaim 2000 docs every query, running over 40 shards. This means every shard is actually transfering to our frontend 2000 docs every document-match request (the first you were referring to). Even if lazily loaded, reading 2000 id's (on 40 servers) and lazy loading the fields is a tough job. Waiting for the slowest shard
Re: distributed search is significantly slower than direct search
Hi Tomás This is just a test environment meant only to reproduce the issue I am currently investigating. The number of documents should grow substantially (billions of docs). On Sun, Nov 17, 2013 at 7:12 PM, Tomás Fernández Löbbe tomasflo...@gmail.com wrote: Hi Yuval, quick question. You say that your code has 750k docs and around 400mb? Is this some kind of test dataset and you expect it to grow significantly? For an index of this size, I wouldn't use distributed search, single shard should be fine. Tomás On Sun, Nov 17, 2013 at 6:50 AM, Yuval Dotan yuvaldo...@gmail.com wrote: Hi, I isolated the case Installed on a new machine (2 x Xeon E5410 2.33GHz) I have an environment with 12Gb of memory. I assigned 6gb of memory to Solr and I’m not running any other memory consuming process so no memory issues should arise. Removed all indexes apart from two: emptyCore – empty – used for routing core1 – holds the stored data – has ~750,000 docs and size of 400Mb Again this is a single machine that holds both indexes. The query http://localhost:8210/solr/emptyCore/select?rows=5000q=*:*shards=127.0.0.1:8210/solr/core1wt=jsonQTime takes ~3 seconds and direct query http://localhost:8210/solr/core1/select?rows=5000q=*:*wt=json Qtime takes ~15 ms - a magnitude difference. I ran the long query several times and got an improvement of about a sec (33%) but that’s it. I need to better understand why this is happening. I tried looking at Solr code and debugging the issue but with no success. The one thing I did notice is that the getFirstMatch method which receives the doc id, searches the term dict and returns the internal id takes most of the time for some reason. I am pretty stuck and would appreciate any ideas My only solution for the moment is to bypass the distributed query, implement code in my own app that directly queries the relevant cores and handles the sorting etc.. Thanks On Sat, Nov 16, 2013 at 2:39 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: Did you say what the memory profile of your machine is? How much memory, and how large are the shards? This is just a random guess, but it might be that if you are memory-constrained, there is a lot of thrashing caused by paging (swapping?) in and out the sharded indexes while a single index can be scanned linearly, even if it does need to be paged in. -Mike On 11/14/2013 8:10 AM, Elran Dvir wrote: Hi, We tried returning just the id field and got exactly the same performance. Our system is distributed but all shards are in a single machine so network issues are not a factor. The code we found where Solr is spending its time is on the shard and not on the routing core, again all shards are local. We investigated the getFirstMatch() method and noticed that the MultiTermEnum.reset (inside MultiTerm.iterator) and MultiTerm.seekExact take 99% of the time. Inside these methods, the call to BlockTreeTermsReader$ FieldReader$SegmentTermsEnum$Frame.loadBlock takes most of the time. Out of the 7 seconds run these methods take ~5 and BinaryResponseWriter.write takes the rest(~ 2 seconds). We tried increasing cache sizes and got hits, but it only improved the query time by a second (~6), so no major effect. We are not indexing during our tests. The performance is similar. (How do we measure doc size? Is it important due to the fact that the performance is the same when returning only id field?) We still don't completely understand why the query takes this much longer although the cores are on the same machine. Is there a way to improve the performance (code, configuration, query)? -Original Message- From: idokis...@gmail.com [mailto:idokis...@gmail.com] On Behalf Of Manuel Le Normand Sent: Thursday, November 14, 2013 1:30 AM To: solr-user@lucene.apache.org Subject: Re: distributed search is significantly slower than direct search It's surprising such a query takes a long time, I would assume that after trying consistently q=*:* you should be getting cache hits and times should be faster. Try see in the adminUI how do your query/doc cache perform. Moreover, the query in itself is just asking the first 5000 docs that were indexed (returing the first [docid]), so seems all this time is wasted on transfer. Out of these 7 secs how much is spent on the above method? What do you return by default? How big is every doc you display in your results? Might be the matter that both collections work on the same ressources. Try elaborating your use-case. Anyway, it seems like you just made a test to see what will be the performance hit in a distributed environment so I'll try to explain some things we encountered in our benchmarks, with a case that has
Re: distributed search is significantly slower than direct search
You are asking for 5000 docs right? And that’s forcing us to look up 5000 external to internal ids. I think this always had a cost, but it’s obviously worse if you ask for a ton of results. I don’t think single node has to do this? And if we had like Searcher leases or something (we will eventually), I think we could avoid it and just use internal ids. - Mark On Nov 17, 2013, at 12:44 PM, Yuval Dotan yuvaldo...@gmail.com wrote: Hi Tomás This is just a test environment meant only to reproduce the issue I am currently investigating. The number of documents should grow substantially (billions of docs). On Sun, Nov 17, 2013 at 7:12 PM, Tomás Fernández Löbbe tomasflo...@gmail.com wrote: Hi Yuval, quick question. You say that your code has 750k docs and around 400mb? Is this some kind of test dataset and you expect it to grow significantly? For an index of this size, I wouldn't use distributed search, single shard should be fine. Tomás On Sun, Nov 17, 2013 at 6:50 AM, Yuval Dotan yuvaldo...@gmail.com wrote: Hi, I isolated the case Installed on a new machine (2 x Xeon E5410 2.33GHz) I have an environment with 12Gb of memory. I assigned 6gb of memory to Solr and I’m not running any other memory consuming process so no memory issues should arise. Removed all indexes apart from two: emptyCore – empty – used for routing core1 – holds the stored data – has ~750,000 docs and size of 400Mb Again this is a single machine that holds both indexes. The query http://localhost:8210/solr/emptyCore/select?rows=5000q=*:*shards=127.0.0.1:8210/solr/core1wt=jsonQTime takes ~3 seconds and direct query http://localhost:8210/solr/core1/select?rows=5000q=*:*wt=json Qtime takes ~15 ms - a magnitude difference. I ran the long query several times and got an improvement of about a sec (33%) but that’s it. I need to better understand why this is happening. I tried looking at Solr code and debugging the issue but with no success. The one thing I did notice is that the getFirstMatch method which receives the doc id, searches the term dict and returns the internal id takes most of the time for some reason. I am pretty stuck and would appreciate any ideas My only solution for the moment is to bypass the distributed query, implement code in my own app that directly queries the relevant cores and handles the sorting etc.. Thanks On Sat, Nov 16, 2013 at 2:39 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: Did you say what the memory profile of your machine is? How much memory, and how large are the shards? This is just a random guess, but it might be that if you are memory-constrained, there is a lot of thrashing caused by paging (swapping?) in and out the sharded indexes while a single index can be scanned linearly, even if it does need to be paged in. -Mike On 11/14/2013 8:10 AM, Elran Dvir wrote: Hi, We tried returning just the id field and got exactly the same performance. Our system is distributed but all shards are in a single machine so network issues are not a factor. The code we found where Solr is spending its time is on the shard and not on the routing core, again all shards are local. We investigated the getFirstMatch() method and noticed that the MultiTermEnum.reset (inside MultiTerm.iterator) and MultiTerm.seekExact take 99% of the time. Inside these methods, the call to BlockTreeTermsReader$ FieldReader$SegmentTermsEnum$Frame.loadBlock takes most of the time. Out of the 7 seconds run these methods take ~5 and BinaryResponseWriter.write takes the rest(~ 2 seconds). We tried increasing cache sizes and got hits, but it only improved the query time by a second (~6), so no major effect. We are not indexing during our tests. The performance is similar. (How do we measure doc size? Is it important due to the fact that the performance is the same when returning only id field?) We still don't completely understand why the query takes this much longer although the cores are on the same machine. Is there a way to improve the performance (code, configuration, query)? -Original Message- From: idokis...@gmail.com [mailto:idokis...@gmail.com] On Behalf Of Manuel Le Normand Sent: Thursday, November 14, 2013 1:30 AM To: solr-user@lucene.apache.org Subject: Re: distributed search is significantly slower than direct search It's surprising such a query takes a long time, I would assume that after trying consistently q=*:* you should be getting cache hits and times should be faster. Try see in the adminUI how do your query/doc cache perform. Moreover, the query in itself is just asking the first 5000 docs that were indexed (returing the first [docid]), so seems all this time is wasted on transfer. Out of these 7 secs how much is spent on the above method? What do you return by default? How big is every doc you display in your results
Re: distributed search is significantly slower than direct search
In order to accelerate the BinaryResponseWriter.write we extended this writer class to implement the docid to id tranformation by docValues (on memory) with no need to access stored field for id reading nor lazy loading of fields that also has a cost. That should improve read rate as docValues are sequential and should avoid disk IO. This docValues implementation is accessed during both query stages (as mentioned above) in case you ask for id's only, or only once, during the distributed search stage, in case you intend asking for stored fields different than id. We just started testing it for performance. I would love hearing any oppinions or test performances for this implementation Manu
Re: distributed search is significantly slower than direct search
Did you say what the memory profile of your machine is? How much memory, and how large are the shards? This is just a random guess, but it might be that if you are memory-constrained, there is a lot of thrashing caused by paging (swapping?) in and out the sharded indexes while a single index can be scanned linearly, even if it does need to be paged in. -Mike On 11/14/2013 8:10 AM, Elran Dvir wrote: Hi, We tried returning just the id field and got exactly the same performance. Our system is distributed but all shards are in a single machine so network issues are not a factor. The code we found where Solr is spending its time is on the shard and not on the routing core, again all shards are local. We investigated the getFirstMatch() method and noticed that the MultiTermEnum.reset (inside MultiTerm.iterator) and MultiTerm.seekExact take 99% of the time. Inside these methods, the call to BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock takes most of the time. Out of the 7 seconds run these methods take ~5 and BinaryResponseWriter.write takes the rest(~ 2 seconds). We tried increasing cache sizes and got hits, but it only improved the query time by a second (~6), so no major effect. We are not indexing during our tests. The performance is similar. (How do we measure doc size? Is it important due to the fact that the performance is the same when returning only id field?) We still don't completely understand why the query takes this much longer although the cores are on the same machine. Is there a way to improve the performance (code, configuration, query)? -Original Message- From: idokis...@gmail.com [mailto:idokis...@gmail.com] On Behalf Of Manuel Le Normand Sent: Thursday, November 14, 2013 1:30 AM To: solr-user@lucene.apache.org Subject: Re: distributed search is significantly slower than direct search It's surprising such a query takes a long time, I would assume that after trying consistently q=*:* you should be getting cache hits and times should be faster. Try see in the adminUI how do your query/doc cache perform. Moreover, the query in itself is just asking the first 5000 docs that were indexed (returing the first [docid]), so seems all this time is wasted on transfer. Out of these 7 secs how much is spent on the above method? What do you return by default? How big is every doc you display in your results? Might be the matter that both collections work on the same ressources. Try elaborating your use-case. Anyway, it seems like you just made a test to see what will be the performance hit in a distributed environment so I'll try to explain some things we encountered in our benchmarks, with a case that has at least the similarity of the num of docs fetched. We reclaim 2000 docs every query, running over 40 shards. This means every shard is actually transfering to our frontend 2000 docs every document-match request (the first you were referring to). Even if lazily loaded, reading 2000 id's (on 40 servers) and lazy loading the fields is a tough job. Waiting for the slowest shard to respond, then sorting the docs and reloading (lazy or not) the top 2000 docs might take a long time. Our times are 4-8 secs, but do it's not possible comparing cases. We've done few steps that improved it along the way, steps that led to others. These were our starters: 1. Profile these queries from different servers and solr instances, try putting your finger what collection is working hard and why. Check if you're stuck on components that don't have an added value for you but are used by default. 2. Consider eliminating the doc cache. It loads lots of (partly) lazy documents that their probability of secondary usage is low. There's no such thing popular docs when requesting so many docs. You may be using your memory in a better way. 3. Bottleneck check - inner server metrics as cpu user / iowait, packets transferred over the network, page faults etc. are excellent in order to understand if the disk/network/cpu is slowing you down. Then upgrade hardware in one of the shards to check if it helps by looking at the upgraded shard qTime compared to other. 4. Warm up the index after commiting - try to benchmark how do queries performs before and after some warm-up, let's say some few hundreds of queries (from your previous system) in order to warm up the os cache (assuming your using NRTDirectoryFactory) Good luck, Manu On Wed, Nov 13, 2013 at 2:38 PM, Erick Erickson erickerick...@gmail.comwrote: One thing you can try, and this is more diagnostic than a cure, is return just the id field (and insure that lazy field loading is true). That'll tell you whether the issue is actually fetching the document off disk and decompressing, although frankly that's unlikely since you can get your 5,000 rows from a single machine quickly. The code you found where Solr is spending its time
RE: distributed search is significantly slower than direct search
Hi, We tried returning just the id field and got exactly the same performance. Our system is distributed but all shards are in a single machine so network issues are not a factor. The code we found where Solr is spending its time is on the shard and not on the routing core, again all shards are local. We investigated the getFirstMatch() method and noticed that the MultiTermEnum.reset (inside MultiTerm.iterator) and MultiTerm.seekExact take 99% of the time. Inside these methods, the call to BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock takes most of the time. Out of the 7 seconds run these methods take ~5 and BinaryResponseWriter.write takes the rest(~ 2 seconds). We tried increasing cache sizes and got hits, but it only improved the query time by a second (~6), so no major effect. We are not indexing during our tests. The performance is similar. (How do we measure doc size? Is it important due to the fact that the performance is the same when returning only id field?) We still don't completely understand why the query takes this much longer although the cores are on the same machine. Is there a way to improve the performance (code, configuration, query)? -Original Message- From: idokis...@gmail.com [mailto:idokis...@gmail.com] On Behalf Of Manuel Le Normand Sent: Thursday, November 14, 2013 1:30 AM To: solr-user@lucene.apache.org Subject: Re: distributed search is significantly slower than direct search It's surprising such a query takes a long time, I would assume that after trying consistently q=*:* you should be getting cache hits and times should be faster. Try see in the adminUI how do your query/doc cache perform. Moreover, the query in itself is just asking the first 5000 docs that were indexed (returing the first [docid]), so seems all this time is wasted on transfer. Out of these 7 secs how much is spent on the above method? What do you return by default? How big is every doc you display in your results? Might be the matter that both collections work on the same ressources. Try elaborating your use-case. Anyway, it seems like you just made a test to see what will be the performance hit in a distributed environment so I'll try to explain some things we encountered in our benchmarks, with a case that has at least the similarity of the num of docs fetched. We reclaim 2000 docs every query, running over 40 shards. This means every shard is actually transfering to our frontend 2000 docs every document-match request (the first you were referring to). Even if lazily loaded, reading 2000 id's (on 40 servers) and lazy loading the fields is a tough job. Waiting for the slowest shard to respond, then sorting the docs and reloading (lazy or not) the top 2000 docs might take a long time. Our times are 4-8 secs, but do it's not possible comparing cases. We've done few steps that improved it along the way, steps that led to others. These were our starters: 1. Profile these queries from different servers and solr instances, try putting your finger what collection is working hard and why. Check if you're stuck on components that don't have an added value for you but are used by default. 2. Consider eliminating the doc cache. It loads lots of (partly) lazy documents that their probability of secondary usage is low. There's no such thing popular docs when requesting so many docs. You may be using your memory in a better way. 3. Bottleneck check - inner server metrics as cpu user / iowait, packets transferred over the network, page faults etc. are excellent in order to understand if the disk/network/cpu is slowing you down. Then upgrade hardware in one of the shards to check if it helps by looking at the upgraded shard qTime compared to other. 4. Warm up the index after commiting - try to benchmark how do queries performs before and after some warm-up, let's say some few hundreds of queries (from your previous system) in order to warm up the os cache (assuming your using NRTDirectoryFactory) Good luck, Manu On Wed, Nov 13, 2013 at 2:38 PM, Erick Erickson erickerick...@gmail.comwrote: One thing you can try, and this is more diagnostic than a cure, is return just the id field (and insure that lazy field loading is true). That'll tell you whether the issue is actually fetching the document off disk and decompressing, although frankly that's unlikely since you can get your 5,000 rows from a single machine quickly. The code you found where Solr is spending its time, is that on the routing core or on the shards? I actually have a hard time understanding how that code could take a long time, doesn't seem right. You are transferring 5,000 docs across the network, so it's possible that your network is just slow, that's certainly a difference between the local and remote case, but that's a stab in the dark. Not much help I know, Erick On Wed, Nov 13, 2013 at 2:52 AM, Elran
Re: distributed search is significantly slower than direct search
One thing you can try, and this is more diagnostic than a cure, is return
just
the id field (and insure that lazy field loading is true). That'll tell you
whether
the issue is actually fetching the document off disk and decompressing,
although
frankly that's unlikely since you can get your 5,000 rows from a single
machine
quickly.
The code you found where Solr is spending its time, is that on the
routing core
or on the shards? I actually have a hard time understanding how that
code could take a long time, doesn't seem right.
You are transferring 5,000 docs across the network, so it's possible that
your network is just slow, that's certainly a difference between the local
and remote case, but that's a stab in the dark.
Not much help I know,
Erick
On Wed, Nov 13, 2013 at 2:52 AM, Elran Dvir elr...@checkpoint.com wrote:
Erick, Thanks for your response.
We are upgrading our system using Solr.
We need to preserve old functionality. Our client displays 5K document
and groups them.
Is there a way to refactor code in order to improve distributed documents
fetching?
Thanks.
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Wednesday, October 30, 2013 3:17 AM
To: solr-user@lucene.apache.org
Subject: Re: distributed search is significantly slower than direct search
You can't. There will inevitably be some overhead in the distributed case.
That said, 7 seconds is quite long.
5,000 rows is excessive, and probably where your issue is. You're having
to go out and fetch the docs across the wire. Perhaps there is some
batching that could be done there, I don't know whether this is one
document per request or not.
Why 5K docs?
Best,
Erick
On Tue, Oct 29, 2013 at 2:54 AM, Elran Dvir elr...@checkpoint.com wrote:
Hi all,
I am using Solr 4.4 with multi cores. One core (called template) is my
routing core.
When I run
http://127.0.0.1:8983/solr/template/select?rows=5000q=*:*shards=127.
0.0.1:8983/solr/core1,
it consistently takes about 7s.
When I run http://127.0.0.1:8983/solr/core1/select?rows=5000q=*:*, it
consistently takes about 40ms.
I profiled the distributed query.
This is the distributed query process (I hope the terms are accurate):
When solr identifies a distributed query, it sends the query to the
shard and get matched shard docs.
Then it sends another query to the shard to get the Solr documents.
Most time is spent in the last stage in the function process of
QueryComponent in:
for (int i=0; iidArr.size(); i++) {
int id = req.getSearcher().getFirstMatch(
new Term(idField.getName(),
idField.getType().toInternal(idArr.get(i;
How can I make my distributed query as fast as the direct one?
Thanks.
Email secured by Check Point
Re: distributed search is significantly slower than direct search
It's surprising such a query takes a long time, I would assume that after trying consistently q=*:* you should be getting cache hits and times should be faster. Try see in the adminUI how do your query/doc cache perform. Moreover, the query in itself is just asking the first 5000 docs that were indexed (returing the first [docid]), so seems all this time is wasted on transfer. Out of these 7 secs how much is spent on the above method? What do you return by default? How big is every doc you display in your results? Might be the matter that both collections work on the same ressources. Try elaborating your use-case. Anyway, it seems like you just made a test to see what will be the performance hit in a distributed environment so I'll try to explain some things we encountered in our benchmarks, with a case that has at least the similarity of the num of docs fetched. We reclaim 2000 docs every query, running over 40 shards. This means every shard is actually transfering to our frontend 2000 docs every document-match request (the first you were referring to). Even if lazily loaded, reading 2000 id's (on 40 servers) and lazy loading the fields is a tough job. Waiting for the slowest shard to respond, then sorting the docs and reloading (lazy or not) the top 2000 docs might take a long time. Our times are 4-8 secs, but do it's not possible comparing cases. We've done few steps that improved it along the way, steps that led to others. These were our starters: 1. Profile these queries from different servers and solr instances, try putting your finger what collection is working hard and why. Check if you're stuck on components that don't have an added value for you but are used by default. 2. Consider eliminating the doc cache. It loads lots of (partly) lazy documents that their probability of secondary usage is low. There's no such thing popular docs when requesting so many docs. You may be using your memory in a better way. 3. Bottleneck check - inner server metrics as cpu user / iowait, packets transferred over the network, page faults etc. are excellent in order to understand if the disk/network/cpu is slowing you down. Then upgrade hardware in one of the shards to check if it helps by looking at the upgraded shard qTime compared to other. 4. Warm up the index after commiting - try to benchmark how do queries performs before and after some warm-up, let's say some few hundreds of queries (from your previous system) in order to warm up the os cache (assuming your using NRTDirectoryFactory) Good luck, Manu On Wed, Nov 13, 2013 at 2:38 PM, Erick Erickson erickerick...@gmail.comwrote: One thing you can try, and this is more diagnostic than a cure, is return just the id field (and insure that lazy field loading is true). That'll tell you whether the issue is actually fetching the document off disk and decompressing, although frankly that's unlikely since you can get your 5,000 rows from a single machine quickly. The code you found where Solr is spending its time, is that on the routing core or on the shards? I actually have a hard time understanding how that code could take a long time, doesn't seem right. You are transferring 5,000 docs across the network, so it's possible that your network is just slow, that's certainly a difference between the local and remote case, but that's a stab in the dark. Not much help I know, Erick On Wed, Nov 13, 2013 at 2:52 AM, Elran Dvir elr...@checkpoint.com wrote: Erick, Thanks for your response. We are upgrading our system using Solr. We need to preserve old functionality. Our client displays 5K document and groups them. Is there a way to refactor code in order to improve distributed documents fetching? Thanks. -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Wednesday, October 30, 2013 3:17 AM To: solr-user@lucene.apache.org Subject: Re: distributed search is significantly slower than direct search You can't. There will inevitably be some overhead in the distributed case. That said, 7 seconds is quite long. 5,000 rows is excessive, and probably where your issue is. You're having to go out and fetch the docs across the wire. Perhaps there is some batching that could be done there, I don't know whether this is one document per request or not. Why 5K docs? Best, Erick On Tue, Oct 29, 2013 at 2:54 AM, Elran Dvir elr...@checkpoint.com wrote: Hi all, I am using Solr 4.4 with multi cores. One core (called template) is my routing core. When I run http://127.0.0.1:8983/solr/template/select?rows=5000q=*:*shards=127. 0.0.1:8983/solr/core1, it consistently takes about 7s. When I run http://127.0.0.1:8983/solr/core1/select?rows=5000q=*:*, it consistently takes about 40ms. I profiled the distributed query. This is the distributed query process (I hope the terms
RE: distributed search is significantly slower than direct search
Erick, Thanks for your response.
We are upgrading our system using Solr.
We need to preserve old functionality. Our client displays 5K document and
groups them.
Is there a way to refactor code in order to improve distributed documents
fetching?
Thanks.
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Wednesday, October 30, 2013 3:17 AM
To: solr-user@lucene.apache.org
Subject: Re: distributed search is significantly slower than direct search
You can't. There will inevitably be some overhead in the distributed case. That
said, 7 seconds is quite long.
5,000 rows is excessive, and probably where your issue is. You're having to go
out and fetch the docs across the wire. Perhaps there is some batching that
could be done there, I don't know whether this is one document per request or
not.
Why 5K docs?
Best,
Erick
On Tue, Oct 29, 2013 at 2:54 AM, Elran Dvir elr...@checkpoint.com wrote:
Hi all,
I am using Solr 4.4 with multi cores. One core (called template) is my
routing core.
When I run
http://127.0.0.1:8983/solr/template/select?rows=5000q=*:*shards=127.
0.0.1:8983/solr/core1,
it consistently takes about 7s.
When I run http://127.0.0.1:8983/solr/core1/select?rows=5000q=*:*, it
consistently takes about 40ms.
I profiled the distributed query.
This is the distributed query process (I hope the terms are accurate):
When solr identifies a distributed query, it sends the query to the
shard and get matched shard docs.
Then it sends another query to the shard to get the Solr documents.
Most time is spent in the last stage in the function process of
QueryComponent in:
for (int i=0; iidArr.size(); i++) {
int id = req.getSearcher().getFirstMatch(
new Term(idField.getName(),
idField.getType().toInternal(idArr.get(i;
How can I make my distributed query as fast as the direct one?
Thanks.
Email secured by Check Point
Re: distributed search is significantly slower than direct search
You can't. There will inevitably be some overhead in the
distributed case. That said, 7 seconds is quite long.
5,000 rows is excessive, and probably where your issue is. You're
having to go out and fetch the docs across the wire. Perhaps there
is some batching that could be done there, I don't know whether this
is one document per request or not.
Why 5K docs?
Best,
Erick
On Tue, Oct 29, 2013 at 2:54 AM, Elran Dvir elr...@checkpoint.com wrote:
Hi all,
I am using Solr 4.4 with multi cores. One core (called template) is my
routing core.
When I run
http://127.0.0.1:8983/solr/template/select?rows=5000q=*:*shards=127.0.0.1:8983/solr/core1,
it consistently takes about 7s.
When I run http://127.0.0.1:8983/solr/core1/select?rows=5000q=*:*, it
consistently takes about 40ms.
I profiled the distributed query.
This is the distributed query process (I hope the terms are accurate):
When solr identifies a distributed query, it sends the query to the shard
and get matched shard docs.
Then it sends another query to the shard to get the Solr documents.
Most time is spent in the last stage in the function process of
QueryComponent in:
for (int i=0; iidArr.size(); i++) {
int id = req.getSearcher().getFirstMatch(
new Term(idField.getName(),
idField.getType().toInternal(idArr.get(i;
How can I make my distributed query as fast as the direct one?
Thanks.
