Re: Inconsistent slave performance after optimize
On Wed, Oct 27, 2010 at 8:59 PM, Jonathan Rochkind wrote: > Seriously, at least try JVM argument -XX:+UseConcMarkSweepGC . That > argument took care of very similar symptoms I was having. I never did > figure out exactly what was causing them, but at some point I tried that JVM > argument, and they went away never to come back (which I guess is a clue > about what was causing the slowdown, but the JVM still confuses me). > Will do. Thanks for the tip. Mason > > From: Mason Hale [masonh...@gmail.com] > Sent: Wednesday, October 27, 2010 9:03 PM > To: solr-user@lucene.apache.org > Subject: Re: Inconsistent slave performance after optimize > > On Wed, Oct 27, 2010 at 7:18 PM, Ken Krugler >wrote: > > > Normally I'd say like you were getting into swap hell, but based on your > > settings you only have 5GB of JVM space being used, on a 16GB box. > > > > Just to confirm, nothing else is using lots of memory, right? And the > "top" > > command isn't showing any swap usage, right? > > > > > Correct. Only thing of note running on this machine is Solr. > > I don't have a poor performing server on hand at the moment, but I recall > checking top when it was tanking, and it was not showing any swap usage. > > > > When you encounter very slow search times, what does the top command say > > about system load and cpu vs. I/O percentages? > > > > I did look at iostat -x when the server was running slow and IO util was > 100%. > > This lead me to believe the problem was cache-warming related, and that > data > needed to be loaded into Solr caches and/or files loaded into the file > system cache. > > Does that yield any additional clues. > > In this does happen again, what stats should I collect? > > (Note to self: need to install sar on these servers to collect historical > performance data...) > > Mason >
RE: Inconsistent slave performance after optimize
Seriously, at least try JVM argument -XX:+UseConcMarkSweepGC . That argument took care of very similar symptoms I was having. I never did figure out exactly what was causing them, but at some point I tried that JVM argument, and they went away never to come back (which I guess is a clue about what was causing the slowdown, but the JVM still confuses me). From: Mason Hale [masonh...@gmail.com] Sent: Wednesday, October 27, 2010 9:03 PM To: solr-user@lucene.apache.org Subject: Re: Inconsistent slave performance after optimize On Wed, Oct 27, 2010 at 7:18 PM, Ken Krugler wrote: > Normally I'd say like you were getting into swap hell, but based on your > settings you only have 5GB of JVM space being used, on a 16GB box. > > Just to confirm, nothing else is using lots of memory, right? And the "top" > command isn't showing any swap usage, right? > > Correct. Only thing of note running on this machine is Solr. I don't have a poor performing server on hand at the moment, but I recall checking top when it was tanking, and it was not showing any swap usage. > When you encounter very slow search times, what does the top command say > about system load and cpu vs. I/O percentages? > I did look at iostat -x when the server was running slow and IO util was 100%. This lead me to believe the problem was cache-warming related, and that data needed to be loaded into Solr caches and/or files loaded into the file system cache. Does that yield any additional clues. In this does happen again, what stats should I collect? (Note to self: need to install sar on these servers to collect historical performance data...) Mason
Re: Inconsistent slave performance after optimize
On Wed, Oct 27, 2010 at 7:18 PM, Ken Krugler wrote: > Normally I'd say like you were getting into swap hell, but based on your > settings you only have 5GB of JVM space being used, on a 16GB box. > > Just to confirm, nothing else is using lots of memory, right? And the "top" > command isn't showing any swap usage, right? > > Correct. Only thing of note running on this machine is Solr. I don't have a poor performing server on hand at the moment, but I recall checking top when it was tanking, and it was not showing any swap usage. > When you encounter very slow search times, what does the top command say > about system load and cpu vs. I/O percentages? > I did look at iostat -x when the server was running slow and IO util was 100%. This lead me to believe the problem was cache-warming related, and that data needed to be loaded into Solr caches and/or files loaded into the file system cache. Does that yield any additional clues. In this does happen again, what stats should I collect? (Note to self: need to install sar on these servers to collect historical performance data...) Mason
Re: Inconsistent slave performance after optimize
Normally I'd say like you were getting into swap hell, but based on your settings you only have 5GB of JVM space being used, on a 16GB box. Just to confirm, nothing else is using lots of memory, right? And the "top" command isn't showing any swap usage, right? When you encounter very slow search times, what does the top command say about system load and cpu vs. I/O percentages? -- Ken On Oct 27, 2010, at 3:33pm, Mason Hale wrote: Hi Lance -- Thanks for the reply. Did you restart all of these slave servers? That would help. We discovered independently that restarting the slave nodes resulted in dramatically improved performance (e.g. from 2.0 sec average response to 0.25 sec average). Can you please explain why this is the case? I would expect a process restart to invalid caches and thus trigger additional cache-warming overhead, slowing things down, not speeding things up. What garbage collection options do you use? We've not tweaked the garbage collection settings. We're using - Xms512M -Xmx5000M on the command line. Which release of Solr? version 1.4.0 How many Searchers are there in admin/stats.jsp? I'm looking much later, and after a restart -- but I currently see 2 searchers listed. I admit I'm not sure what I'm looking for on this page. thanks, Mason On Wed, Oct 27, 2010 at 2:25 AM, Lance Norskog wrote: Did you restart all of these slave servers? That would help. What garbage collection options do you use? Which release of Solr? How many Searchers are there in admin/stats.jsp? Searchers hold open all kinds of memory. They are supposed to cycle out. These are standard questions, but- what you are seeing is definitely not normal. Separately, if you want a regular optimization regime, there is a new option called 'maxSegments' to the optimize command. If you have solrconfig mergeFactor set to 10, then optimize 'maxSegments=8' will roll up the very smallest segments. This allows you to have a gradual optimization (and replication overhead) instead of big ones. Mason Hale wrote: Hello esteemed Solr community -- I'm observing some inconsistent performance on our slave servers after recently optimizing our master server. Our configuration is as follows: - all servers are hosted at Amazon EC2, running Ubuntu 8.04 - 1 master with heavy insert/update traffic, about 125K new documents per day (m1.large, ~8GB RAM) - autocommit every 1 minute - 3 slaves (m2.xlarge instance sizes, ~16GB RAM) - replicate every 5 minutes - we have configured autowarming queries for these machines - autowarmCount = 0 - Total index size is ~7M documents We were seeing increasing, but gradual performance degradation across all nodes. So we decided to try optimizing our index to improve performance. In preparation for the optimize we disabled replication polling on all slaves. We also turned off all workers that were writing to the index. Then we ran optimize on the master. The optimize took 45-60 minutes to complete, and the total size went from 68GB down to 23GB. We then enabled replication on each slave one at a time. The first slave we re-enabled took about 15 minutes to copy the new files. Once the files were copied the performance of slave plummeted. Average response time went from 0.75 sec to 45 seconds. Over the past 18 hours the average response time has gradually gown down to around 1.2 seconds now. Before re-enabling replication the second slave, we first removed it from our load-balanced pool of available search servers. This server's average query performance also degraded quickly, and then (unlike the first slave we replicated) did not improve. It stayed at around 30 secs per query. On the theory that this is a cache-warming issue, we added this server back to the pool in hopes that additional traffic would warm the cache. But what we saw was a quick spike of much worse performance (50 sec / query on average) followed by a slow/gradual decline in average response times. As of now (10 hours after the initial replication) this server is still reporting an average response time of ~2 seconds. This is much worse than before the optimize and is a counter- intuitive result. We expected an index 1/3 the size would be faster, not slower. On the theory that the index files needed to be loaded into the file system cache, I used the 'dd' command to copy the contents of the data/index directory to /dev/null, but that did not result in any noticeable performance improvement. At this point, things were not going as expected. We did not expect the replication after an optimize to result in such horrid performance. So we decided to let the last slave continue to serve stale results while we waited 4 hours for the other two slaves to approach some acceptable performance level. After the 4 hour break, we re-moved the 3rd and last slave server from our load-balancin
RE: Inconsistent slave performance after optimize
I'm guessing the slaves you restarted were running low on RAM, and possibly engaged in out of control GC. I have had good luck using the JVM option "-XX:+UseConcMarkSweepGC ", which seems to result in GC happening in another thread and not interfering with the servicing of requests. If that's what's going on, it may also indicate that you should give your Solr more RAM, for your current cache settings. On the other hand, apparently sometimes out of control GC can also indicate that your JVM has too MUCH ram. I don't know the answers, there doesn't seem to be any simple way to figure out how much RAM your Solr needs, or if it's having problems as a result of not enough (or too much!), I haven't really figured out how to do it. For a server process like Solr, I also think it makes sense to set -xms the same value as -xmx, I don't think there's much to be gained by leaving a window, and by setting them the same the JVM won't spend any time increasing it's heap size. From: Mason Hale [masonh...@gmail.com] Sent: Wednesday, October 27, 2010 6:33 PM To: solr-user@lucene.apache.org Subject: Re: Inconsistent slave performance after optimize Hi Lance -- Thanks for the reply. > Did you restart all of these slave servers? That would help. We discovered independently that restarting the slave nodes resulted in dramatically improved performance (e.g. from 2.0 sec average response to 0.25 sec average). Can you please explain why this is the case? I would expect a process restart to invalid caches and thus trigger additional cache-warming overhead, slowing things down, not speeding things up. > What garbage collection options do you use? We've not tweaked the garbage collection settings. We're using -Xms512M -Xmx5000M on the command line. > Which release of Solr? version 1.4.0 > How many Searchers are there in admin/stats.jsp? I'm looking much later, and after a restart -- but I currently see 2 searchers listed. I admit I'm not sure what I'm looking for on this page. thanks, Mason On Wed, Oct 27, 2010 at 2:25 AM, Lance Norskog wrote: > Did you restart all of these slave servers? That would help. > What garbage collection options do you use? > Which release of Solr? > How many Searchers are there in admin/stats.jsp? > Searchers hold open all kinds of memory. They are supposed to cycle out. > > These are standard questions, but- what you are seeing is definitely not > normal. > > Separately, if you want a regular optimization regime, there is a new > option called 'maxSegments' to the optimize command. If you have solrconfig > mergeFactor set to 10, then optimize 'maxSegments=8' will roll up the very > smallest segments. This allows you to have a gradual optimization (and > replication overhead) instead of big ones. > > > Mason Hale wrote: > >> Hello esteemed Solr community -- >> >> I'm observing some inconsistent performance on our slave servers after >> recently optimizing our master server. >> >> Our configuration is as follows: >> >> - all servers are hosted at Amazon EC2, running Ubuntu 8.04 >> - 1 master with heavy insert/update traffic, about 125K new documents >> per day (m1.large, ~8GB RAM) >>- autocommit every 1 minute >> - 3 slaves (m2.xlarge instance sizes, ~16GB RAM) >>- replicate every 5 minutes >>- we have configured autowarming queries for these machines >>- autowarmCount = 0 >> - Total index size is ~7M documents >> >> We were seeing increasing, but gradual performance degradation across all >> nodes. >> So we decided to try optimizing our index to improve performance. >> >> In preparation for the optimize we disabled replication polling on all >> slaves. We also turned off all >> workers that were writing to the index. Then we ran optimize on the >> master. >> >> The optimize took 45-60 minutes to complete, and the total size went from >> 68GB down to 23GB. >> >> We then enabled replication on each slave one at a time. >> >> The first slave we re-enabled took about 15 minutes to copy the new files. >> Once the files were copied >> the performance of slave plummeted. Average response time went from 0.75 >> sec >> to 45 seconds. >> Over the past 18 hours the average response time has gradually gown down >> to >> around 1.2 seconds now. >> >> Before re-enabling replication the second slave, we first removed it from >> our load-balanced pool of available search servers. >> This server's average query performance also degraded quickly, and then
Re: Inconsistent slave performance after optimize
Hi Lance -- Thanks for the reply. > Did you restart all of these slave servers? That would help. We discovered independently that restarting the slave nodes resulted in dramatically improved performance (e.g. from 2.0 sec average response to 0.25 sec average). Can you please explain why this is the case? I would expect a process restart to invalid caches and thus trigger additional cache-warming overhead, slowing things down, not speeding things up. > What garbage collection options do you use? We've not tweaked the garbage collection settings. We're using -Xms512M -Xmx5000M on the command line. > Which release of Solr? version 1.4.0 > How many Searchers are there in admin/stats.jsp? I'm looking much later, and after a restart -- but I currently see 2 searchers listed. I admit I'm not sure what I'm looking for on this page. thanks, Mason On Wed, Oct 27, 2010 at 2:25 AM, Lance Norskog wrote: > Did you restart all of these slave servers? That would help. > What garbage collection options do you use? > Which release of Solr? > How many Searchers are there in admin/stats.jsp? > Searchers hold open all kinds of memory. They are supposed to cycle out. > > These are standard questions, but- what you are seeing is definitely not > normal. > > Separately, if you want a regular optimization regime, there is a new > option called 'maxSegments' to the optimize command. If you have solrconfig > mergeFactor set to 10, then optimize 'maxSegments=8' will roll up the very > smallest segments. This allows you to have a gradual optimization (and > replication overhead) instead of big ones. > > > Mason Hale wrote: > >> Hello esteemed Solr community -- >> >> I'm observing some inconsistent performance on our slave servers after >> recently optimizing our master server. >> >> Our configuration is as follows: >> >> - all servers are hosted at Amazon EC2, running Ubuntu 8.04 >> - 1 master with heavy insert/update traffic, about 125K new documents >> per day (m1.large, ~8GB RAM) >>- autocommit every 1 minute >> - 3 slaves (m2.xlarge instance sizes, ~16GB RAM) >>- replicate every 5 minutes >>- we have configured autowarming queries for these machines >>- autowarmCount = 0 >> - Total index size is ~7M documents >> >> We were seeing increasing, but gradual performance degradation across all >> nodes. >> So we decided to try optimizing our index to improve performance. >> >> In preparation for the optimize we disabled replication polling on all >> slaves. We also turned off all >> workers that were writing to the index. Then we ran optimize on the >> master. >> >> The optimize took 45-60 minutes to complete, and the total size went from >> 68GB down to 23GB. >> >> We then enabled replication on each slave one at a time. >> >> The first slave we re-enabled took about 15 minutes to copy the new files. >> Once the files were copied >> the performance of slave plummeted. Average response time went from 0.75 >> sec >> to 45 seconds. >> Over the past 18 hours the average response time has gradually gown down >> to >> around 1.2 seconds now. >> >> Before re-enabling replication the second slave, we first removed it from >> our load-balanced pool of available search servers. >> This server's average query performance also degraded quickly, and then >> (unlike the first slave we replicated) did not improve. >> It stayed at around 30 secs per query. On the theory that this is a >> cache-warming issue, we added this server >> back to the pool in hopes that additional traffic would warm the cache. >> But >> what we saw was a quick spike of much worse >> performance (50 sec / query on average) followed by a slow/gradual decline >> in average response times. >> As of now (10 hours after the initial replication) this server is still >> reporting an average response time of ~2 seconds. >> This is much worse than before the optimize and is a counter-intuitive >> result. We expected an index 1/3 the size would be faster, not slower. >> >> On the theory that the index files needed to be loaded into the file >> system >> cache, I used the 'dd' command to copy >> the contents of the data/index directory to /dev/null, but that did not >> result in any noticeable performance improvement. >> >> At this point, things were not going as expected. We did not expect the >> replication after an optimize to result in such horrid >> performance. So we decided to let the last slave continue to serve stale >> results while we waited 4 hours for the >> other two slaves to approach some acceptable performance level. >> >> After the 4 hour break, we re-moved the 3rd and last slave server from our >> load-balancing pool, then re-enabled replication. >> This time we saw a tiny blip. The average performance went up to 1 second >> briefly then went back to the (normal for us) >> 0.25 to 0.5 second range. We then added this server back to the >> load-balancing pool and observed no degradation in performance.
Re: Inconsistent slave performance after optimize
Did you restart all of these slave servers? That would help. What garbage collection options do you use? Which release of Solr? How many Searchers are there in admin/stats.jsp? Searchers hold open all kinds of memory. They are supposed to cycle out. These are standard questions, but- what you are seeing is definitely not normal. Separately, if you want a regular optimization regime, there is a new option called 'maxSegments' to the optimize command. If you have solrconfig mergeFactor set to 10, then optimize 'maxSegments=8' will roll up the very smallest segments. This allows you to have a gradual optimization (and replication overhead) instead of big ones. Mason Hale wrote: Hello esteemed Solr community -- I'm observing some inconsistent performance on our slave servers after recently optimizing our master server. Our configuration is as follows: - all servers are hosted at Amazon EC2, running Ubuntu 8.04 - 1 master with heavy insert/update traffic, about 125K new documents per day (m1.large, ~8GB RAM) - autocommit every 1 minute - 3 slaves (m2.xlarge instance sizes, ~16GB RAM) - replicate every 5 minutes - we have configured autowarming queries for these machines - autowarmCount = 0 - Total index size is ~7M documents We were seeing increasing, but gradual performance degradation across all nodes. So we decided to try optimizing our index to improve performance. In preparation for the optimize we disabled replication polling on all slaves. We also turned off all workers that were writing to the index. Then we ran optimize on the master. The optimize took 45-60 minutes to complete, and the total size went from 68GB down to 23GB. We then enabled replication on each slave one at a time. The first slave we re-enabled took about 15 minutes to copy the new files. Once the files were copied the performance of slave plummeted. Average response time went from 0.75 sec to 45 seconds. Over the past 18 hours the average response time has gradually gown down to around 1.2 seconds now. Before re-enabling replication the second slave, we first removed it from our load-balanced pool of available search servers. This server's average query performance also degraded quickly, and then (unlike the first slave we replicated) did not improve. It stayed at around 30 secs per query. On the theory that this is a cache-warming issue, we added this server back to the pool in hopes that additional traffic would warm the cache. But what we saw was a quick spike of much worse performance (50 sec / query on average) followed by a slow/gradual decline in average response times. As of now (10 hours after the initial replication) this server is still reporting an average response time of ~2 seconds. This is much worse than before the optimize and is a counter-intuitive result. We expected an index 1/3 the size would be faster, not slower. On the theory that the index files needed to be loaded into the file system cache, I used the 'dd' command to copy the contents of the data/index directory to /dev/null, but that did not result in any noticeable performance improvement. At this point, things were not going as expected. We did not expect the replication after an optimize to result in such horrid performance. So we decided to let the last slave continue to serve stale results while we waited 4 hours for the other two slaves to approach some acceptable performance level. After the 4 hour break, we re-moved the 3rd and last slave server from our load-balancing pool, then re-enabled replication. This time we saw a tiny blip. The average performance went up to 1 second briefly then went back to the (normal for us) 0.25 to 0.5 second range. We then added this server back to the load-balancing pool and observed no degradation in performance. While we were happy to avoid a repeat of the poor performance we saw on the previous slaves, we are at a loss to explain why this slave did not also have such poor performance. At this point we're scratching our heads trying to understand: (a) Why the performance of the first two slaves was so terrible after the optimize. We think its cache-warming related, but we're not sure. > 10 hours seems like a long time to wait for the cache to warm up (b) Why the performance of the third slave was barely impacted. It should have hit the same cold-cache issues as the other servers, if that is indeed the root cause. (c) Why performance of the first 2 slaves is still much worse after the optimize than it was before the optimize, where the performance of the 3rd slave is pretty much unchanged. We expected the optimize to *improve* performance. All 3 slave servers are identically configured, and the procedure for re-enabling replication was identical for the 2nd and 3rd slaves, with the exception of a 4-hour wait period. We have confirmed that the 3rd slave did replicate, the number of documents and total index s
Inconsistent slave performance after optimize
Hello esteemed Solr community -- I'm observing some inconsistent performance on our slave servers after recently optimizing our master server. Our configuration is as follows: - all servers are hosted at Amazon EC2, running Ubuntu 8.04 - 1 master with heavy insert/update traffic, about 125K new documents per day (m1.large, ~8GB RAM) - autocommit every 1 minute - 3 slaves (m2.xlarge instance sizes, ~16GB RAM) - replicate every 5 minutes - we have configured autowarming queries for these machines - autowarmCount = 0 - Total index size is ~7M documents We were seeing increasing, but gradual performance degradation across all nodes. So we decided to try optimizing our index to improve performance. In preparation for the optimize we disabled replication polling on all slaves. We also turned off all workers that were writing to the index. Then we ran optimize on the master. The optimize took 45-60 minutes to complete, and the total size went from 68GB down to 23GB. We then enabled replication on each slave one at a time. The first slave we re-enabled took about 15 minutes to copy the new files. Once the files were copied the performance of slave plummeted. Average response time went from 0.75 sec to 45 seconds. Over the past 18 hours the average response time has gradually gown down to around 1.2 seconds now. Before re-enabling replication the second slave, we first removed it from our load-balanced pool of available search servers. This server's average query performance also degraded quickly, and then (unlike the first slave we replicated) did not improve. It stayed at around 30 secs per query. On the theory that this is a cache-warming issue, we added this server back to the pool in hopes that additional traffic would warm the cache. But what we saw was a quick spike of much worse performance (50 sec / query on average) followed by a slow/gradual decline in average response times. As of now (10 hours after the initial replication) this server is still reporting an average response time of ~2 seconds. This is much worse than before the optimize and is a counter-intuitive result. We expected an index 1/3 the size would be faster, not slower. On the theory that the index files needed to be loaded into the file system cache, I used the 'dd' command to copy the contents of the data/index directory to /dev/null, but that did not result in any noticeable performance improvement. At this point, things were not going as expected. We did not expect the replication after an optimize to result in such horrid performance. So we decided to let the last slave continue to serve stale results while we waited 4 hours for the other two slaves to approach some acceptable performance level. After the 4 hour break, we re-moved the 3rd and last slave server from our load-balancing pool, then re-enabled replication. This time we saw a tiny blip. The average performance went up to 1 second briefly then went back to the (normal for us) 0.25 to 0.5 second range. We then added this server back to the load-balancing pool and observed no degradation in performance. While we were happy to avoid a repeat of the poor performance we saw on the previous slaves, we are at a loss to explain why this slave did not also have such poor performance. At this point we're scratching our heads trying to understand: (a) Why the performance of the first two slaves was so terrible after the optimize. We think its cache-warming related, but we're not sure. > 10 hours seems like a long time to wait for the cache to warm up (b) Why the performance of the third slave was barely impacted. It should have hit the same cold-cache issues as the other servers, if that is indeed the root cause. (c) Why performance of the first 2 slaves is still much worse after the optimize than it was before the optimize, where the performance of the 3rd slave is pretty much unchanged. We expected the optimize to *improve* performance. All 3 slave servers are identically configured, and the procedure for re-enabling replication was identical for the 2nd and 3rd slaves, with the exception of a 4-hour wait period. We have confirmed that the 3rd slave did replicate, the number of documents and total index size matches the master and other slave servers. I'm writing to fish for an explanation or ideas that might explain this inconsistent performance. Obviously, we'd like to be able to reproduce the performance of the 3rd slave, and avoid the poor performance of the first two slaves the next time we decide it's time to optimize our index. thanks in advance, Mason
