Re: Master/Slave High CPU Usage
ok, we ran some tests and doing the commit for the slave as a post commit event of the master reloaded the index and allowed us to achieve a master slave configuration, without replication This is useful only if your master and slave are on the same machine, and it helps reducing the resources needed, as you don't have 2 indexes and you don't need to copy the data from one to the other. Thanks Lance for that proposal. On Sun, Nov 21, 2010 at 2:12 PM, Ofer Fort ofer...@gmail.com wrote: do i really need a commit? or can i use the *readercycle*http://wiki.apache.org/solr/SolrOperationsToolsscript? since i don't need to comit anything, just reopen the reader. thanks On Sun, Nov 21, 2010 at 12:17 PM, Lance Norskog goks...@gmail.com wrote: Yes, the Solr commit operations always reloads the index. And it always throws away the Solr caches: queryresult, document, filter query. If you do this, please post your results. On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote: OK, so to make sure i understand, even though the slave doesn't do any indexing, i will call commit and it will do nothing to the index itself, but will reload it? thanks On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com wrote: Ah! If the program doing the indexing has manual commits, the program could send a commit to the slave. If the indexer uses automatic commits, there is a trick: you can add a program as a postCommit event in solrconfig.xml. This can just be a shell script or a curl command that sends a commit to the slave Solr. Be sure to make all of the wait options false to this command; you don't want the master to block while the slave loads up the new index. Or, to control the maximum load on your server, you might actually want to make the master wait while the slave loads up Lance On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote: thanks Erick, but my question was regard the configuration Lance suggested, a configuration where i have two servers, set set logical master and slave, not as a true replication. Since both are running on the same machine, just have one only doing updates, and the other only queries, but both are using the same index files. Ofer On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.com wrote: The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again.
Re: Master/Slave High CPU Usage
You're welcome. On Tue, Nov 23, 2010 at 3:55 PM, Ofer Fort ofer...@gmail.com wrote: ok, we ran some tests and doing the commit for the slave as a post commit event of the master reloaded the index and allowed us to achieve a master slave configuration, without replication This is useful only if your master and slave are on the same machine, and it helps reducing the resources needed, as you don't have 2 indexes and you don't need to copy the data from one to the other. Thanks Lance for that proposal. On Sun, Nov 21, 2010 at 2:12 PM, Ofer Fort ofer...@gmail.com wrote: do i really need a commit? or can i use the *readercycle*http://wiki.apache.org/solr/SolrOperationsToolsscript? since i don't need to comit anything, just reopen the reader. thanks On Sun, Nov 21, 2010 at 12:17 PM, Lance Norskog goks...@gmail.com wrote: Yes, the Solr commit operations always reloads the index. And it always throws away the Solr caches: queryresult, document, filter query. If you do this, please post your results. On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote: OK, so to make sure i understand, even though the slave doesn't do any indexing, i will call commit and it will do nothing to the index itself, but will reload it? thanks On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com wrote: Ah! If the program doing the indexing has manual commits, the program could send a commit to the slave. If the indexer uses automatic commits, there is a trick: you can add a program as a postCommit event in solrconfig.xml. This can just be a shell script or a curl command that sends a commit to the slave Solr. Be sure to make all of the wait options false to this command; you don't want the master to block while the slave loads up the new index. Or, to control the maximum load on your server, you might actually want to make the master wait while the slave loads up Lance On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote: thanks Erick, but my question was regard the configuration Lance suggested, a configuration where i have two servers, set set logical master and slave, not as a true replication. Since both are running on the same machine, just have one only doing updates, and the other only queries, but both are using the same index files. Ofer On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.com wrote: The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two
Re: Master/Slave High CPU Usage
Yes, the Solr commit operations always reloads the index. And it always throws away the Solr caches: queryresult, document, filter query. If you do this, please post your results. On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote: OK, so to make sure i understand, even though the slave doesn't do any indexing, i will call commit and it will do nothing to the index itself, but will reload it? thanks On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com wrote: Ah! If the program doing the indexing has manual commits, the program could send a commit to the slave. If the indexer uses automatic commits, there is a trick: you can add a program as a postCommit event in solrconfig.xml. This can just be a shell script or a curl command that sends a commit to the slave Solr. Be sure to make all of the wait options false to this command; you don't want the master to block while the slave loads up the new index. Or, to control the maximum load on your server, you might actually want to make the master wait while the slave loads up Lance On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote: thanks Erick, but my question was regard the configuration Lance suggested, a configuration where i have two servers, set set logical master and slave, not as a true replication. Since both are running on the same machine, just have one only doing updates, and the other only queries, but both are using the same index files. Ofer On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.com wrote: The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com -- Lance Norskog goks...@gmail.com -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
ok, i'll try that and update the group thanks On Sun, Nov 21, 2010 at 12:17 PM, Lance Norskog goks...@gmail.com wrote: Yes, the Solr commit operations always reloads the index. And it always throws away the Solr caches: queryresult, document, filter query. If you do this, please post your results. On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote: OK, so to make sure i understand, even though the slave doesn't do any indexing, i will call commit and it will do nothing to the index itself, but will reload it? thanks On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com wrote: Ah! If the program doing the indexing has manual commits, the program could send a commit to the slave. If the indexer uses automatic commits, there is a trick: you can add a program as a postCommit event in solrconfig.xml. This can just be a shell script or a curl command that sends a commit to the slave Solr. Be sure to make all of the wait options false to this command; you don't want the master to block while the slave loads up the new index. Or, to control the maximum load on your server, you might actually want to make the master wait while the slave loads up Lance On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote: thanks Erick, but my question was regard the configuration Lance suggested, a configuration where i have two servers, set set logical master and slave, not as a true replication. Since both are running on the same machine, just have one only doing updates, and the other only queries, but both are using the same index files. Ofer On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.com wrote: The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com -- Lance Norskog goks...@gmail.com -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
do i really need a commit? or can i use the *readercycle*http://wiki.apache.org/solr/SolrOperationsToolsscript? since i don't need to comit anything, just reopen the reader. thanks On Sun, Nov 21, 2010 at 12:17 PM, Lance Norskog goks...@gmail.com wrote: Yes, the Solr commit operations always reloads the index. And it always throws away the Solr caches: queryresult, document, filter query. If you do this, please post your results. On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote: OK, so to make sure i understand, even though the slave doesn't do any indexing, i will call commit and it will do nothing to the index itself, but will reload it? thanks On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com wrote: Ah! If the program doing the indexing has manual commits, the program could send a commit to the slave. If the indexer uses automatic commits, there is a trick: you can add a program as a postCommit event in solrconfig.xml. This can just be a shell script or a curl command that sends a commit to the slave Solr. Be sure to make all of the wait options false to this command; you don't want the master to block while the slave loads up the new index. Or, to control the maximum load on your server, you might actually want to make the master wait while the slave loads up Lance On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote: thanks Erick, but my question was regard the configuration Lance suggested, a configuration where i have two servers, set set logical master and slave, not as a true replication. Since both are running on the same machine, just have one only doing updates, and the other only queries, but both are using the same index files. Ofer On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.com wrote: The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com -- Lance Norskog goks...@gmail.com -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
thanks Erick, but my question was regard the configuration Lance suggested, a configuration where i have two servers, set set logical master and slave, not as a true replication. Since both are running on the same machine, just have one only doing updates, and the other only queries, but both are using the same index files. Ofer On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.comwrote: The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
Ah! If the program doing the indexing has manual commits, the program could send a commit to the slave. If the indexer uses automatic commits, there is a trick: you can add a program as a postCommit event in solrconfig.xml. This can just be a shell script or a curl command that sends a commit to the slave Solr. Be sure to make all of the wait options false to this command; you don't want the master to block while the slave loads up the new index. Or, to control the maximum load on your server, you might actually want to make the master wait while the slave loads up Lance On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote: thanks Erick, but my question was regard the configuration Lance suggested, a configuration where i have two servers, set set logical master and slave, not as a true replication. Since both are running on the same machine, just have one only doing updates, and the other only queries, but both are using the same index files. Ofer On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.comwrote: The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
OK, so to make sure i understand, even though the slave doesn't do any indexing, i will call commit and it will do nothing to the index itself, but will reload it? thanks On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com wrote: Ah! If the program doing the indexing has manual commits, the program could send a commit to the slave. If the indexer uses automatic commits, there is a trick: you can add a program as a postCommit event in solrconfig.xml. This can just be a shell script or a curl command that sends a commit to the slave Solr. Be sure to make all of the wait options false to this command; you don't want the master to block while the slave loads up the new index. Or, to control the maximum load on your server, you might actually want to make the master wait while the slave loads up Lance On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote: thanks Erick, but my question was regard the configuration Lance suggested, a configuration where i have two servers, set set logical master and slave, not as a true replication. Since both are running on the same machine, just have one only doing updates, and the other only queries, but both are using the same index files. Ofer On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.com wrote: The slave polls. See: http://wiki.apache.org/solr/SolrReplication Best Erick On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote: Another question on that configuration, when the master commits, how does the slave knows that the index has changed? Does it check the index and finds out that it has a newer version? Thanks again for the help, Ofer ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
That sounds like a great option, and it will also free some storage space on that sever (now each index is about 130GB). Other than the lock policy (we use single), any other things to look out to? Thanks ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה: If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com
Re: Master/Slave High CPU Usage
If they are on the same server, you do not need to replicate. If you only do queries, the query server can use the same index directory as the master. Works quite well. Both have to have the same LockPolicy in solrconfig.xml. For security reasons, I would run the query server as a different user who has read-only access to the index; that way it cannot touch the index. On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote: anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high? -- Lance Norskog goks...@gmail.com
Master/Slave High CPU Usage
Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high?
Re: Master/Slave High CPU Usage
Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high?
Re: Master/Slave High CPU Usage
anybody? On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote: Hi, I'm working with Erez, we experienced this again, and this time the slave index folder didn't contain the index.XXX folder, only one index folder. if we shutdown the slave, the CPU on the master was normal, as soon as we started the slave again, the CPU went up to 100% again. thanks for any help ofer On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote: Hi all, We've been seeing this for the second time already. I have a solr (1.4.1) master and a slave. both are located on the same machine (16GB RAM, 4GB allocated to the slave and 3GB to the master) All our updates are going towards the master, and all the queries are towards the slave. Once in a while the slave gets OutOfMemoryError. This is not the big problem (i have a about 100M documents) The problem is that from that moment the CPU of the slave AND the master is almost 100%. If i shutdown the slave, the CPU of the master drops. If i start the slave again, the CPU is 100% again. I have the replication set on commit and startup. I see that in the data folder contains three index folders: index, index.XXXYYY and index.XXXYYY.ZZZ The only way i was able to get pass it (worked two times already), is to shutdown the two servers, and to copy all the index of the master to the slave, and start them again. From that moment and on, they continue to work and replicate with a very reasonable CPU usage. Our guess is that it failed to replicate due to the OOM and since then tries to do a full replication again and again? but why is the CPU of the master so high?
