Re: Solr in virtual host as opposed to /lib
I think you guys are talking about two different kinds of 'virtual hosts'. Lance is talking about CPU virtualization. Eric appears to be talking about apache virtual web hosts, although Eric hasn't told us how apache is involved in his setup in the first place, so it's unclear. Assuming you are using apache to reverse proxy to Solr, there is no reason I can think of that your front-end apache setup would effect CPU utilizaton by Solr, let alone by nutch. Eric Martin wrote: Oh. So I should take out the installations and move them to /some_dir as opposed to inside my virtual host of /home/my solr nutch is here/www ' -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: Sunday, October 31, 2010 7:26 PM To: solr-user@lucene.apache.org Subject: Re: Solr in virtual host as opposed to /lib With virtual hosting you can give CPU memory quotas to your different VMs. This allows you to control the Nutch v.s. The World problem. Unforch, you cannot allocate disk channel. With two i/o bound apps, this is a problem. On Sun, Oct 31, 2010 at 4:38 PM, Eric Martin e...@makethembite.com wrote: Excellent information. Thank you. Solr is acting just fine then. I can connect to it no issues, it indexes fine and there didn't seem to be any complication with it. Now I can rule it out and go about solving, what you pointed out, and I agree, to be a java/nutch issue. Nutch is a crawler I use to feed URL's into Solr for indexing. Nutch is open source and found on apache.org Thanks for your time. -Original Message- From: Jonathan Rochkind [mailto:rochk...@jhu.edu] Sent: Sunday, October 31, 2010 4:33 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib What servlet container are you putting your Solr in? Jetty? Tomcat? Something else? Are you fronting it with apache on top of that? (I think maybe you are, otherwise I'm not sure how the phrase 'virtual host' applies). In general, Solr of course doesn't care what directory it's in on disk, so long as the process running solr has the neccesary read/write permissions to the neccesary directories (and if it doesn't, you'd usually find out right away with an error message). And clients to Solr don't care what directory it's in on disk either, they only care that they can get it to it connecting to a certain port at a certain hostname. In general, if they can't get to it on a certain port at a certain hostname, that's something you'd discover right away, not something that would be intermittent. But I'm not familiar with nutch, you may want to try connecting to the port you have Solr running on (the hostname/port you have told nutch to find solr on?) yourself manually, and just make sure it is connectable. I can't think of any reason that what directory you have Solr in could cause CPU utilization issues. I think it's got nothing to do with that. I am not familar with nutch, if it's nutch that's taking 100% of your CPU, you might want to find some nutch experts to ask. Perhaps there's a nutch listserv? I am also not familiar with hadoop; you mention just in passing that you're using hadoop too, maybe that's an added complication, I don't know. One obvious reason nutch could be taking 100% cpu would be simply because you've asked it to do a lot of work quickly, and it's trying to. One reason I have seen Solr take 100% of CPU and become responsive, is when the Solr process gets caught up in terrible Java garbage collection. If that's what's happening, then giving the Solr JVM a higher maximum heap size can sometimes help (although confusingly, I've seen people suggest that if you give the Solr JVM too MUCH heap it can also result in long GC pauses), and if you have a multi-core/multi-CPU machine, I've found the JVM argument -XX:+UseConcMarkSweepGC to be very helpful. Other than that, it sounds to me like you've got a nutch/hadoop issue, not a Solr issue. From: Eric Martin [e...@makethembite.com] Sent: Sunday, October 31, 2010 7:16 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib Hi, Thank you. This is more than idle curiosity. I am trying to debug an issue I am having with my installation and this is one step in verifying that I have a setup that does not consume resources. I am trying to debunk my internal myth that having Solr nad Nutch in a virtual host would be causing these issues. Here is the main issue that involves Nutch/Solr and Drupal: /home/mootlaw/lib/solr /home/mootlaw/lib/nutch /home/mootlaw/www/Drupal site I'm running a 1333 FSB Dual Socket Xeon 5500 Series @ 2.4ghz, Enterprise Linux - x86_64 - OS, 12 Gig RAM. My Solr and Nutch are running. I am using jetty for my Solr. My server is not rooted. Nutch is using 100% of my cpus. I see this in my CPU utilization in my whm: /usr/bin/java -Xmx1000m -Dhadoop.log.dir=/home/mootlaw/lib/nutch/logs -Dhadoop.log.file=hadoop.log
RE: Solr in virtual host as opposed to /lib
I was speaking about apache virtual hosts. I was concerned that there was an increase processing time due to the solr and nutch instance being housed inside a virtual host as opposed to being dropped in root of my distro. Thank you for the astute clarification. -Original Message- From: Jonathan Rochkind [mailto:rochk...@jhu.edu] Sent: Monday, November 01, 2010 9:52 AM To: solr-user@lucene.apache.org Subject: Re: Solr in virtual host as opposed to /lib I think you guys are talking about two different kinds of 'virtual hosts'. Lance is talking about CPU virtualization. Eric appears to be talking about apache virtual web hosts, although Eric hasn't told us how apache is involved in his setup in the first place, so it's unclear. Assuming you are using apache to reverse proxy to Solr, there is no reason I can think of that your front-end apache setup would effect CPU utilizaton by Solr, let alone by nutch. Eric Martin wrote: Oh. So I should take out the installations and move them to /some_dir as opposed to inside my virtual host of /home/my solr nutch is here/www ' -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: Sunday, October 31, 2010 7:26 PM To: solr-user@lucene.apache.org Subject: Re: Solr in virtual host as opposed to /lib With virtual hosting you can give CPU memory quotas to your different VMs. This allows you to control the Nutch v.s. The World problem. Unforch, you cannot allocate disk channel. With two i/o bound apps, this is a problem. On Sun, Oct 31, 2010 at 4:38 PM, Eric Martin e...@makethembite.com wrote: Excellent information. Thank you. Solr is acting just fine then. I can connect to it no issues, it indexes fine and there didn't seem to be any complication with it. Now I can rule it out and go about solving, what you pointed out, and I agree, to be a java/nutch issue. Nutch is a crawler I use to feed URL's into Solr for indexing. Nutch is open source and found on apache.org Thanks for your time. -Original Message- From: Jonathan Rochkind [mailto:rochk...@jhu.edu] Sent: Sunday, October 31, 2010 4:33 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib What servlet container are you putting your Solr in? Jetty? Tomcat? Something else? Are you fronting it with apache on top of that? (I think maybe you are, otherwise I'm not sure how the phrase 'virtual host' applies). In general, Solr of course doesn't care what directory it's in on disk, so long as the process running solr has the neccesary read/write permissions to the neccesary directories (and if it doesn't, you'd usually find out right away with an error message). And clients to Solr don't care what directory it's in on disk either, they only care that they can get it to it connecting to a certain port at a certain hostname. In general, if they can't get to it on a certain port at a certain hostname, that's something you'd discover right away, not something that would be intermittent. But I'm not familiar with nutch, you may want to try connecting to the port you have Solr running on (the hostname/port you have told nutch to find solr on?) yourself manually, and just make sure it is connectable. I can't think of any reason that what directory you have Solr in could cause CPU utilization issues. I think it's got nothing to do with that. I am not familar with nutch, if it's nutch that's taking 100% of your CPU, you might want to find some nutch experts to ask. Perhaps there's a nutch listserv? I am also not familiar with hadoop; you mention just in passing that you're using hadoop too, maybe that's an added complication, I don't know. One obvious reason nutch could be taking 100% cpu would be simply because you've asked it to do a lot of work quickly, and it's trying to. One reason I have seen Solr take 100% of CPU and become responsive, is when the Solr process gets caught up in terrible Java garbage collection. If that's what's happening, then giving the Solr JVM a higher maximum heap size can sometimes help (although confusingly, I've seen people suggest that if you give the Solr JVM too MUCH heap it can also result in long GC pauses), and if you have a multi-core/multi-CPU machine, I've found the JVM argument -XX:+UseConcMarkSweepGC to be very helpful. Other than that, it sounds to me like you've got a nutch/hadoop issue, not a Solr issue. From: Eric Martin [e...@makethembite.com] Sent: Sunday, October 31, 2010 7:16 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib Hi, Thank you. This is more than idle curiosity. I am trying to debug an issue I am having with my installation and this is one step in verifying that I have a setup that does not consume resources. I am trying to debunk my internal myth that having Solr nad Nutch in a virtual host would be causing these issues.
Re: Solr in virtual host as opposed to /lib
: References: aanlktimvv5foc2b=gxo+xs1zwgps9o5t5jorwv3id...@mail.gmail.com : aanlktim30aat8s0nxq_8utxcokv8myyabz8wtxeyl...@mail.gmail.com : aanlktimpo9v_krgaxomd4hocqabibgzdhc+jhhgsq...@mail.gmail.com : aanlktimdvaawj7=b7=pgu+rzm+nobvzdfh4o39nkp...@mail.gmail.com : aanlktindzuwyjxwqqmtr5-rrp4gekvmj5vzzc_f0n...@mail.gmail.com : In-Reply-To: aanlktindzuwyjxwqqmtr5-rrp4gekvmj5vzzc_f0n...@mail.gmail.com : Subject: Solr in virtual host as opposed to /lib http://people.apache.org/~hossman/#threadhijack Thread Hijacking on Mailing Lists When starting a new discussion on a mailing list, please do not reply to an existing message, instead start a fresh email. Even if you change the subject line of your email, other mail headers still track which thread you replied to and your question is hidden in that thread and gets less attention. It makes following discussions in the mailing list archives particularly difficult. See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking -Hoss
RE: Solr in virtual host as opposed to /lib
I don't think you read the entire thread. I'm assuming you made a mistake. -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Monday, November 01, 2010 11:49 AM To: solr-user@lucene.apache.org Subject: Re: Solr in virtual host as opposed to /lib : References: aanlktimvv5foc2b=gxo+xs1zwgps9o5t5jorwv3id...@mail.gmail.com : aanlktim30aat8s0nxq_8utxcokv8myyabz8wtxeyl...@mail.gmail.com : aanlktimpo9v_krgaxomd4hocqabibgzdhc+jhhgsq...@mail.gmail.com : aanlktimdvaawj7=b7=pgu+rzm+nobvzdfh4o39nkp...@mail.gmail.com : aanlktindzuwyjxwqqmtr5-rrp4gekvmj5vzzc_f0n...@mail.gmail.com : In-Reply-To: aanlktindzuwyjxwqqmtr5-rrp4gekvmj5vzzc_f0n...@mail.gmail.com : Subject: Solr in virtual host as opposed to /lib http://people.apache.org/~hossman/#threadhijack Thread Hijacking on Mailing Lists When starting a new discussion on a mailing list, please do not reply to an existing message, instead start a fresh email. Even if you change the subject line of your email, other mail headers still track which thread you replied to and your question is hidden in that thread and gets less attention. It makes following discussions in the mailing list archives particularly difficult. See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking -Hoss
Re: Solr in virtual host as opposed to /lib
No, he didn't make a mistake but you did. Next time, please start a new thread not by conveniently replying to an existing thread and just changing the subject. Now we have two threads in thread. :) I don't think you read the entire thread. I'm assuming you made a mistake. -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Monday, November 01, 2010 11:49 AM To: solr-user@lucene.apache.org Subject: Re: Solr in virtual host as opposed to /lib : References: : aanlktimvv5foc2b=gxo+xs1zwgps9o5t5jorwv3id...@mail.gmail.com : : aanlktim30aat8s0nxq_8utxcokv8myyabz8wtxeyl...@mail.gmail.com : aanlktimpo9v_krgaxomd4hocqabibgzdhc+jhhgsq...@mail.gmail.com : aanlktimdvaawj7=b7=pgu+rzm+nobvzdfh4o39nkp...@mail.gmail.com : aanlktindzuwyjxwqqmtr5-rrp4gekvmj5vzzc_f0n...@mail.gmail.com : : In-Reply-To: aanlktindzuwyjxwqqmtr5-rrp4gekvmj5vzzc_f0n...@mail.gmail.com : Subject: Solr in virtual host as opposed to /lib http://people.apache.org/~hossman/#threadhijack Thread Hijacking on Mailing Lists When starting a new discussion on a mailing list, please do not reply to an existing message, instead start a fresh email. Even if you change the subject line of your email, other mail headers still track which thread you replied to and your question is hidden in that thread and gets less attention. It makes following discussions in the mailing list archives particularly difficult. See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking -Hoss
RE: Solr in virtual host as opposed to /lib
: I don't think you read the entire thread. I'm assuming you made a mistake. No mistake. When you sent your first message with the subject Solr in virtual host as opposed to /lib you did so in response to a completely unrelated thread (Searching with wrong keyboard layout or using translit) Please note the headers i quoted below documenting this, or consult any mailing list archive that displays full threads... http://markmail.org/thread/bjl23qcigp6w3kyl : : -Original Message- : From: Chris Hostetter [mailto:hossman_luc...@fucit.org] : Sent: Monday, November 01, 2010 11:49 AM : To: solr-user@lucene.apache.org : Subject: Re: Solr in virtual host as opposed to /lib : : : : References: aanlktimvv5foc2b=gxo+xs1zwgps9o5t5jorwv3id...@mail.gmail.com : : aanlktim30aat8s0nxq_8utxcokv8myyabz8wtxeyl...@mail.gmail.com : : aanlktimpo9v_krgaxomd4hocqabibgzdhc+jhhgsq...@mail.gmail.com : : aanlktimdvaawj7=b7=pgu+rzm+nobvzdfh4o39nkp...@mail.gmail.com : : aanlktindzuwyjxwqqmtr5-rrp4gekvmj5vzzc_f0n...@mail.gmail.com : : In-Reply-To: : aanlktindzuwyjxwqqmtr5-rrp4gekvmj5vzzc_f0n...@mail.gmail.com : : Subject: Solr in virtual host as opposed to /lib : : http://people.apache.org/~hossman/#threadhijack : Thread Hijacking on Mailing Lists : : When starting a new discussion on a mailing list, please do not reply to : an existing message, instead start a fresh email. Even if you change the : subject line of your email, other mail headers still track which thread : you replied to and your question is hidden in that thread and gets less : attention. It makes following discussions in the mailing list archives : particularly difficult. : See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking : : : : -Hoss : -Hoss
Re: Solr in virtual host as opposed to /lib
Can you expand on your question? Are you having a problem? Is this idle curiosity? Because I have no idea how to respond when there is so little information. Best Erick On Sun, Oct 31, 2010 at 5:32 PM, Eric Martin e...@makethembite.com wrote: Is there an issue running Solr in /home/lib as opposed to running it somewhere outside of the virtual hosts like /lib? Eric
RE: Solr in virtual host as opposed to /lib
Hi, Thank you. This is more than idle curiosity. I am trying to debug an issue I am having with my installation and this is one step in verifying that I have a setup that does not consume resources. I am trying to debunk my internal myth that having Solr nad Nutch in a virtual host would be causing these issues. Here is the main issue that involves Nutch/Solr and Drupal: /home/mootlaw/lib/solr /home/mootlaw/lib/nutch /home/mootlaw/www/Drupal site I'm running a 1333 FSB Dual Socket Xeon 5500 Series @ 2.4ghz, Enterprise Linux - x86_64 - OS, 12 Gig RAM. My Solr and Nutch are running. I am using jetty for my Solr. My server is not rooted. Nutch is using 100% of my cpus. I see this in my CPU utilization in my whm: /usr/bin/java -Xmx1000m -Dhadoop.log.dir=/home/mootlaw/lib/nutch/logs -Dhadoop.log.file=hadoop.log -Djava.library.path=/home/mootlaw/lib/nutch/lib/native/Linux-amd64-64 -classpath /home/mootlaw/lib/nutch/conf:/usr/lib/tools.jar:/home/mootlaw/lib/nutch/buil d:/home/mootlaw/lib/nutch/build/test/classes:/home/mootlaw/lib/nutch/build/n utch-1.2.job:/home/mootlaw/lib/nutch/nutch-*.job:/home/mootlaw/lib/nutch/lib /apache-solr-core-1.4.0.jar:/home/mootlaw/lib/nutch/lib/apache-solr-solrj-1. 4.0.jar:/home/mootlaw/lib/nutch/lib/commons-beanutils-1.8.0.jar:/home/mootla w/lib/nutch/lib/commons-cli-1.2.jar:/home/mootlaw/lib/nutch/lib/commons-code c-1.3.jar:/home/mootlaw/lib/nutch/lib/commons-collections-3.2.1.jar:/home/mo otlaw/lib/nutch/lib/commons-el-1.0.jar:/home/mootlaw/lib/nutch/lib/commons-h ttpclient-3.1.jar:/home/mootlaw/lib/nutch/lib/commons-io-1.4.jar:/home/mootl aw/lib/nutch/lib/commons-lang-2.1.jar:/home/mootlaw/lib/nutch/lib/commons-lo gging-1.0.4.jar:/home/mootlaw/lib/nutch/lib/commons-logging-api-1.0.4.jar:/h ome/mootlaw/lib/nutch/lib/commons-net-1.4.1.jar:/home/mootlaw/lib/nutch/lib/ core-3.1.1.jar:/home/mootlaw/lib/nutch/lib/geronimo-stax-api_1.0_spec-1.0.1. jar:/home/mootlaw/lib/nutch/lib/hadoop-0.20.2-core.jar:/home/mootlaw/lib/nut ch/lib/hadoop-0.20.2-tools.jar:/home/mootlaw/lib/nutch/lib/hsqldb-1.8.0.10.j ar:/home/mootlaw/lib/nutch/lib/icu4j-4_0_1.jar:/home/mootlaw/lib/nutch/lib/j akarta-oro-2.0.8.jar:/home/mootlaw/lib/nutch/lib/jasper-compiler-5.5.12.jar: /home/mootlaw/lib/nutch/lib/jasper-runtime-5.5.12.jar:/home/mootlaw/lib/nutc h/lib/jcl-over-slf4j-1.5.5.jar:/home/mootlaw/lib/nutch/lib/jets3t-0.6.1.jar: /home/mootlaw/lib/nutch/lib/jetty-6.1.14.jar:/home/mootlaw/lib/nutch/lib/jet ty-util-6.1.14.jar:/home/mootlaw/lib/nutch/lib/junit-3.8.1.jar:/home/mootlaw /lib/nutch/lib/kfs-0.2.2.jar:/home/mootlaw/lib/nutch/lib/log4j-1.2.15.jar:/h ome/mootlaw/lib/nutch/lib/lucene-core-3.0.1.jar:/home/mootlaw/lib/nutch/lib/ lucene-misc-3.0.1.jar:/home/mootlaw/lib/nutch/lib/oro-2.0.8.jar:/home/mootla w/lib/nutch/lib/resolver.jar:/home/mootlaw/lib/nutch/lib/serializer.jar:/hom e/mootlaw/lib/nutch/lib/servlet-api-2.5-6.1.14.jar:/home/mootlaw/lib/nutch/l ib/slf4j-api-1.5.5.jar:/home/mootlaw/lib/nutch/lib/slf4j-log4j12-1.4.3.jar:/ home/mootlaw/lib/nutch/lib/taglibs-i18n.jar:/home/mootlaw/lib/nutch/lib/tika -core-0.7.jar:/home/mootlaw/lib/nutch/lib/wstx-asl-3.2.7.jar:/home/mootlaw/l ib/nutch/lib/xercesImpl.jar:/home/mootlaw/lib/nutch/lib/xml-apis.jar:/home/m ootlaw/lib/nutch/lib/xmlenc-0.52.jar:/home/mootlaw/lib/nutch/lib/jsp-2.1/jsp -2.1.jar:/home/mootlaw/lib/nutch/lib/jsp-2.1/jsp-api-2.1.jar org.apache.nutch.fetcher.Fetcher /home/mootlaw/lib/nutch/crawl/segments/2010103113 -threads 50 My PIDS cannot be traced and my mem usage is at 5% My hadoop logs show: 2010-10-31 15:44:11,040 INFO fetcher.Fetcher - fetching http://caselaw.findlaw.com/us-5th-circuit/1454354.html 2010-10-31 15:44:11,294 INFO fetcher.Fetcher - fetching http://www.dallastxcriminaldefenseattorney.com/atom.xml 2010-10-31 15:44:11,337 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=48, fetchQueues.totalSize=2499 2010-10-31 15:44:12,339 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500 2010-10-31 15:44:13,341 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500 2010-10-31 15:44:14,344 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500 2010-10-31 15:44:15,346 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500 2010-10-31 15:44:16,349 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=50, fetchQueues.totalSize=2500 2010-10-31 15:44:16,568 INFO fetcher.Fetcher - fetching http://caselaw.findlaw.com/il-court-of-appeals/1542438.html 2010-10-31 15:44:17,308 INFO fetcher.Fetcher - fetching http://lcweb2.loc.gov/const/const.html 2010-10-31 15:44:17,352 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=49, fetchQueues.totalSize=2499 2010-10-31 15:44:18,354 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=49, fetchQueues.totalSize=2500 2010-10-31 15:44:19,356 INFO fetcher.Fetcher - -activeThreads=50, spinWaiting=49, fetchQueues.totalSize=2500 2010-10-31 15:44:20,358 INFO
RE: Solr in virtual host as opposed to /lib
What servlet container are you putting your Solr in? Jetty? Tomcat? Something else? Are you fronting it with apache on top of that? (I think maybe you are, otherwise I'm not sure how the phrase 'virtual host' applies). In general, Solr of course doesn't care what directory it's in on disk, so long as the process running solr has the neccesary read/write permissions to the neccesary directories (and if it doesn't, you'd usually find out right away with an error message). And clients to Solr don't care what directory it's in on disk either, they only care that they can get it to it connecting to a certain port at a certain hostname. In general, if they can't get to it on a certain port at a certain hostname, that's something you'd discover right away, not something that would be intermittent. But I'm not familiar with nutch, you may want to try connecting to the port you have Solr running on (the hostname/port you have told nutch to find solr on?) yourself manually, and just make sure it is connectable. I can't think of any reason that what directory you have Solr in could cause CPU utilization issues. I think it's got nothing to do with that. I am not familar with nutch, if it's nutch that's taking 100% of your CPU, you might want to find some nutch experts to ask. Perhaps there's a nutch listserv? I am also not familiar with hadoop; you mention just in passing that you're using hadoop too, maybe that's an added complication, I don't know. One obvious reason nutch could be taking 100% cpu would be simply because you've asked it to do a lot of work quickly, and it's trying to. One reason I have seen Solr take 100% of CPU and become responsive, is when the Solr process gets caught up in terrible Java garbage collection. If that's what's happening, then giving the Solr JVM a higher maximum heap size can sometimes help (although confusingly, I've seen people suggest that if you give the Solr JVM too MUCH heap it can also result in long GC pauses), and if you have a multi-core/multi-CPU machine, I've found the JVM argument -XX:+UseConcMarkSweepGC to be very helpful. Other than that, it sounds to me like you've got a nutch/hadoop issue, not a Solr issue. From: Eric Martin [e...@makethembite.com] Sent: Sunday, October 31, 2010 7:16 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib Hi, Thank you. This is more than idle curiosity. I am trying to debug an issue I am having with my installation and this is one step in verifying that I have a setup that does not consume resources. I am trying to debunk my internal myth that having Solr nad Nutch in a virtual host would be causing these issues. Here is the main issue that involves Nutch/Solr and Drupal: /home/mootlaw/lib/solr /home/mootlaw/lib/nutch /home/mootlaw/www/Drupal site I'm running a 1333 FSB Dual Socket Xeon 5500 Series @ 2.4ghz, Enterprise Linux - x86_64 - OS, 12 Gig RAM. My Solr and Nutch are running. I am using jetty for my Solr. My server is not rooted. Nutch is using 100% of my cpus. I see this in my CPU utilization in my whm: /usr/bin/java -Xmx1000m -Dhadoop.log.dir=/home/mootlaw/lib/nutch/logs -Dhadoop.log.file=hadoop.log -Djava.library.path=/home/mootlaw/lib/nutch/lib/native/Linux-amd64-64 -classpath /home/mootlaw/lib/nutch/conf:/usr/lib/tools.jar:/home/mootlaw/lib/nutch/buil d:/home/mootlaw/lib/nutch/build/test/classes:/home/mootlaw/lib/nutch/build/n utch-1.2.job:/home/mootlaw/lib/nutch/nutch-*.job:/home/mootlaw/lib/nutch/lib /apache-solr-core-1.4.0.jar:/home/mootlaw/lib/nutch/lib/apache-solr-solrj-1. 4.0.jar:/home/mootlaw/lib/nutch/lib/commons-beanutils-1.8.0.jar:/home/mootla w/lib/nutch/lib/commons-cli-1.2.jar:/home/mootlaw/lib/nutch/lib/commons-code c-1.3.jar:/home/mootlaw/lib/nutch/lib/commons-collections-3.2.1.jar:/home/mo otlaw/lib/nutch/lib/commons-el-1.0.jar:/home/mootlaw/lib/nutch/lib/commons-h ttpclient-3.1.jar:/home/mootlaw/lib/nutch/lib/commons-io-1.4.jar:/home/mootl aw/lib/nutch/lib/commons-lang-2.1.jar:/home/mootlaw/lib/nutch/lib/commons-lo gging-1.0.4.jar:/home/mootlaw/lib/nutch/lib/commons-logging-api-1.0.4.jar:/h ome/mootlaw/lib/nutch/lib/commons-net-1.4.1.jar:/home/mootlaw/lib/nutch/lib/ core-3.1.1.jar:/home/mootlaw/lib/nutch/lib/geronimo-stax-api_1.0_spec-1.0.1. jar:/home/mootlaw/lib/nutch/lib/hadoop-0.20.2-core.jar:/home/mootlaw/lib/nut ch/lib/hadoop-0.20.2-tools.jar:/home/mootlaw/lib/nutch/lib/hsqldb-1.8.0.10.j ar:/home/mootlaw/lib/nutch/lib/icu4j-4_0_1.jar:/home/mootlaw/lib/nutch/lib/j akarta-oro-2.0.8.jar:/home/mootlaw/lib/nutch/lib/jasper-compiler-5.5.12.jar: /home/mootlaw/lib/nutch/lib/jasper-runtime-5.5.12.jar:/home/mootlaw/lib/nutc h/lib/jcl-over-slf4j-1.5.5.jar:/home/mootlaw/lib/nutch/lib/jets3t-0.6.1.jar: /home/mootlaw/lib/nutch/lib/jetty-6.1.14.jar:/home/mootlaw/lib/nutch/lib/jet ty-util-6.1.14.jar:/home/mootlaw/lib/nutch/lib/junit-3.8.1.jar:/home/mootlaw
RE: Solr in virtual host as opposed to /lib
Excellent information. Thank you. Solr is acting just fine then. I can connect to it no issues, it indexes fine and there didn't seem to be any complication with it. Now I can rule it out and go about solving, what you pointed out, and I agree, to be a java/nutch issue. Nutch is a crawler I use to feed URL's into Solr for indexing. Nutch is open source and found on apache.org Thanks for your time. -Original Message- From: Jonathan Rochkind [mailto:rochk...@jhu.edu] Sent: Sunday, October 31, 2010 4:33 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib What servlet container are you putting your Solr in? Jetty? Tomcat? Something else? Are you fronting it with apache on top of that? (I think maybe you are, otherwise I'm not sure how the phrase 'virtual host' applies). In general, Solr of course doesn't care what directory it's in on disk, so long as the process running solr has the neccesary read/write permissions to the neccesary directories (and if it doesn't, you'd usually find out right away with an error message). And clients to Solr don't care what directory it's in on disk either, they only care that they can get it to it connecting to a certain port at a certain hostname. In general, if they can't get to it on a certain port at a certain hostname, that's something you'd discover right away, not something that would be intermittent. But I'm not familiar with nutch, you may want to try connecting to the port you have Solr running on (the hostname/port you have told nutch to find solr on?) yourself manually, and just make sure it is connectable. I can't think of any reason that what directory you have Solr in could cause CPU utilization issues. I think it's got nothing to do with that. I am not familar with nutch, if it's nutch that's taking 100% of your CPU, you might want to find some nutch experts to ask. Perhaps there's a nutch listserv? I am also not familiar with hadoop; you mention just in passing that you're using hadoop too, maybe that's an added complication, I don't know. One obvious reason nutch could be taking 100% cpu would be simply because you've asked it to do a lot of work quickly, and it's trying to. One reason I have seen Solr take 100% of CPU and become responsive, is when the Solr process gets caught up in terrible Java garbage collection. If that's what's happening, then giving the Solr JVM a higher maximum heap size can sometimes help (although confusingly, I've seen people suggest that if you give the Solr JVM too MUCH heap it can also result in long GC pauses), and if you have a multi-core/multi-CPU machine, I've found the JVM argument -XX:+UseConcMarkSweepGC to be very helpful. Other than that, it sounds to me like you've got a nutch/hadoop issue, not a Solr issue. From: Eric Martin [e...@makethembite.com] Sent: Sunday, October 31, 2010 7:16 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib Hi, Thank you. This is more than idle curiosity. I am trying to debug an issue I am having with my installation and this is one step in verifying that I have a setup that does not consume resources. I am trying to debunk my internal myth that having Solr nad Nutch in a virtual host would be causing these issues. Here is the main issue that involves Nutch/Solr and Drupal: /home/mootlaw/lib/solr /home/mootlaw/lib/nutch /home/mootlaw/www/Drupal site I'm running a 1333 FSB Dual Socket Xeon 5500 Series @ 2.4ghz, Enterprise Linux - x86_64 - OS, 12 Gig RAM. My Solr and Nutch are running. I am using jetty for my Solr. My server is not rooted. Nutch is using 100% of my cpus. I see this in my CPU utilization in my whm: /usr/bin/java -Xmx1000m -Dhadoop.log.dir=/home/mootlaw/lib/nutch/logs -Dhadoop.log.file=hadoop.log -Djava.library.path=/home/mootlaw/lib/nutch/lib/native/Linux-amd64-64 -classpath /home/mootlaw/lib/nutch/conf:/usr/lib/tools.jar:/home/mootlaw/lib/nutch/buil d:/home/mootlaw/lib/nutch/build/test/classes:/home/mootlaw/lib/nutch/build/n utch-1.2.job:/home/mootlaw/lib/nutch/nutch-*.job:/home/mootlaw/lib/nutch/lib /apache-solr-core-1.4.0.jar:/home/mootlaw/lib/nutch/lib/apache-solr-solrj-1. 4.0.jar:/home/mootlaw/lib/nutch/lib/commons-beanutils-1.8.0.jar:/home/mootla w/lib/nutch/lib/commons-cli-1.2.jar:/home/mootlaw/lib/nutch/lib/commons-code c-1.3.jar:/home/mootlaw/lib/nutch/lib/commons-collections-3.2.1.jar:/home/mo otlaw/lib/nutch/lib/commons-el-1.0.jar:/home/mootlaw/lib/nutch/lib/commons-h ttpclient-3.1.jar:/home/mootlaw/lib/nutch/lib/commons-io-1.4.jar:/home/mootl aw/lib/nutch/lib/commons-lang-2.1.jar:/home/mootlaw/lib/nutch/lib/commons-lo gging-1.0.4.jar:/home/mootlaw/lib/nutch/lib/commons-logging-api-1.0.4.jar:/h ome/mootlaw/lib/nutch/lib/commons-net-1.4.1.jar:/home/mootlaw/lib/nutch/lib/ core-3.1.1.jar:/home/mootlaw/lib/nutch/lib/geronimo-stax-api_1.0_spec-1.0.1.
Re: Solr in virtual host as opposed to /lib
With virtual hosting you can give CPU memory quotas to your different VMs. This allows you to control the Nutch v.s. The World problem. Unforch, you cannot allocate disk channel. With two i/o bound apps, this is a problem. On Sun, Oct 31, 2010 at 4:38 PM, Eric Martin e...@makethembite.com wrote: Excellent information. Thank you. Solr is acting just fine then. I can connect to it no issues, it indexes fine and there didn't seem to be any complication with it. Now I can rule it out and go about solving, what you pointed out, and I agree, to be a java/nutch issue. Nutch is a crawler I use to feed URL's into Solr for indexing. Nutch is open source and found on apache.org Thanks for your time. -Original Message- From: Jonathan Rochkind [mailto:rochk...@jhu.edu] Sent: Sunday, October 31, 2010 4:33 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib What servlet container are you putting your Solr in? Jetty? Tomcat? Something else? Are you fronting it with apache on top of that? (I think maybe you are, otherwise I'm not sure how the phrase 'virtual host' applies). In general, Solr of course doesn't care what directory it's in on disk, so long as the process running solr has the neccesary read/write permissions to the neccesary directories (and if it doesn't, you'd usually find out right away with an error message). And clients to Solr don't care what directory it's in on disk either, they only care that they can get it to it connecting to a certain port at a certain hostname. In general, if they can't get to it on a certain port at a certain hostname, that's something you'd discover right away, not something that would be intermittent. But I'm not familiar with nutch, you may want to try connecting to the port you have Solr running on (the hostname/port you have told nutch to find solr on?) yourself manually, and just make sure it is connectable. I can't think of any reason that what directory you have Solr in could cause CPU utilization issues. I think it's got nothing to do with that. I am not familar with nutch, if it's nutch that's taking 100% of your CPU, you might want to find some nutch experts to ask. Perhaps there's a nutch listserv? I am also not familiar with hadoop; you mention just in passing that you're using hadoop too, maybe that's an added complication, I don't know. One obvious reason nutch could be taking 100% cpu would be simply because you've asked it to do a lot of work quickly, and it's trying to. One reason I have seen Solr take 100% of CPU and become responsive, is when the Solr process gets caught up in terrible Java garbage collection. If that's what's happening, then giving the Solr JVM a higher maximum heap size can sometimes help (although confusingly, I've seen people suggest that if you give the Solr JVM too MUCH heap it can also result in long GC pauses), and if you have a multi-core/multi-CPU machine, I've found the JVM argument -XX:+UseConcMarkSweepGC to be very helpful. Other than that, it sounds to me like you've got a nutch/hadoop issue, not a Solr issue. From: Eric Martin [e...@makethembite.com] Sent: Sunday, October 31, 2010 7:16 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib Hi, Thank you. This is more than idle curiosity. I am trying to debug an issue I am having with my installation and this is one step in verifying that I have a setup that does not consume resources. I am trying to debunk my internal myth that having Solr nad Nutch in a virtual host would be causing these issues. Here is the main issue that involves Nutch/Solr and Drupal: /home/mootlaw/lib/solr /home/mootlaw/lib/nutch /home/mootlaw/www/Drupal site I'm running a 1333 FSB Dual Socket Xeon 5500 Series @ 2.4ghz, Enterprise Linux - x86_64 - OS, 12 Gig RAM. My Solr and Nutch are running. I am using jetty for my Solr. My server is not rooted. Nutch is using 100% of my cpus. I see this in my CPU utilization in my whm: /usr/bin/java -Xmx1000m -Dhadoop.log.dir=/home/mootlaw/lib/nutch/logs -Dhadoop.log.file=hadoop.log -Djava.library.path=/home/mootlaw/lib/nutch/lib/native/Linux-amd64-64 -classpath /home/mootlaw/lib/nutch/conf:/usr/lib/tools.jar:/home/mootlaw/lib/nutch/buil d:/home/mootlaw/lib/nutch/build/test/classes:/home/mootlaw/lib/nutch/build/n utch-1.2.job:/home/mootlaw/lib/nutch/nutch-*.job:/home/mootlaw/lib/nutch/lib /apache-solr-core-1.4.0.jar:/home/mootlaw/lib/nutch/lib/apache-solr-solrj-1. 4.0.jar:/home/mootlaw/lib/nutch/lib/commons-beanutils-1.8.0.jar:/home/mootla w/lib/nutch/lib/commons-cli-1.2.jar:/home/mootlaw/lib/nutch/lib/commons-code c-1.3.jar:/home/mootlaw/lib/nutch/lib/commons-collections-3.2.1.jar:/home/mo otlaw/lib/nutch/lib/commons-el-1.0.jar:/home/mootlaw/lib/nutch/lib/commons-h ttpclient-3.1.jar:/home/mootlaw/lib/nutch/lib/commons-io-1.4.jar:/home/mootl
RE: Solr in virtual host as opposed to /lib
Oh. So I should take out the installations and move them to /some_dir as opposed to inside my virtual host of /home/my solr nutch is here/www ' -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: Sunday, October 31, 2010 7:26 PM To: solr-user@lucene.apache.org Subject: Re: Solr in virtual host as opposed to /lib With virtual hosting you can give CPU memory quotas to your different VMs. This allows you to control the Nutch v.s. The World problem. Unforch, you cannot allocate disk channel. With two i/o bound apps, this is a problem. On Sun, Oct 31, 2010 at 4:38 PM, Eric Martin e...@makethembite.com wrote: Excellent information. Thank you. Solr is acting just fine then. I can connect to it no issues, it indexes fine and there didn't seem to be any complication with it. Now I can rule it out and go about solving, what you pointed out, and I agree, to be a java/nutch issue. Nutch is a crawler I use to feed URL's into Solr for indexing. Nutch is open source and found on apache.org Thanks for your time. -Original Message- From: Jonathan Rochkind [mailto:rochk...@jhu.edu] Sent: Sunday, October 31, 2010 4:33 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib What servlet container are you putting your Solr in? Jetty? Tomcat? Something else? Are you fronting it with apache on top of that? (I think maybe you are, otherwise I'm not sure how the phrase 'virtual host' applies). In general, Solr of course doesn't care what directory it's in on disk, so long as the process running solr has the neccesary read/write permissions to the neccesary directories (and if it doesn't, you'd usually find out right away with an error message). And clients to Solr don't care what directory it's in on disk either, they only care that they can get it to it connecting to a certain port at a certain hostname. In general, if they can't get to it on a certain port at a certain hostname, that's something you'd discover right away, not something that would be intermittent. But I'm not familiar with nutch, you may want to try connecting to the port you have Solr running on (the hostname/port you have told nutch to find solr on?) yourself manually, and just make sure it is connectable. I can't think of any reason that what directory you have Solr in could cause CPU utilization issues. I think it's got nothing to do with that. I am not familar with nutch, if it's nutch that's taking 100% of your CPU, you might want to find some nutch experts to ask. Perhaps there's a nutch listserv? I am also not familiar with hadoop; you mention just in passing that you're using hadoop too, maybe that's an added complication, I don't know. One obvious reason nutch could be taking 100% cpu would be simply because you've asked it to do a lot of work quickly, and it's trying to. One reason I have seen Solr take 100% of CPU and become responsive, is when the Solr process gets caught up in terrible Java garbage collection. If that's what's happening, then giving the Solr JVM a higher maximum heap size can sometimes help (although confusingly, I've seen people suggest that if you give the Solr JVM too MUCH heap it can also result in long GC pauses), and if you have a multi-core/multi-CPU machine, I've found the JVM argument -XX:+UseConcMarkSweepGC to be very helpful. Other than that, it sounds to me like you've got a nutch/hadoop issue, not a Solr issue. From: Eric Martin [e...@makethembite.com] Sent: Sunday, October 31, 2010 7:16 PM To: solr-user@lucene.apache.org Subject: RE: Solr in virtual host as opposed to /lib Hi, Thank you. This is more than idle curiosity. I am trying to debug an issue I am having with my installation and this is one step in verifying that I have a setup that does not consume resources. I am trying to debunk my internal myth that having Solr nad Nutch in a virtual host would be causing these issues. Here is the main issue that involves Nutch/Solr and Drupal: /home/mootlaw/lib/solr /home/mootlaw/lib/nutch /home/mootlaw/www/Drupal site I'm running a 1333 FSB Dual Socket Xeon 5500 Series @ 2.4ghz, Enterprise Linux - x86_64 - OS, 12 Gig RAM. My Solr and Nutch are running. I am using jetty for my Solr. My server is not rooted. Nutch is using 100% of my cpus. I see this in my CPU utilization in my whm: /usr/bin/java -Xmx1000m -Dhadoop.log.dir=/home/mootlaw/lib/nutch/logs -Dhadoop.log.file=hadoop.log -Djava.library.path=/home/mootlaw/lib/nutch/lib/native/Linux-amd64-64 -classpath /home/mootlaw/lib/nutch/conf:/usr/lib/tools.jar:/home/mootlaw/lib/nutch/buil d:/home/mootlaw/lib/nutch/build/test/classes:/home/mootlaw/lib/nutch/build/n utch-1.2.job:/home/mootlaw/lib/nutch/nutch-*.job:/home/mootlaw/lib/nutch/lib /apache-solr-core-1.4.0.jar:/home/mootlaw/lib/nutch/lib/apache-solr-solrj-1.
