Hi Konstantinos and others, I've logged an issue on Jira where you can track this. https://issues.apache.org/jira/browse/OODT-719
On Wed, Jul 9, 2014 at 5:21 AM, Konstantinos Mavrommatis < [email protected]> wrote: > Hi Lewis, > Thank you for your quick response: > Information about the system: > OS is MacOS Mavericks, > the SOLR version is 4.6.1 running on Tomcat 7.0.50, > SOLR runs on a single server, no replication at all. > OODT version is 0.6 > > The interesting thing is that after a day of 'moderate' activity I have now > 198 CLOSE_WAIT instances on port 8081 (which is where SOLR is listening) > with corresponding instances in state FIN_WAIT_2 from the client side (i.e. > filemanager) > I also have another 104 CLOSE_WAIT instances on port 9000, which is where > filemanager listens to for connections from the compute nodes of the > cluster. > e.g. > java 92065 kmavrommatis 363u IPv6 0x392c3fa39f6adf0f 0t0 TCP > 10.130.0.26:cslistener->cluster:51194 (CLOSE_WAIT) > java 92065 kmavrommatis 364u IPv6 0x392c3fa3b62759cf 0t0 TCP > 10.130.0.26:cslistener->cluster:33123 (CLOSE_WAIT) > java 92065 kmavrommatis 365u IPv6 0x392c3fa3b6d3314f 0t0 TCP > 10.130.0.26:cslistener->cluster:35167 (CLOSE_WAIT) > java 92065 kmavrommatis 366u IPv6 0x392c3fa3b5faff0f 0t0 TCP > 10.130.0.26:cslistener->cluster:46788 (CLOSE_WAIT) > java 92065 kmavrommatis 367u IPv6 0x392c3fa39f6abd0f 0t0 TCP > 10.130.0.26:cslistener->cluster:52658 (CLOSE_WAIT) > java 92065 kmavrommatis 368u IPv6 0x392c3fa3b6d15e0f 0t0 TCP > 10.130.0.26:cslistener->cluster:45768 (CLOSE_WAIT) > java 92065 kmavrommatis 369u IPv6 0x392c3fa3b60fa58f 0t0 TCP > 10.130.0.26:cslistener->cluster:58830 (CLOSE_WAIT) > java 92065 kmavrommatis 370u IPv6 0x392c3fa39fd849cf 0t0 TCP > 10.130.0.26:cslistener->cluster:59112 (CLOSE_WAIT) > > > I am not sure how to debug this problem, my TCP stack knowledge is > limited, any ideas? > Thanks > Konstantinos > > > -----Original Message----- > > From: Lewis John Mcgibbney [mailto:[email protected]] > > Sent: Wednesday, July 09, 2014 4:55 AM > > To: [email protected] > > Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT state > > for hours > > > > Hi Konstantinos, > > OK, I was ablel to scope this one and I have a few questions for you. > > 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA? If > > so please scope this issue [0], the solution would be to upgrade if you > > are not too long ahead with ingestion as fixes in Solr are worth having > > based on recent release cycles. > > 2) How many cores do you have on Solr server? Also what kind of setUp > > do you have? Replication at all? > > In recent versions of Solr 4.X SolrJ clients should now call shutdown() > > on their SolrServer object to let it know they don't want to re-use any > > existing connections anymore, and when Solr internally uses SolrJ to > > talk to other nodes in SolrCloud it should be doing this (as of 4.0- > > ALPHA)so this is why I ask. > > Lewis > > > > [0] https://issues.apache.org/jira/browse/SOLR-3280 > > > > > > On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis < > > [email protected]> wrote: > > > > > Hi, > > > I have setup OODT filemanager on port 9000, using SOLR as the > > indexing > > > service on port 8081. They are both setup on the same computer, while > > > crawler runs on a number of different compute nodes spread across the > > > local network and the cloud. > > > > > > When the crawler runs and ingests files I notice that there are > > > several connections that open to solr and remain in CLOSE_WAIT state > > for hours. > > > any idea why this happens? Moving forward I am planning to use > > several > > > hundreds of crawler instances, each running on different computer, > > > that will create thousands of such connections and will probably > > > create problems to the system. > > > Thanks in advance for any help > > > Kostas > > > > > > $lsof -i :8081 > > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF > > NODE > > > NAME > > > java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0 > > TCP > > > localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT) > > > java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0 > > TCP > > > localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT) > > > java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0 > > TCP > > > localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT) > > > java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0 > > TCP > > > localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT) > > > java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0 > > TCP > > > localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT) > > > java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0 > > TCP > > > localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT) > > > java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0 > > TCP > > > localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT) > > > java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0 > > TCP > > > localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT) > > > > > > > > > process 92065 is: > > > /usr/bin/java -Djava.ext.dirs=../lib > > > -Djava.util.logging.config.file=../etc/logging.properties > > > -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties > > > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum 9000 > > > > > > ********************************************************* > > > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND > > > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE > > > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE. > > > If the reader is not the intended recipient, or the employee or agent > > > responsible to deliver it to the intended recipient, you are hereby > > > notified that any dissemination, distribution or copying of this > > > communication is strictly prohibited. If you have received this > > > communication in error, please reply to the sender to notify us of > > the > > > error and delete the original message. Thank You. > > > > > > > > > > > -- > > *Lewis* > > ********************************************************* > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS > CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED > INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL > OR INDIVIDUALS NAMED ABOVE. > If the reader is not the intended recipient, or the > employee or agent responsible to deliver it to the > intended recipient, you are hereby notified that any > dissemination, distribution or copying of this > communication is strictly prohibited. If you have > received this communication in error, please reply to the > sender to notify us of the error and delete the original > message. Thank You. > -- *Lewis*
