Hi Konstantinos, OK, I was ablel to scope this one and I have a few questions for you. 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA? If so please scope this issue [0], the solution would be to upgrade if you are not too long ahead with ingestion as fixes in Solr are worth having based on recent release cycles. 2) How many cores do you have on Solr server? Also what kind of setUp do you have? Replication at all? In recent versions of Solr 4.X SolrJ clients should now call shutdown() on their SolrServer object to let it know they don't want to re-use any existing connections anymore, and when Solr internally uses SolrJ to talk to other nodes in SolrCloud it should be doing this (as of 4.0-ALPHA)so this is why I ask. Lewis
[0] https://issues.apache.org/jira/browse/SOLR-3280 On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis < [email protected]> wrote: > Hi, > I have setup OODT filemanager on port 9000, using SOLR as the indexing > service on port 8081. They are both setup on the same computer, while > crawler runs on a number of different compute nodes spread across the local > network and the cloud. > > When the crawler runs and ingests files I notice that there are several > connections that open to solr and remain in CLOSE_WAIT state for hours. > any idea why this happens? Moving forward I am planning to use several > hundreds of crawler instances, each running on different computer, that > will create thousands of such connections and will probably create problems > to the system. > Thanks in advance for any help > Kostas > > $lsof -i :8081 > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE > NAME > java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0 TCP > localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT) > java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0 TCP > localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT) > java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0 TCP > localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT) > java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0 TCP > localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT) > java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0 TCP > localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT) > java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0 TCP > localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT) > java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0 TCP > localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT) > java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0 TCP > localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT) > > > process 92065 is: > /usr/bin/java -Djava.ext.dirs=../lib > -Djava.util.logging.config.file=../etc/logging.properties > -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum 9000 > > ********************************************************* > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS > CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED > INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL > OR INDIVIDUALS NAMED ABOVE. > If the reader is not the intended recipient, or the > employee or agent responsible to deliver it to the > intended recipient, you are hereby notified that any > dissemination, distribution or copying of this > communication is strictly prohibited. If you have > received this communication in error, please reply to the > sender to notify us of the error and delete the original > message. Thank You. > -- *Lewis*
