Can someone log into jira and grant kostas perms? Sent from my iPhone
> On Jul 10, 2014, at 7:12 AM, "Konstantinos Mavrommatis" > <[email protected]> wrote: > > Unless I am missing something I am not able to create an issue in JIRA :( > Thanks > K > >> -----Original Message----- >> From: Ramirez, Paul M (398J) [mailto:[email protected]] >> Sent: Thursday, July 10, 2014 4:06 PM >> To: <[email protected]> >> Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to >> SOLR in CLOSE_WAIT state for hours >> >> Konstantinos, >> >> Could you open a Jira issue for this and attach a patch? This would be >> helpful for the project. >> >> https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.j >> ira.jira-projects-plugin:summary-panel >> >> >> Thanks, >> Paul Ramirez >> >>>> On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis" >>> <[email protected]> wrote: >>> >>> Hi, >>> >>> Following the suggestion at >>> >>> http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly- >> a >>> void-closewait-tcp-connections/ >>> >>> >>> >>> I modified the code in >> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java >>> >>> >>> >>> and added the line highlighted in red >> (method.setRequestHeader("Connection","close"). >>> >>> This seems to have resolved the problem at least until now ingestion >> of few thousand files have not produced any stale connection. >>> >>> Please let me know your thoughts on this solution. >>> >>> Best, >>> >>> Konstantinos >>> >>> >>> >>> ====================================================== >>> >>> private String doHttp(HttpMethod method) throws Exception { >>> >>> >>> >>> StringBuilder response = new StringBuilder(); >>> >>> BufferedReader br = null; >>> >>> try { >>> >>> >>> >>> // send request >>> >>> HttpClient httpClient = new HttpClient(); >>> >>> // 10 July 2014. attempting to avoid problems >>> with CLOSE_WAIT connections Based on >>> >>> >>> //http://blogs.nuxeo.com/development/2013/02/using-httpclient- >> properly >>> -avoid-closewait-tcp-connections/ >>> >>> method.setRequestHeader("Connection", >> "close"); >>> >>> int statusCode = httpClient.executeMethod(method); >>> >>> >>> >>> // read response >>> >>> if (statusCode != HttpStatus.SC_OK) { >>> >>> >>> >>> // still consume the response >>> >>> method.getResponseBodyAsString(); >>> >>> throw new CatalogException("HTTP method failed: " + >>> method.getStatusLine()); >>> >>> >>> >>> } else { >>> >>> >>> >>> // read the response body. >>> >>> br = new BufferedReader(new >>> InputStreamReader(method.getResponseBodyAsStream())); >>> >>> String readLine; >>> >>> while(((readLine = br.readLine()) != null)) { >>> >>> response.append(readLine); >>> >>> } >>> >>> >>> >>> ======================================================= >>> >>> >>> >>> Information about the system: >>> >>> OS is MacOS Mavericks, >>> >>> SOLR version : 4.6.1 , SOLR runs on a single server, no replication >> at all. >>> >>> Tomcat verision: 7.0.50 >>> >>> OODT version is 0.6 >>> >>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13) >>> >>> >>> >>> >>> >>> >>> >>>> -----Original Message----- >>> >>>> From: Lewis John Mcgibbney [mailto:[email protected]] >>> >>>> Sent: Wednesday, July 09, 2014 4:57 AM >>> >>>> To: [email protected] >>> >>>> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT >>>> state >>> >>>> for hours >>> >>> >>>> It's just occoured to me that everything in >>> >>>> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well, >>> >>>> specifically >>> >>>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through >>> >>>> HTTPClient so all of the above may not be relevant. >>> >>>> Can you please scope all the same? >>> >>>> Thanks >>> >>>> Lewis >>> >>> >>> >>>> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney < >>> >>>> [email protected]<mailto:[email protected]>> wrote: >>> >>> >>>>> Hi Konstantinos, >>> >>>>> OK, I was ablel to scope this one and I have a few questions for >> you. >>> >>>>> 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA? >>> >>>> If >>> >>>>> so please scope this issue [0], the solution would be to upgrade if >>> >>>>> you are not too long ahead with ingestion as fixes in Solr are >> worth >>> >>>>> having based on recent release cycles. >>> >>>>> 2) How many cores do you have on Solr server? Also what kind of >>>>> setUp >>> >>>>> do you have? Replication at all? >>> >>>>> In recent versions of Solr 4.X SolrJ clients should now call >>> >>>>> shutdown() on their SolrServer object to let it know they don't >> want >>> >>>>> to re-use any existing connections anymore, and when Solr >> internally >>> >>>>> uses SolrJ to talk to other nodes in SolrCloud it should be doing >>> >>>> this >>> >>>>> (as of 4.0-ALPHA)so this is why I ask. >>> >>>>> Lewis >>> >>> >>>>> [0] https://issues.apache.org/jira/browse/SOLR-3280 >>> >>> >>> >>>>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis < >>> >>>>> [email protected]<mailto:[email protected]>> wrote: >>> >>> >>>>>> Hi, >>> >>>>>> I have setup OODT filemanager on port 9000, using SOLR as the >>> >>>>>> indexing service on port 8081. They are both setup on the same >>> >>>>>> computer, while crawler runs on a number of different compute >> nodes >>> >>>>>> spread across the local network and the cloud. >>> >>> >>>>>> When the crawler runs and ingests files I notice that there are >>> >>>>>> several connections that open to solr and remain in CLOSE_WAIT >>>>>> state >>> >>>> for hours. >>> >>>>>> any idea why this happens? Moving forward I am planning to use >>> >>>>>> several hundreds of crawler instances, each running on different >>> >>>>>> computer, that will create thousands of such connections and will >>> >>>>>> probably create problems to the system. >>> >>>>>> Thanks in advance for any help >>> >>>>>> Kostas >>> >>> >>>>>> $lsof -i :8081 >>> >>>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF >>> >>>> NODE >>> >>>>>> NAME >>> >>>>>> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0 >>> >>>> TCP >>> >>>>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT) >>> >>>>>> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0 >>> >>>> TCP >>> >>>>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT) >>> >>>>>> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0 >>> >>>> TCP >>> >>>>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT) >>> >>>>>> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0 >>> >>>> TCP >>> >>>>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT) >>> >>>>>> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0 >>> >>>> TCP >>> >>>>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT) >>> >>>>>> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0 >>> >>>> TCP >>> >>>>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT) >>> >>>>>> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0 >>> >>>> TCP >>> >>>>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT) >>> >>>>>> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0 >>> >>>> TCP >>> >>>>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT) >>> >>> >>> >>>>>> process 92065 is: >>> >>>>>> /usr/bin/java -Djava.ext.dirs=../lib >>> >>>>>> -Djava.util.logging.config.file=../etc/logging.properties >>> >>>>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties >>> >>>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum >> 9000 >>> >>> >>>>>> ********************************************************* >>> >>>>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL >> AND >>> >>>>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE >>>>>> USE >>> >>>>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE. >>> >>>>>> If the reader is not the intended recipient, or the employee or >>> >>>> agent >>> >>>>>> responsible to deliver it to the intended recipient, you are >> hereby >>> >>>>>> notified that any dissemination, distribution or copying of this >>> >>>>>> communication is strictly prohibited. If you have received this >>> >>>>>> communication in error, please reply to the sender to notify us of >>> >>>>>> the error and delete the original message. Thank You. >>> >>> >>> >>> >>> >>>>> -- >>> >>>>> *Lewis* >>> >>> >>> >>> >>> >>>> -- >>> >>>> *Lewis* >>> >>> ********************************************************* >>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND >>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE >>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE. >>> If the reader is not the intended recipient, or the employee or agent >>> responsible to deliver it to the intended recipient, you are hereby >>> notified that any dissemination, distribution or copying of this >>> communication is strictly prohibited. If you have received this >>> communication in error, please reply to the sender to notify us of >> the >>> error and delete the original message. Thank You. > > ********************************************************* > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS > CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED > INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL > OR INDIVIDUALS NAMED ABOVE. > If the reader is not the intended recipient, or the > employee or agent responsible to deliver it to the > intended recipient, you are hereby notified that any > dissemination, distribution or copying of this > communication is strictly prohibited. If you have > received this communication in error, please reply to the > sender to notify us of the error and delete the original > message. Thank You.
