It was a pleasure to help :-) Siegfried Goeschl
> On 06 Feb 2016, at 03:53, Chris Gamache <[email protected]> wrote: > > I saw that! I couldn't have done it without you. Many thanks! > > CG > > Sent from my iPhone > >> On Feb 5, 2016, at 12:24 PM, Siegfried Goeschl <[email protected]> >> wrote: >> >> Hi Chris, >> >> according to https://github.com/SeleniumHQ/selenium/pull/1545 the PR is >> merged and closed :-) >> >> Cheers, >> >> Siegfried Goeschl >> >>> On 27.01.2016, at 14:52, Siegfried Göschl <[email protected]> >>> wrote: >>> >>> Hi Chris, >>> >>> glad to help - please note that I took the freedom to post my findings at >>> https://github.com/SeleniumHQ/selenium/issues/1080 - could you do the same >>> :-) >>> >>> Cheers, >>> >>> Siegfried Goeschl >>> >>> >>> ----- Ursprüngliche Mail ----- >>> Von: "Chris Gamache" <[email protected]> >>> An: "Commons Users List" <[email protected]> >>> Gesendet: Mittwoch, 27. Januar 2016 14:43:32 >>> Betreff: Re: Help needed: commons-exec CLOSE_WAIT problem >>> >>> I really appreciate the thorough inspection you have given. With your help >>> I think I've targeted the real problem... >>> >>> GhostDriver can't exist outside of phantomjs. GhostDriver is an >>> implementation of WebDriver Wire Protocol which runs on phantomjs's >>> JavaScript engine. It can't run without phantomjs. >>> >>> We can rest easy that commons-exec is not holding open the file handle. >>> That's good. I think the PhantomJSDriver is firing a copy of phantomjs up >>> using commons-exec, but it's not communicating with the service through >>> commons-exec. Upon closer inspection selenium-java is communicating through >>> tcp/ip to a WebDriver server phantomjs is popping up on random available >>> ports. This happens to be an instance of some class that implements >>> org.apache.http.client.HttpClient, by default it looks to >>> be org.apache.http.impl.client.CloseableHttpClient ... I think that when >>> commons-exec terminates phantomjs, thereby terminating the GhostDriver >>> server, the HttpClient is still attached to that port. >>> >>> Here's my take on the real problem: >>> >>> I think the selenium folks didn't consider port closing >>> (org.apache.http.client.HttpClient doesn't have a close method to >>> implement) a necessary step and they figured that if whatever service the >>> HttpClient was attached to shut itself down that the port would >>> automatically close itself down also. The standard use for selenium usually >>> doesn't require it to run for days and days on end. I can't fault them for >>> missing it. >>> >>> That being said, they made this difficult to override in consumer classes >>> by keeping a lot of the guts of the connections hidden behind private >>> classes and private member data. >>> >>> So, I'll need to dig into selenium-java, change a few interfaces to make >>> sure that whatever HttpClient is used will have a close() hook, and then >>> make sure that whatever is driving the HttpClient has access to the close >>> hook on shutdown, and fires it. When the HttpClient is closed, the file >>> descriptor should disappear. >>> >>> Thank you again for your help. It was indispensable! >>> >>> CG >>> >>> >>> On Tue, Jan 26, 2016 at 4:07 PM, Siegfried Göschl < >>> [email protected]> wrote: >>> >>>> Hi Chris, >>>> >>>> I played with your GitHub repo [ >>>> https://github.com/cgamache/openfile](https://github.com/cgamache/openfile) >>>> >>>> ### 1. First Take >>>> >>>> Your test program actually starts a JVM and the Selenium library start >>>> multiple "phantomjs" executable to run the test as shown below >>>> >>>> ``` >>>> application> ps -ef | grep phantom >>>> 501 1021 460 0 9:26PM ttys000 0:00.71 java -jar openfile.jar >>>> /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs >>>> 501 1073 1062 0 9:26PM ttys000 0:00.08 >>>> /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs --webdriver=36418 >>>> --webdriver-logfile=/Users/sgoeschl/work/github/cgamache/openfile/target/phantomjsdriver.log >>>> ``` >>>> >>>> So commons-exec is used under the hood to run the "phantomjs" binary and >>>> make sure nothing hangs (output & error thread pumping, watchdog, ..) >>>> >>>> When the "WebDriver" has finished its work there >>>> >>>> * is no ***phantomjs*** process running >>>> * is only your program running as shown below >>>> >>>> ``` >>>> application> ps -ef | grep phantom >>>> 501 1021 460 0 9:23PM ttys000 0:00.72 java -jar openfile.jar >>>> /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs >>>> 501 1034 469 0 9:23PM ttys001 0:00.00 grep phantom >>>> ``` >>>> >>>> And as described we have a couple of ports in CLOSE_WAIT stemming from >>>> your program >>>> >>>> ``` >>>> application> lsof -p 1021 | grep CLOSE_WAIT >>>> java 1021 sgoeschl 16u IPv6 0x78418468470e25c7 0t0 TCP >>>> localhost:50581->localhost:40560 (CLOSE_WAIT) >>>> java 1021 sgoeschl 18u IPv6 0x784184685efd95a7 0t0 TCP >>>> localhost:50599->localhost:45319 (CLOSE_WAIT) >>>> java 1021 sgoeschl 19u IPv6 0x784184685efdc0a7 0t0 TCP >>>> localhost:50620->localhost:17950 (CLOSE_WAIT) >>>> java 1021 sgoeschl 20u IPv6 0x78418468470e3087 0t0 TCP >>>> localhost:50641->localhost:41916 (CLOSE_WAIT) >>>> java 1021 sgoeschl 21u IPv6 0x78418468470e4b67 0t0 TCP >>>> localhost:50661->localhost:25113 (CLOSE_WAIT) >>>> ``` >>>> >>>> As far as common-exec is concerned everything is fine - the >>>> ***phantomjs*** processes where nicely executed and no process is left >>>> >>>> So you have 5 ports in limbo which exactly corresponds to the number of >>>> created ***phantom.js*** instances >>>> >>>> ``` >>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs >>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs >>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs >>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs >>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs >>>> ``` >>>> >>>> This nicely corresponds to your test code >>>> >>>> ``` >>>> public static final int REPS = 5; >>>> >>>> for (; i < REPS; i++) { >>>> WebDriver driver = new PhantomJSDriver(caps); >>>> driver.get("http://www.apache.org";); >>>> driver.quit(); >>>> } >>>> ``` >>>> >>>> In other words each ***phantomjs*** process leaves exactly on port behind >>>> :-) >>>> >>>> >>>> ### 2. Second Take >>>> >>>> So where is this port coming from? The next session - starting the test >>>> programm leaves the 5 ports behind >>>> >>>> ``` >>>> application> lsof -p 1329 | grep CLOSE_WAIT >>>> java 1329 sgoeschl 16u IPv6 0x784184686037bb07 0t0 TCP >>>> localhost:51358->localhost:30177 (CLOSE_WAIT) >>>> java 1329 sgoeschl 18u IPv6 0x784184685f5c3b47 0t0 TCP >>>> localhost:51379->localhost:cadkey-tablet (CLOSE_WAIT) >>>> java 1329 sgoeschl 19u IPv6 0x784184686037f0c7 0t0 TCP >>>> localhost:51400->localhost:43648 (CLOSE_WAIT) >>>> java 1329 sgoeschl 20u IPv6 0x784184686037c5c7 0t0 TCP >>>> localhost:51418->localhost:34505 (CLOSE_WAIT) >>>> java 1329 sgoeschl 21u IPv6 0x78418468470e1047 0t0 TCP >>>> localhost:51438->localhost:30905 (CLOSE_WAIT) >>>> ``` >>>> >>>> All of ports are actually found in phantomjsdriver.log (cadkey-tablet is >>>> actually port 1400) >>>> >>>> ``` >>>> [INFO - 2016-01-26T20:51:19.697Z] GhostDriver - Main - running on port >>>> 30177 >>>> [INFO - 2016-01-26T20:51:21.685Z] GhostDriver - Main - running on port >>>> 1400 >>>> [INFO - 2016-01-26T20:51:23.427Z] GhostDriver - Main - running on port >>>> 43648 >>>> [INFO - 2016-01-26T20:51:25.156Z] GhostDriver - Main - running on port >>>> 34505 >>>> [INFO - 2016-01-26T20:51:26.887Z] GhostDriver - Main - running on port >>>> 30905 >>>> ``` >>>> >>>> So the ports are actually used/opened for ***GhostDriver*** and left there >>>> even when ***driver.quit()*** was called >>>> >>>> >>>> ### 3. Conclusion >>>> >>>> * ASFAIK it is not commons-exec issue since all ***phantomjs*** were >>>> cleanly terminated >>>> * Each ***PhantomJSDriver*** invocation leaves a port in ***CLOSE_WAIT*** >>>> behind >>>> * Those ports are linked to ***GhostDriver - Main - running on port >>>> 30177*** >>>> >>>> >>>> Cheers, >>>> >>>> Siegfried Goeschl >>>> >>>> >>>> >>>> ----- Ursprüngliche Mail ----- >>>> Von: "Chris Gamache" <[email protected]> >>>> An: "Commons Users List" <[email protected]> >>>> Gesendet: Dienstag, 26. Januar 2016 04:02:41 >>>> Betreff: Re: Help needed: commons-exec CLOSE_WAIT problem >>>> >>>> Most defnintely. This was tested on OSX, but should work properly on Linux >>>> also. You'll need to install phantomjs. >>>> >>>> You can clone this repo (https://github.com/cgamache/openfile) and build >>>> with Maven. The com.codeborne:phantomjsdriver artifact has all of selenium >>>> packaged inside. >>>> >>>> To run, >>>> >>>> $ java -jar openfile.jar /path/to/phantomjs >>>> >>>> To examine, >>>> >>>> $ lsof -p 38270 | grep CLOSE_WAIT | wc -l >>>> >>>> Jan 25, 2016 9:46:17 PM >>>> org.openqa.selenium.phantomjs.PhantomJSDriverService <init> >>>> >>>> ....... super chatty........ >>>> >>>> [INFO - 2016-01-26T02:46:41.169Z] ShutdownReqHand - _handle - About to >>>> shutdown >>>> >>>> >>>> Sleeping for 30 seconds so you can examine file handles... you should see 5 >>>> >>>> Then you can use the lsof command at the top of the output, complete with >>>> process ID, to check the CLOSE_WAIT file handles. >>>> >>>> Thanks for taking a look! >>>> >>>> CG >>>> >>>> >>>> >>>> On Mon, Jan 25, 2016 at 5:28 PM, Siegfried Göschl < >>>> [email protected]> wrote: >>>> >>>>> Hi Chris, >>>>> >>>>> there could be couple of reasons for this behaviour - is there a minimal >>>>> setup to reproduce the problem? >>>>> >>>>> Thanks in advance >>>>> >>>>> Siegfried Goeschl >>>>> >>>>> ----- Ursprüngliche Mail ----- >>>>> Von: "Chris Gamache" <[email protected]> >>>>> An: [email protected] >>>>> Gesendet: Montag, 25. Januar 2016 22:34:34 >>>>> Betreff: Help needed: commons-exec CLOSE_WAIT problem >>>>> >>>>> Hi commons-exec folks, >>>>> >>>>> Hoping you can help me figure this out. Selenium Java uses commons-exec >>>> 1.3 >>>>> under the hood to communicate with phantomjs. When it fires up it opens >>>> up >>>>> a pipe that you can see with lsof: >>>>> >>>>> # lsof -p 19947 >>>>> >>>>> ... >>>>> java 19947 user 64 PIPE 0xd2cd00ccca85f9d 16384 >>>>> ->0xd2cd00ca9fbbf9d >>>>> java 19947 user 66 PIPE 0xd2cd00ca31d445d 16384 >>>>> ->0xd2cd00ca31d4c9d >>>>> >>>>> Then we get this in lsof as selenium is driving phantomjs: >>>>> >>>>> java 19947 user 62u IPv6 0xd2cd00cc904879d 0t0 TCP >>>>> localhost:49757->localhost:23795 (ESTABLISHED) >>>>> >>>>> Then after selenium closes and terminates the executor-- properly, as I >>>>> observed by stepping through the code as it executes, but maybe someone >>>>> knows otherwise-- we can see in lsof: >>>>> >>>>> java 19947 user 62u IPv6 0xd2cd00cc904879d 0t0 TCP >>>>> localhost:49757->localhost:23795 (CLOSE_WAIT) >>>>> >>>>> And each successive web driver instance that gets created cause those to >>>>> build up and build up and build up until you run out of file handles. >>>>> >>>>> ... So are there any considerations for using commons-exec that the >>>>> selenium folks might not be addressing during process destruction which >>>>> might manifest themselves in these file handles just hanging out, taking >>>> up >>>>> space? This can't be the correct/uncorrectable behavior. >>>>> >>>>> I happen to be using Selenium Java 2.49.1 which is the latest version as >>>> of >>>>> this moment, and Java 8. It seems like this has been broken for quite >>>> some >>>>> time though -- https://github.com/SeleniumHQ/selenium/issues/1080 >>>>> >>>>> Please advise! Thanks! >>>>> >>>>> CG >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
