I just did the same. Thank you once more! On Wed, Jan 27, 2016 at 8:52 AM, Siegfried Göschl < siegfried.goes...@it20one.com> wrote:
> Hi Chris, > > glad to help - please note that I took the freedom to post my findings at > https://github.com/SeleniumHQ/selenium/issues/1080 - could you do the > same :-) > > Cheers, > > Siegfried Goeschl > > > ----- Ursprüngliche Mail ----- > Von: "Chris Gamache" <cgama...@gmail.com> > An: "Commons Users List" <user@commons.apache.org> > Gesendet: Mittwoch, 27. Januar 2016 14:43:32 > Betreff: Re: Help needed: commons-exec CLOSE_WAIT problem > > I really appreciate the thorough inspection you have given. With your help > I think I've targeted the real problem... > > GhostDriver can't exist outside of phantomjs. GhostDriver is an > implementation of WebDriver Wire Protocol which runs on phantomjs's > JavaScript engine. It can't run without phantomjs. > > We can rest easy that commons-exec is not holding open the file handle. > That's good. I think the PhantomJSDriver is firing a copy of phantomjs up > using commons-exec, but it's not communicating with the service through > commons-exec. Upon closer inspection selenium-java is communicating through > tcp/ip to a WebDriver server phantomjs is popping up on random available > ports. This happens to be an instance of some class that implements > org.apache.http.client.HttpClient, by default it looks to > be org.apache.http.impl.client.CloseableHttpClient ... I think that when > commons-exec terminates phantomjs, thereby terminating the GhostDriver > server, the HttpClient is still attached to that port. > > Here's my take on the real problem: > > I think the selenium folks didn't consider port closing > (org.apache.http.client.HttpClient doesn't have a close method to > implement) a necessary step and they figured that if whatever service the > HttpClient was attached to shut itself down that the port would > automatically close itself down also. The standard use for selenium usually > doesn't require it to run for days and days on end. I can't fault them for > missing it. > > That being said, they made this difficult to override in consumer classes > by keeping a lot of the guts of the connections hidden behind private > classes and private member data. > > So, I'll need to dig into selenium-java, change a few interfaces to make > sure that whatever HttpClient is used will have a close() hook, and then > make sure that whatever is driving the HttpClient has access to the close > hook on shutdown, and fires it. When the HttpClient is closed, the file > descriptor should disappear. > > Thank you again for your help. It was indispensable! > > CG > > > On Tue, Jan 26, 2016 at 4:07 PM, Siegfried Göschl < > siegfried.goes...@it20one.com> wrote: > > > Hi Chris, > > > > I played with your GitHub repo [ > > > https://github.com/cgamache/openfile](https://github.com/cgamache/openfile) > > > > ### 1. First Take > > > > Your test program actually starts a JVM and the Selenium library start > > multiple "phantomjs" executable to run the test as shown below > > > > ``` > > application> ps -ef | grep phantom > > 501 1021 460 0 9:26PM ttys000 0:00.71 java -jar openfile.jar > > /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs > > 501 1073 1062 0 9:26PM ttys000 0:00.08 > > /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs --webdriver=36418 > > > --webdriver-logfile=/Users/sgoeschl/work/github/cgamache/openfile/target/phantomjsdriver.log > > ``` > > > > So commons-exec is used under the hood to run the "phantomjs" binary and > > make sure nothing hangs (output & error thread pumping, watchdog, ..) > > > > When the "WebDriver" has finished its work there > > > > * is no ***phantomjs*** process running > > * is only your program running as shown below > > > > ``` > > application> ps -ef | grep phantom > > 501 1021 460 0 9:23PM ttys000 0:00.72 java -jar openfile.jar > > /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs > > 501 1034 469 0 9:23PM ttys001 0:00.00 grep phantom > > ``` > > > > And as described we have a couple of ports in CLOSE_WAIT stemming from > > your program > > > > ``` > > application> lsof -p 1021 | grep CLOSE_WAIT > > java 1021 sgoeschl 16u IPv6 0x78418468470e25c7 0t0 TCP > > localhost:50581->localhost:40560 (CLOSE_WAIT) > > java 1021 sgoeschl 18u IPv6 0x784184685efd95a7 0t0 TCP > > localhost:50599->localhost:45319 (CLOSE_WAIT) > > java 1021 sgoeschl 19u IPv6 0x784184685efdc0a7 0t0 TCP > > localhost:50620->localhost:17950 (CLOSE_WAIT) > > java 1021 sgoeschl 20u IPv6 0x78418468470e3087 0t0 TCP > > localhost:50641->localhost:41916 (CLOSE_WAIT) > > java 1021 sgoeschl 21u IPv6 0x78418468470e4b67 0t0 TCP > > localhost:50661->localhost:25113 (CLOSE_WAIT) > > ``` > > > > As far as common-exec is concerned everything is fine - the > > ***phantomjs*** processes where nicely executed and no process is left > > > > So you have 5 ports in limbo which exactly corresponds to the number of > > created ***phantom.js*** instances > > > > ``` > > INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs > > INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs > > INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs > > INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs > > INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs > > ``` > > > > This nicely corresponds to your test code > > > > ``` > > public static final int REPS = 5; > > > > for (; i < REPS; i++) { > > WebDriver driver = new PhantomJSDriver(caps); > > driver.get("http://www.apache.org";); > > driver.quit(); > > } > > ``` > > > > In other words each ***phantomjs*** process leaves exactly on port behind > > :-) > > > > > > ### 2. Second Take > > > > So where is this port coming from? The next session - starting the test > > programm leaves the 5 ports behind > > > > ``` > > application> lsof -p 1329 | grep CLOSE_WAIT > > java 1329 sgoeschl 16u IPv6 0x784184686037bb07 0t0 TCP > > localhost:51358->localhost:30177 (CLOSE_WAIT) > > java 1329 sgoeschl 18u IPv6 0x784184685f5c3b47 0t0 TCP > > localhost:51379->localhost:cadkey-tablet (CLOSE_WAIT) > > java 1329 sgoeschl 19u IPv6 0x784184686037f0c7 0t0 TCP > > localhost:51400->localhost:43648 (CLOSE_WAIT) > > java 1329 sgoeschl 20u IPv6 0x784184686037c5c7 0t0 TCP > > localhost:51418->localhost:34505 (CLOSE_WAIT) > > java 1329 sgoeschl 21u IPv6 0x78418468470e1047 0t0 TCP > > localhost:51438->localhost:30905 (CLOSE_WAIT) > > ``` > > > > All of ports are actually found in phantomjsdriver.log (cadkey-tablet is > > actually port 1400) > > > > ``` > > [INFO - 2016-01-26T20:51:19.697Z] GhostDriver - Main - running on port > > 30177 > > [INFO - 2016-01-26T20:51:21.685Z] GhostDriver - Main - running on port > > 1400 > > [INFO - 2016-01-26T20:51:23.427Z] GhostDriver - Main - running on port > > 43648 > > [INFO - 2016-01-26T20:51:25.156Z] GhostDriver - Main - running on port > > 34505 > > [INFO - 2016-01-26T20:51:26.887Z] GhostDriver - Main - running on port > > 30905 > > ``` > > > > So the ports are actually used/opened for ***GhostDriver*** and left > there > > even when ***driver.quit()*** was called > > > > > > ### 3. Conclusion > > > > * ASFAIK it is not commons-exec issue since all ***phantomjs*** were > > cleanly terminated > > * Each ***PhantomJSDriver*** invocation leaves a port in ***CLOSE_WAIT*** > > behind > > * Those ports are linked to ***GhostDriver - Main - running on port > > 30177*** > > > > > > Cheers, > > > > Siegfried Goeschl > > > > > > > > ----- Ursprüngliche Mail ----- > > Von: "Chris Gamache" <cgama...@gmail.com> > > An: "Commons Users List" <user@commons.apache.org> > > Gesendet: Dienstag, 26. Januar 2016 04:02:41 > > Betreff: Re: Help needed: commons-exec CLOSE_WAIT problem > > > > Most defnintely. This was tested on OSX, but should work properly on > Linux > > also. You'll need to install phantomjs. > > > > You can clone this repo (https://github.com/cgamache/openfile) and build > > with Maven. The com.codeborne:phantomjsdriver artifact has all of > selenium > > packaged inside. > > > > To run, > > > > $ java -jar openfile.jar /path/to/phantomjs > > > > To examine, > > > > $ lsof -p 38270 | grep CLOSE_WAIT | wc -l > > > > Jan 25, 2016 9:46:17 PM > > org.openqa.selenium.phantomjs.PhantomJSDriverService <init> > > > > ....... super chatty........ > > > > [INFO - 2016-01-26T02:46:41.169Z] ShutdownReqHand - _handle - About to > > shutdown > > > > > > Sleeping for 30 seconds so you can examine file handles... you should > see 5 > > > > Then you can use the lsof command at the top of the output, complete with > > process ID, to check the CLOSE_WAIT file handles. > > > > Thanks for taking a look! > > > > CG > > > > > > > > On Mon, Jan 25, 2016 at 5:28 PM, Siegfried Göschl < > > siegfried.goes...@it20one.com> wrote: > > > > > Hi Chris, > > > > > > there could be couple of reasons for this behaviour - is there a > minimal > > > setup to reproduce the problem? > > > > > > Thanks in advance > > > > > > Siegfried Goeschl > > > > > > ----- Ursprüngliche Mail ----- > > > Von: "Chris Gamache" <cgama...@gmail.com> > > > An: user@commons.apache.org > > > Gesendet: Montag, 25. Januar 2016 22:34:34 > > > Betreff: Help needed: commons-exec CLOSE_WAIT problem > > > > > > Hi commons-exec folks, > > > > > > Hoping you can help me figure this out. Selenium Java uses commons-exec > > 1.3 > > > under the hood to communicate with phantomjs. When it fires up it opens > > up > > > a pipe that you can see with lsof: > > > > > > # lsof -p 19947 > > > > > > ... > > > java 19947 user 64 PIPE 0xd2cd00ccca85f9d 16384 > > > ->0xd2cd00ca9fbbf9d > > > java 19947 user 66 PIPE 0xd2cd00ca31d445d 16384 > > > ->0xd2cd00ca31d4c9d > > > > > > Then we get this in lsof as selenium is driving phantomjs: > > > > > > java 19947 user 62u IPv6 0xd2cd00cc904879d 0t0 TCP > > > localhost:49757->localhost:23795 (ESTABLISHED) > > > > > > Then after selenium closes and terminates the executor-- properly, as I > > > observed by stepping through the code as it executes, but maybe someone > > > knows otherwise-- we can see in lsof: > > > > > > java 19947 user 62u IPv6 0xd2cd00cc904879d 0t0 TCP > > > localhost:49757->localhost:23795 (CLOSE_WAIT) > > > > > > And each successive web driver instance that gets created cause those > to > > > build up and build up and build up until you run out of file handles. > > > > > > ... So are there any considerations for using commons-exec that the > > > selenium folks might not be addressing during process destruction which > > > might manifest themselves in these file handles just hanging out, > taking > > up > > > space? This can't be the correct/uncorrectable behavior. > > > > > > I happen to be using Selenium Java 2.49.1 which is the latest version > as > > of > > > this moment, and Java 8. It seems like this has been broken for quite > > some > > > time though -- https://github.com/SeleniumHQ/selenium/issues/1080 > > > > > > Please advise! Thanks! > > > > > > CG > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > > > For additional commands, e-mail: user-h...@commons.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > > For additional commands, e-mail: user-h...@commons.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > For additional commands, e-mail: user-h...@commons.apache.org > >
