I haven't been able to repro this locally. Here are the details on my Ubuntu VM:
uname -a Linux ubuntu 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux java -version java version "1.8.0_45" Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) ant -version Apache Ant(TM) version 1.9.4 compiled on April 29 2014 I'm getting 100% passing test runs with multiple concurrent JUnit processes, including the tests that you mentioned were failing in your environment. I don't have any immediate ideas for what to try next. Everything has been working well on Jenkins and multiple dev machines, so it seems like there is some subtle environmental difference in this VM that I didn't handle in the ZOOKEEPER-2183 patch. Is this problematic for the release candidate? If so, then I recommend doing a quick change to set the default test.junit.threads to 1 in build.xml. That would restore the old single-process testing behavior. We can change test-patch.sh to pass -Dtest.junit.threads=8 on the command line, so we'll still get speedy pre-commit runs on Jenkins where it is working well. We all can do the same when we run ant locally too. Let me know if this is important, and I can put together a patch quickly. Thanks! --Chris Nauroth From: Flavio Junqueira <[email protected]<mailto:[email protected]>> Date: Friday, May 22, 2015 at 3:37 PM To: Chris Nauroth <[email protected]<mailto:[email protected]>> Cc: Zookeeper <[email protected]<mailto:[email protected]>> Subject: Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 1 That's the range I get in the vm. I also checked the load from log test and the port it was trying to bind to is 11222. -Flavio On 22 May 2015, at 23:14, Chris Nauroth <[email protected]<mailto:[email protected]>> wrote: No worries on the delay. Thank you for sharing. That's interesting. The symptoms look similar to something we had seen from an earlier iteration of the ZOOKEEPER-2183 patch that was assigning ports from the ephemeral port range. This would cause a brief (but noticeable) window in which the OS could assign the same ephemeral port to a client socket while a server test still held onto that port assignment. It was particularly noticeable for tests that stop and restart a server on the same port, such as tests covering client reconnect logic. In the final committed version of the ZOOKEEPER-2183 patch, I excluded the ephemeral port range from use by port assignment. Typically, that's 32768 - 61000 on Linux. Is it possible that this VM is configured to use a different ephemeral port range? Here is what I get from recent stock Ubuntu and CentOS installs: > cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 --Chris Nauroth From: Flavio Junqueira <[email protected]<mailto:[email protected]>> Date: Friday, May 22, 2015 at 2:47 PM To: Chris Nauroth <[email protected]<mailto:[email protected]>> Cc: Zookeeper <[email protected]<mailto:[email protected]>> Subject: Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 1 Sorry about the delay, here are the logs: http://people.apache.org/~fpj/logs-3.5.1-rc1/ the load test is giving bind exceptions. -Flavio On 21 May 2015, at 23:02, Chris Nauroth <[email protected]<mailto:[email protected]>> wrote: Thanks, sharing logs would be great. I'll try to repro independently with JDK8 too. --Chris Nauroth On 5/21/15, 2:30 PM, "Flavio Junqueira" <[email protected]<mailto:[email protected]>> wrote: I accidently removed dev from the response, bringing it back in. The tests are failing intermittently for me. In the last run, I got these failing: [junit] Tests run: 8, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 30.444 sec[junit] Test org.apache.zookeeper.test.LoadFromLogTest FAILED [junit] Tests run: 86, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 264.272 sec[junit] Test org.apache.zookeeper.test.NioNettySuiteTest FAILED Still the same setup, linux + jdk 8. I can share logs if necessary. -Flavio On Thursday, May 21, 2015 8:28 PM, Chris Nauroth <[email protected]<mailto:[email protected]>> wrote: Ah, my mistake. I saw "Azure" and my brain jumped right to "Windows". I suppose the thing for me to check then is JDK8. I believe all prior testing was on JDK7. --Chris Nauroth From: Flavio Junqueira <[email protected]<mailto:[email protected]>> Date: Thursday, May 21, 2015 at 12:18 PM To: Chris Nauroth <[email protected]<mailto:[email protected]>> Subject: RE: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 1 Yeah, I started with an Ubuntu vm, so it's Linux. I haven't tested the RC on windows yet. -FlavioFrom:Chris Nauroth Sent:?5/?21/?2015 6:46 PM To:[email protected]<http://zookeeper.apache.org/>;Flavio Junqueira Subject:Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 1 If I understand correctly, you're seeing test failures specifically on Windows (not Linux) after ZOOKEEPER-2183. Is that right? Tests have been stable in Linux Jenkins and dev environments after that patch, but perhaps there is another issue specific to Windows. I'll take a look on Windows. It might also be worthwhile to detect Windows and set test.junit.threads to 1 automatically in build.xml as a stop-gap. --Chris Nauroth On 5/21/15, 9:05 AM, "Flavio Junqueira" <[email protected]<mailto:[email protected]>> wrote: Yep, that did it. -Flavio On Thursday, May 21, 2015 5:23 AM, Michi Mutsuzaki <[email protected]<mailto:[email protected]>> wrote: I wonder if it's related to ZOOKEEPER-2183. Could you try setting test.junit.threads to 1 in build.xml? On Wed, May 20, 2015 at 1:44 PM, Flavio Junqueira <[email protected]<mailto:[email protected]>> wrote: I'm not being able to get a clean build for the RC. I'm running it on an azure vm with ubuntu and oracle jdk8. The java tests failing vary. At this point, I just wanted to check if I'm the only one seeing failures. -Flavio On Saturday, May 16, 2015 6:25 AM, Michi Mutsuzaki <[email protected]<mailto:[email protected]>> wrote: This is the second release candidate for 3.5.1-alpha. This candidate fixes some issues found in the first candidate, including ZOOKEEPER-2171. The full release notes is available at: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=1231080 1 &version=12326786 *** Please download, test and vote by May 29th 2015, 23:59 UTC+0. *** Source files: http://people.apache.org/~michim/zookeeper-3.5.1-alpha-candidate-1/ Maven staging repo: https://repository.apache.org/content/groups/staging/org/apache/zookeepe r /zookeeper/3.5.1-alpha/ The tag to be voted upon: https://svn.apache.org/repos/asf/zookeeper/tags/release-3.5.1-rc1/ ZooKeeper's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/zookeeper/KEYS Should we release this candidate?
