Hi all, My problem isn't that it's flaky (although that is a problem, a long standing one), but that it looks like it fails consistently on macOS, with the same error, on 2 different environments. (Anyone had success running tests on macOS? That would be a push for me to also +1 it) I'm not strongly -1 because of this, as tests run fine on linux systems, and it's our primary platform. Also, it builds and runs fine. So this is something we can perhaps fix for the upcoming 3.7.1, and just roll out 3.7.0 as is.
Sorry for the many emails, here is a summary so far: Patrick Hunt +1 (binding) Máté Szalay-Bekő +0 (non-binding) Norbert Kalmar -1 (non-binding, and tentative) Thanks for putting together the release Damien, everything looks flawless on that part :) - Norbert On Tue, Jan 26, 2021 at 10:17 PM Damien Diederen <ddiede...@apache.org> wrote: > > Greetings, all, > > I have seen your reports, and am planning to dig into them ASAP. > > As Máté remarked, I have found tests with -Dsurefire-forkcount > 1 to be > quite unreliable as of late. This is something I had been planning to > look into—but perhaps not at the same time as preparing this release :) > > FWIW, branch-3.7.0 "passed" on Jenkins: > > > https://ci-hadoop.apache.org/view/ZooKeeper/job/zookeeper-multi-branch-build/ > > That was just luck, however; one can see that branch-3.7 has failed > despite holding the same contents. > > Best, -D > > > > Norbert Kalmar <nkal...@cloudera.com.INVALID> writes: > > Sorry, the linked test is just an assertion error, no exception there. > > I had another test failure which had an error thrown. I will investigate > > further, as this might be some macOD address issue/collission yet again: > > > > [ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > > 26.229 s <<< FAILURE! - in > org.apache.zookeeper.server.quorum.CnxManagerTest > > [ERROR] testCnxManagerListenerThreadConfigurableRetry Time elapsed: > 0.178 > > s <<< ERROR! > > java.net.BindException: Address already in use > > at sun.nio.ch.Net.bind0(Native Method) > > at > > > org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:662) > > at > > > org.apache.zookeeper.server.ServerCnxnFactory.configure(ServerCnxnFactory.java:109) > > at > > > org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:191) > > at > > > org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:186) > > at > > > org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:1253) > > at > > > org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxManagerListenerThreadConfigurableRetry(CnxManagerTest.java:309) > > > > > > > > On Mon, Jan 25, 2021 at 5:20 PM Norbert Kalmar <nkal...@cloudera.com> > wrote: > > > >> -1 (non-binding!) > >> > >> - built the source, run locally, gave a few basic commands > >> - run tests on macOS 10.13.6, openjdk 1.8.0_242 - issues found, see > below > >> - checked files in tarball > >> - checked signatures > >> > >> I had the same problem as Máté. unit tests did not pass on MacOS. > Nothing > >> has changed in my environment since I did the 3.5.9 release, and it had > no > >> issue back then. > >> > >> I gave a -1 because this error doesn't only occur to me. > >> > >> The test that failed: > >> > >> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: > >> 62.332 s <<< FAILURE! - in > >> org.apache.zookeeper.server.util.RequestPathMetricsCollectorTest > >> > >> [ERROR] testMultiThreadPerf Time elapsed: 4.816 s <<< FAILURE! > >> > >> org.opentest4j.AssertionFailedError: expected: <true> but was: <false> > >> > >> at > >> > org.apache.zookeeper.server.util.RequestPathMetricsCollectorTest.testMultiThreadPerf(RequestPathMetricsCollectorTest.java:448) > >> > >> > >> > >> - Norbert > >> > >> > >> > >> On Mon, Jan 25, 2021 at 1:39 PM Szalay-Bekő Máté < > >> szalay.beko.m...@gmail.com> wrote: > >> > >>> +0 (and not even binding :) ) > >>> > >>> - I built the source code (-Pfull-build) on Ubuntu 18.04.3 using > OpenJDK > >>> 8u265 and maven 3.6.3. > >>> - I also built and executed unit tests for zkpython > >>> - the unit tests passed for the C-client and for python client > >>> - checkstyle and spotbugs passed > >>> - apache-rat passed > >>> - owasp (CVE check) passed > >>> - fatjar built (-Pfatjar) > >>> - I executed a quick rolling-upgrade test from 3.5.9 and from 3.6.2. > >>> (using > >>> https://github.com/symat/zk-rolling-upgrade-test) > >>> > >>> for some reason the java unit tests failed for me. > >>> > >>> On mac (jdk 1.8.212 and maven 3.6.3), I got all the unit tests executed > >>> successfully, but then the maven job still failed for hbase-server test > >>> with error message (with -DforkCount=4 and even with -DforkCount=1) > like: > >>> ------------- > >>> [ERROR] ExecutionException There was an error in the forked process > >>> [ERROR] unable to create new native thread > >>> [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: > >>> ExecutionException There was an error in the forked process > >>> [ERROR] unable to create new native thread > >>> [ERROR] at > >>> > >>> > org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510) > >>> ------------- > >>> > >>> > >>> Then I tried on a dockerized environment (ubuntu 18.4, OpenJDK 8u265 > and > >>> maven 3.6.3) and I got other kinds of strange maven errors: > >>> ------------------- > >>> [ERROR] Caused by: > >>> org.apache.maven.surefire.booter.SurefireBooterForkException: The > forked > >>> VM > >>> terminated without properly saying goodbye. VM crash or System.exit > >>> called? > >>> [ERROR] Command was /bin/sh -c cd > >>> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server && > >>> /home/symat/.sdkman/candidates/java/8.0.265-open/jre/bin/java -Xmx512m > >>> -Dtest.junit.threads=8 -Dzookeeper.junit.threadid=3 > >>> > >>> > -javaagent:/home/symat/.m2/repository/org/jmockit/jmockit/1.48/jmockit-1.48.jar > >>> -jar > >>> > >>> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire/surefirebooter8828313385463488429.jar > >>> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire > >>> 2021-01-25T11-54-03_621-jvmRun3 surefire4024538135165099286tmp > >>> surefire_37800399112966511000tmp > >>> [ERROR] Process Exit Code: 0 > >>> [ERROR] at > >>> > >>> > org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669) > >>> [ERROR] at > >>> > >>> > org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115) > >>> [ERROR] at > >>> > >>> > org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:444) > >>> [ERROR] at > >>> > >>> > org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:420) > >>> [ERROR] at > >>> java.util.concurrent.FutureTask.run(FutureTask.java:266) > >>> [ERROR] at > >>> > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >>> [ERROR] at > >>> > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >>> [ERROR] at java.lang.Thread.run(Thread.java:748) > >>> [ERROR] > >>> -------------------------------------- > >>> > >>> > >>> These issues might be very well specific to my local (mac or docker on > >>> mac) > >>> environments. This is why I didn't vote with -1 > >>> Can someone else run the java unit tests successfully locally? > >>> > >>> I also tried to check if the CI was green for the last PR on 3.7.0 ( > >>> https://github.com/apache/zookeeper/pull/1586/checks), but it looks > the > >>> CI > >>> haven't even started to execute the tests, due to errors in the > "install C > >>> dependencies" step. > >>> > >>> Regards, > >>> Mate > >>> > >>> On Sun, Jan 24, 2021 at 11:39 PM Patrick Hunt <ph...@apache.org> > wrote: > >>> > >>> > +1. xsum/sig verified. rat ran clean. built and dependency checks are > >>> fine. > >>> > Tried running some manual clusters and it was successful. > >>> > > >>> > Regards, > >>> > > >>> > Patrick > >>> > > >>> > > >>> > On Sun, Jan 24, 2021 at 12:11 PM Damien Diederen < > ddiede...@apache.org> > >>> > wrote: > >>> > > >>> > > > >>> > > Dear all, > >>> > > > >>> > > This is a second release candidate for ZooKeeper 3.7.0. Compared > to > >>> > > RC0, it fixes a tarball generation issue, includes a description of > >>> the > >>> > > 'whoami' CLI command, and incorporates a contribution to > ZooInspector. > >>> > > > >>> > > ZooKeeper 3.7.0 introduces a number of new features, notably: > >>> > > > >>> > > * An API to start a ZooKeeper server from Java (ZOOKEEPER-3874); > >>> > > > >>> > > * Quota enforcement (ZOOKEEPER-3301); > >>> > > > >>> > > * Host name canonicalization in quorum SASL authentication > >>> > > (ZOOKEEPER-4030); > >>> > > > >>> > > * Support for BCFKS key/trust store format (ZOOKEEPER-3950); > >>> > > > >>> > > * A choice of mandatory authentication scheme(s) > (ZOOKEEPER-3561); > >>> > > > >>> > > * A "whoami" API and CLI command (ZOOKEEPER-3969); > >>> > > > >>> > > * The possibility of disabling digest authentication > >>> (ZOOKEEPER-3979); > >>> > > > >>> > > * Multiple SASL "superUsers" (ZOOKEEPER-3959); > >>> > > > >>> > > * Fast-tracking of throttled requests (ZOOKEEPER-3683); > >>> > > > >>> > > * Additional security metrics (ZOOKEEPER-3978); > >>> > > > >>> > > * SASL support in the C and Perl clients (ZOOKEEPER-1112, > >>> > > ZOOKEEPER-3714); > >>> > > > >>> > > * A new zkSnapshotComparer.sh tool (ZOOKEEPER-3427); > >>> > > > >>> > > * Notes on how to benchmark ZooKeeper with the YCSB tool > >>> > > (ZOOKEEPER-3264). > >>> > > > >>> > > The release notes are available here: > >>> > > > >>> > > > >>> > > > >>> > > >>> > https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/releasenotes.html > >>> > > > >>> > > > >>> > > >>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12346617 > >>> > > > >>> > > *** Please download, test and vote by January 31st 2020, 23:59 > UTC+0. > >>> *** > >>> > > > >>> > > Source files: > >>> > > > >>> > > > https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/ > >>> > > > >>> > > Maven staging repo: > >>> > > > >>> > > > >>> > > > >>> > > >>> > https://repository.apache.org/content/repositories/orgapachezookeeper-1066/ > >>> > > > >>> > > The release candidate tag in git to be voted upon: release-3.7.0-1 > >>> > > > >>> > > https://github.com/apache/zookeeper/tree/release-3.7.0-1 > >>> > > > >>> > > ZooKeeper's KEYS file containing PGP keys we use to sign the > release: > >>> > > > >>> > > https://www.apache.org/dist/zookeeper/KEYS > >>> > > > >>> > > The staging version of the website is: > >>> > > > >>> > > > >>> > > > >>> > > >>> > https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/ > >>> > > > >>> > > Should we release this candidate? > >>> > > > >>> > > Damien Diederen > >>> > > > >>> > > >>> > >> >
