Digging into the console log, I found evidence of a memory corruption error at
the JVM level.
[exec] [junit] Running org.apache.zookeeper.test.ClientPortBindTest in
thread 2
[exec] [junit] 2015-10-31 22:43:13,546 [myid:] - INFO
[main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
[exec] [junit] 2015-10-31 22:43:13,558 [myid:] - INFO
[main:ZKTestCase$1@55] - STARTING testBindByAddress
[exec] [junit] 2015-10-31 22:43:13,559 [myid:] - INFO
[main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD
testBindByAddress
[exec] [junit] *** Error in
`/home/jenkins/tools/java/jdk1.7.0-64/jre/bin/java': malloc(): memory
corruption: 0x00007ffc2c31e220 ***
[exec] [junit] Test org.apache.zookeeper.test.ClientPortBindTest
FAILED (crashed)
[exec] [junit] Running org.apache.zookeeper.test.ClientPortBindTest in
thread 2
[exec] [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 0 sec, Thread: 2, Class: org.apache.zookeeper.test.ClientPortBindTest
That's odd. ClientPortBindTest is purely a Java test. We aren't doing JNI or
anything.
The test calls NetworkInterface.getNetworkInterfaces(). Looking through JDK
bug reports, I see that there have been memory management bugs in this method.
https://bugs.openjdk.java.net/browse/JDK-7075227
https://bugs.openjdk.java.net/browse/JDK-8013140
https://bugs.openjdk.java.net/browse/JDK-8033146
The console log also shows that we're running a fairly old version of Java 7.
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) Server VM (build 23.25-b01, mixed mode)
I cannot repro this in any of my own environments, including a Java 1.7.0_67
environment and a Java 1.8.0_45 environment.
I think it's likely that we're looking at an old Java bug, not a problem in the
ZooKeeper codebase. If others agree, then I can file an INFRA JIRA to ask them
for an upgrade to the latest Java 7 revision on our Jenkins hosts.
--Chris Nauroth
From: Flavio Junqueira <[email protected]<mailto:[email protected]>>
Reply-To: Zookeeper <[email protected]<mailto:[email protected]>>
Date: Sunday, November 1, 2015 at 8:24 AM
To: Zookeeper <[email protected]<mailto:[email protected]>>
Subject: Test failure
This test seems to be failing frequently in trunk, has anyone had a look at
this:
org.apache.zookeeper.test.ClientPortBindTest.testBindByAddress<https://builds.apache.org/view/All/job/ZooKeeper-trunk/2803/testReport/org.apache.zookeeper.test/ClientPortBindTest/testBindByAddress/>
[cid:429F009F-4D21-4A1F-AC3F-6164CF19DC0D@home] Error Details
Forked Java VM exited abnormally. Please note the time in the report does not
reflect the time until the VM exit.
[cid:69D1E67F-567C-4206-93ED-970DBD884C8D@home] Stack Trace
-Flavio
