I think I may have fixed this issue. I think it is caused by Maven running the
tests in parallel. I changed my pom.xml to have the following configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18</version>
<configuration>
<useUnlimitedThreads>true</useUnlimitedThreads>
<parallel>suites</parallel>
<reuseForks>false</reuseForks>
<includes>
<include>**/*Test.java</include>
<include>**/Test*.java</include>
</includes>
</configuration>
</plugin>
This is telling maven to use only a single thread for the execution of my
entire test suite. This way nothing runs in parallel. With this I don't see the
same connection issues.
I would still like to know if this general approach to writing Accumulo unit
tests is good.
Thanks,
Jon Parise
From: Parise, Jonathan [mailto:[email protected]]
Sent: Friday, June 26, 2015 9:35 AM
To: [email protected]
Subject: MiniAccumuloClutser Unit Test Problems
Hello,
I have been writing some J-unit tests based on the MiniAccumuloCluster class.
I'm experiencing some issues when several of the tests run back to back. Before
I get into the error, let me explain how the tests work in general. Also, I am
using Accumulo 1.6.2.
Each test has an @BeforeClass method that first creates a new random directory.
Then makes a new MiniAccumuloCluster instance using that directory as the dir
parameter. Then, I call MiniAccumuloCluster.start().
There are several @Test methods in each test class. The typical pattern for
them is that they create any necessary tables, write some data into those
tables and then scan to verify it was written correctly. Basically they are
testing that I can serialize and deserialize various types of Objects correctly.
Then the test class as an @AfterClass method that calls
MiniAccumuloCluster.stop(). It also deletes the random directory used by the
previous test.
The issue I am running into is that generally the first test or two run fine.
However, the third test usually gets stuck in the MiniAccumuloCluster startup.
It just keeps complaining about being unable to connect. Note that if the test
is run independently it passes just fine. When run back to back, I see errors
like this one repeatedly:
2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket
connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket
connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004
for server null, unexpected error, closing socket connection and attempting
reconnect
2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket
connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket
connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket
connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket
connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket
connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004
for server null, unexpected error, closing socket connection and attempting
reconnect
In the ZooKeeperServerMain.out I see lines like this repeating several thousand
times:
2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
(NIOServerCnxn.java:finishSessionInit(1580)) - Established session
0x14e2fe179eb0000 with negotiated timeout 30000 for client /127.0.0.1:36311
2015-06-26 08:39:45,278 INFO [ProcessThread:-1] server.PrepRequestProcessor
(PrepRequestProcessor.java:pRequest(419)) - Got user-level KeeperException when
processing sessionid:0x14e2fe179eb0000 type:create cxid:0x31
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
Error:KeeperErrorCode = NodeExists for
/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
2015-06-26 08:39:45,300 INFO [ProcessThread:-1] server.PrepRequestProcessor
(PrepRequestProcessor.java:pRequest(419)) - Got user-level KeeperException when
processing sessionid:0x14e2fe179eb0000 type:create cxid:0x33
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
Error:KeeperErrorCode = NodeExists for
/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
I've thought about putting a Thread.sleep() call after the
MiniAccumuloCluster.stop() call, but that certainly seems brittle. I'm not sure
if that would improve the situation.
It seems to me that the MiniAccumuloCluster does not behave well when instances
are started and stopped several times. I am running the tests through Maven
using the default test behavior.
Could it be something with Maven? Maybe I need to be more explicit when telling
it how to run the tests?
Anyone have any insight into what is going wrong here? In general is my usage
pattern correct for MiniAccumuloCluster?
Thanks,
Jon Parise
Senior Software Engineer
Viz | General Dynamics Missons Systems
This message and/or attachments may include information subject to GD Corporate
Policies 07-103 and 07-105 and is intended to be accessed only by authorized
recipients. Use, storage and transmission are governed by General Dynamics and
its policies. Contractual restrictions apply to third parties. Recipients
should refer to the policies or contract to determine proper handling.
Unauthorized review, use, disclosure or distribution is prohibited. If you are
not an intended recipient, please contact the sender and destroy all copies of
the original message.
