I hit very strange timeouts waiting for the network server to start when
running the jmx management suite. The test does this:
1) start network server within same vm
2) run some tests
3) stop network server
4) start network server as separate process
5) run some tests
6) stop network server
The test was failing in step 4, waiting for the server to come up.
These timeouts have been blamed in the past on TCP/IP behaviour, delay
on the socket becoming free again. I don't see how this can be the case
as I could run this test multiple times, right after each other and
steps 1-3 always worked and step 4 always failed, thus showing the 1527
port was available for re-use. Indeed when this complete suite runs
successfully the network server is started and stopped over 20 times in
12 seconds, seemingly showing there's no issue in re-using a port.
Even stranger was on the same machine one codeline was not having any
issues while another was seeing the timeout. The timeouts continued
after a re-boot and then after I added some debugging they went away and
have not re-appeared.
Looking at the separate process code in NetworkServerTestSetup it does
have some issues, one coding and one design:
- The streams for the processed are not handled correctly which can
cause the process to block, which would cause a timeout. Thus if for
some reason the server does produce more output than normal it could
hang thus leading to timeouts. Since the streams are not handled
correctly the extra output is never seen making debugging this
intermittent problem hard.
- The server is booted without setting derby.system.home. For the vm
running the junit tests derby.system.home=${user.dir}/system. Since
d.s.h is not set the server is running with a system home of user.dir. I
think this explains the wombat database appearing in the current
directory when running junit tests. I'm not sure if this intended when
this decorator was added, the concept of the system folder was to have
all Derby databases beneath that folder. Of course if d.s.h is set to
${user.dir}/system for the spawned server then we have two jvms
accessing the same system and poentially the same databases. There are
three possible solutions to this:
a) Set d.s.h to ${user.dir}/system but shutdown Derby in the test
vm before starting the separate server.
b) Set d.s.h to ${user.dir}/ns (a different sub-folder) and ensure
tests wrapped in this decorator only use databases under d.s.h.
c) Leave it as-is, d.s.h defaults to ${user.dir} which leaves a
wombat in the current folder.
Wiki page on junit folders ...
http://wiki.apache.org/db-derby/DerbyJunitTestConfiguration?#head-af5a240fccda5c96f104776dfbd20a8cabde55e0
Dan.
