GitHub user neykov opened a pull request:
https://github.com/apache/brooklyn-server/pull/400
Tests - stop persistence before rebind
This is an attempt to fix the frequent test failures on Apache Jenkins
Windows slaves. The tests would fail with `The process cannot access the file
because it is being used by another process` while rebinding against the
on-disk persisted state.
While I wasn't able to reproduce this I believe it's due to the tests not
stopping persistence on the old management context. It probably tries to write
to the persisted state while the new management context rebinds, leading to
file locking errors. The fix is to stop persistence in the old management
context before rebind. Note that the old management context is still kept
alive, only persistence is stopped, so we can work with the old entities.
A couple of example failures:
*
https://builds.apache.org/view/Brooklyn/job/brooklyn-master-windows/85/consoleFull
*
https://builds.apache.org/view/Brooklyn/job/brooklyn-master-windows/84/consoleFull
```
2016-11-01 13:34:16,299 INFO TESTNG FAILED: "Surefire test" -
org.apache.brooklyn.entity.software.base.SoftwareProcessEntityHttpFeedRebindTest.testRebind()
finished in 723 ms
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Failure
rebinding; 2 errors including: problem loading memento: memento vykkmus4ho read
error: Problem reading String contents of file
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho:
FileNotFoundException:
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho
(The process cannot access the file because it is being used by another
process)
at
org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:129)
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebind(RebindManagerImpl.java:513)
at
org.apache.brooklyn.core.mgmt.rebind.RebindTestUtils.rebindAll(RebindTestUtils.java:454)
at
org.apache.brooklyn.core.mgmt.rebind.RebindTestUtils.rebind(RebindTestUtils.java:331)
at
org.apache.brooklyn.core.mgmt.rebind.RebindTestFixture.rebind(RebindTestFixture.java:295)
at
org.apache.brooklyn.core.mgmt.rebind.RebindTestFixtureWithApp.rebind(RebindTestFixtureWithApp.java:49)
at
org.apache.brooklyn.core.mgmt.rebind.RebindTestFixtureWithApp.rebind(RebindTestFixtureWithApp.java:32)
at
org.apache.brooklyn.core.mgmt.rebind.RebindTestFixture.rebind(RebindTestFixture.java:231)
at
org.apache.brooklyn.entity.software.base.SoftwareProcessEntityHttpFeedRebindTest.runRebindWithHttpFeed(SoftwareProcessEntityHttpFeedRebindTest.java:136)
at
org.apache.brooklyn.entity.software.base.SoftwareProcessEntityHttpFeedRebindTest.testRebind(SoftwareProcessEntityHttpFeedRebindTest.java:104)
Caused by: java.util.concurrent.ExecutionException:
org.apache.brooklyn.util.exceptions.CompoundRuntimeException: Failure
rebinding; 2 errors including: problem loading memento: memento vykkmus4ho read
error: Problem reading String contents of file
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho:
FileNotFoundException:
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho
(The process cannot access the file because it is being used by another
process)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:361)
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebind(RebindManagerImpl.java:511)
... 36 more
Caused by: org.apache.brooklyn.util.exceptions.CompoundRuntimeException:
Failure rebinding; 2 errors including: problem loading memento: memento
vykkmus4ho read error: Problem reading String contents of file
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho:
FileNotFoundException:
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho
(The process cannot access the file because it is being used by another
process)
at
org.apache.brooklyn.util.exceptions.Exceptions.create(Exceptions.java:439)
at
org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onDoneImpl(RebindExceptionHandlerImpl.java:497)
at
org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onDone(RebindExceptionHandlerImpl.java:413)
at
org.apache.brooklyn.core.mgmt.rebind.RebindIteration.run(RebindIteration.java:268)
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebindImpl(RebindManagerImpl.java:558)
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:508)
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:506)
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:522)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: problem loading memento:
memento vykkmus4ho read error
at
org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onLoadMementoFailed(RebindExceptionHandlerImpl.java:178)
at
org.apache.brooklyn.core.mgmt.persist.BrooklynMementoPersisterToObjectStore$2.visit(BrooklynMementoPersisterToObjectStore.java:294)
at
org.apache.brooklyn.core.mgmt.persist.BrooklynMementoPersisterToObjectStore$1VisitorWrapper.run(BrooklynMementoPersisterToObjectStore.java:452)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
... 4 more
Caused by: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
Problem reading String contents of file
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho:
FileNotFoundException:
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho
(The process cannot access the file because it is being used by another
process)
at
org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:165)
at
org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:140)
at
org.apache.brooklyn.core.mgmt.persist.FileBasedStoreObjectAccessor.get(FileBasedStoreObjectAccessor.java:58)
at
org.apache.brooklyn.core.mgmt.persist.BrooklynMementoPersisterToObjectStore.read(BrooklynMementoPersisterToObjectStore.java:641)
at
org.apache.brooklyn.core.mgmt.persist.BrooklynMementoPersisterToObjectStore.access$000(BrooklynMementoPersisterToObjectStore.java:83)
at
org.apache.brooklyn.core.mgmt.persist.BrooklynMementoPersisterToObjectStore$2.visit(BrooklynMementoPersisterToObjectStore.java:291)
... 6 more
Caused by: java.io.FileNotFoundException:
C:\Users\jenkins\AppData\Local\Temp\1\SoftwareProcessEntityHttpFeedRebindTest-z8Ub\locations\vykkmus4ho
(The process cannot access the file because it is being used by another
process)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at com.google.common.io.Files$FileByteSource.openStream(Files.java:126)
at com.google.common.io.Files$FileByteSource.openStream(Files.java:116)
at
com.google.common.io.ByteSource$AsCharSource.openStream(ByteSource.java:434)
at com.google.common.io.CharSource.read(CharSource.java:161)
at
org.apache.brooklyn.core.mgmt.persist.FileBasedStoreObjectAccessor.get(FileBasedStoreObjectAccessor.java:56)
... 9 more
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/neykov/brooklyn-server fix/test-persistence
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-server/pull/400.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #400
----
commit adb0173a73a36e9186b972de26e0ba03ad78e6a5
Author: Svetoslav Neykov <[email protected]>
Date: 2016-11-03T07:48:13Z
Create a new management context for rebind test
The tests in RebindManagerExceptionHandlerTest were reusing the same
management context on rebind - create a new management context instead to make
sure we are rebinding against the on-disk persisted state.
commit ca9a7c9e425f888d76ef556a2c2c432a8672b8af
Author: Svetoslav Neykov <[email protected]>
Date: 2016-11-03T07:52:10Z
Stop persitence before rebind attempt
This is an attempt to fix test failures on the Apache Jenkins Windows
slaves. The tests would frequrently fail with "The process cannot access the
file because it is being used by another process" when trying to read the
persisted state. Persistence in the old management context wasn't being stopped
before trying the rebind so there's a chance that it would try to overwrite the
files while the new management context rebinds, leading to file locking errors.
Stopping persistence before rebind should solve the locking failures.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
