Build failed in Jenkins: brooklyn-master-build-docker #428

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H26 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/brooklyn.git # timeout=10
Fetching upstream changes from 
https://git-wip-us.apache.org/repos/asf/brooklyn.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/brooklyn.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d136ae3a3cc022a4da7d62f10e3e886c4e3e26ae (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d136ae3a3cc022a4da7d62f10e3e886c4e3e26ae
Commit message: "Merge and close #14"
 > git rev-list --no-walk d136ae3a3cc022a4da7d62f10e3e886c4e3e26ae # timeout=10
 > git remote # timeout=10
 > git submodule init # timeout=10
 > git submodule sync # timeout=10
 > git config --get remote.origin.url # timeout=10
 > git submodule init # timeout=10
 > git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10
 > git config --get submodule.brooklyn-ui.url # timeout=10
 > git config -f .gitmodules --get submodule.brooklyn-ui.path # timeout=10
 > git submodule update --init --recursive --remote brooklyn-ui
 > git config --get submodule.brooklyn-server.url # timeout=10
 > git config -f .gitmodules --get submodule.brooklyn-server.path # timeout=10
 > git submodule update --init --recursive --remote brooklyn-server
 > git config --get submodule.brooklyn-library.url # timeout=10
 > git config -f .gitmodules --get submodule.brooklyn-library.path # timeout=10
 > git submodule update --init --recursive --remote brooklyn-library
 > git config --get submodule.brooklyn-dist.url # timeout=10
 > git config -f .gitmodules --get submodule.brooklyn-dist.path # timeout=10
 > git submodule update --init --recursive --remote brooklyn-dist
 > git config --get submodule.brooklyn-docs.url # timeout=10
 > git config -f .gitmodules --get submodule.brooklyn-docs.path # timeout=10
 > git submodule update --init --recursive --remote brooklyn-docs
 > git config --get submodule.brooklyn-client.url # timeout=10
 > git config -f .gitmodules --get submodule.brooklyn-client.path # timeout=10
 > git submodule update --init --recursive --remote brooklyn-client
[brooklyn-master-build-docker] $ /bin/bash -xe 
/tmp/jenkins5381828748502340515.sh
+ mkdir -p .m2
[brooklyn-master-build-docker] $ /bin/bash -xe 
/tmp/jenkins1414610741093077107.sh
+ git submodule update --remote --merge --recursive
[brooklyn-master-build-docker] $ /bin/bash -xe 
/tmp/jenkins6387756193374281877.sh
+ docker build -t brooklyn:latest .
Error checking context: 'can't stat 
'
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 6 days 0 hr old

TestNG Reports Processing: START
Looking for TestNG results report in workspace using pattern: 
**/testng-results.xml
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.
testng-results.xml was last modified before this build started. Ignoring it.

Jenkins build is back to normal : brooklyn-master-build-docker #427

[Apache Jenkins Server]

See 

Re: Brooklyn highAvailabilityMode: default as AUTO?

[Geoff Macartney]

+1 sounds like a change that's needed to match the intended behaviour
anyway, as described in the docs.  We should update the docs as part of
this to include your explanation above, Aled, of the details of the
behaviour.

regards
Geoff

On Tue, 22 May 2018 at 15:27 Thomas Bouron 
wrote:

> +1, sounds sensible to me
>
> Best.
>
> On Tue, 22 May 2018 at 14:51 Duncan Grant 
> wrote:
>
> > Aled,
> >
> > +1 sounds like a sensible plan
> >
> > Duncan
> >
> > On Tue, 22 May 2018 at 13:59 Aled Sage  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to change the default value of highAvailabilityMode from
> > > DISABLED to AUTO.
> > >
> > > Currently, if you start two Brooklyn servers pointing at the same
> > > persisted state (file-system directory or object store's bucket), then
> > > they are independent (because HA is 'disabled' by default). However,
> > > they both write to that same persisted state, which will lead to
> > > surprising behaviour, particularly when a Brooklyn server is next
> > > restarted.
> > >
> > > Changing to 'AUTO' would (almost entirely) have the same behaviour as
> we
> > > have currently for a single Brooklyn server. In the case of two servers
> > > pointing at the same persisted state, the second would come up as
> > > 'standby', and will be automatically promoted to 'master' if the first
> > > stops or fails.
> > >
> > > I say "almost entirely":
> > > 1. If you run Brooklyn and then kill it (e.g. `kill -9` or turn off the
> > > VM), when you start Brooklyn again it will wait to confirm the previous
> > > server is really dead. It waits for 30 seconds after the server's last
> > > heartbeat, by default.
> > > 2. The HA status shows all previous runs of the Brooklyn server (it
> gets
> > > a new node-id each time it restarts). This list will get longer and
> > > longer if you keep restarting Brooklyn, pointing at the same persisted
> > > state, until you clear out terminates instances from the list (via the
> > > UI or the REST api).
> > > 3. The logging at startup will be quite different (e.g. "Brooklyn
> > > initialisation (part two) complete" now means that the server has
> > > finished becoming the 'standby'. If anyone has tools/scripts that
> > > search/parse these logs, then they may be affected.
> > >
> > > ---
> > >
> > > Note the current behaviour contradicts the docs [1], which say:
> > > "Brooklyn will automatically run in HA mode if multiple Brooklyn
> > > instances are started pointing at the same persistence store."
> > >
> > > Thoughts?
> > >
> > > Aled
> > >
> > > p.s. another option would be to try to fail-fast when
> > > highAvailabilityMode is disabled but there is another Brooklyn using
> the
> > > same persisted state. However, distinguishing that from (1) above is
> > > tricky.
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/brooklyn-docs/blob/master/guide/ops/high-availability/index.md
> > >
> > >
> > >
> >
> --
>
> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> https://cloudsoft.io/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>

Build failed in Jenkins: brooklyn-master-build-docker #426

[Apache Jenkins Server]

See 


--
[...truncated 10.76 KB...]
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.tryOnceDeleteRecursive(Util.java:369)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:389)
at hudson.Util.deleteContentsRecursive(Util.java:225)
... 11 more
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Failed to delete workspace
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:558)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:207)
at hudson.remoting.UserRequest.perform(UserRequest.java:53)
at hudson.remoting.Request$2.run(Request.java:358)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to 
ubuntu-4
at 
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
at hudson.remoting.Channel.call(Channel.java:908)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor884.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy134.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1120)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
at hudson.scm.SCM.checkout(SCM.java:495)
at 
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at 
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at 
hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: java.io.IOException: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts.
at hudson.Util.deleteContentsRecursive(Util.java:230)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:555)
... 10 more
Caused by: java.nio.file.AccessDeniedException: 

at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at 

Re: Brooklyn highAvailabilityMode: default as AUTO?

[Thomas Bouron]

+1, sounds sensible to me

Best.

On Tue, 22 May 2018 at 14:51 Duncan Grant  wrote:

> Aled,
>
> +1 sounds like a sensible plan
>
> Duncan
>
> On Tue, 22 May 2018 at 13:59 Aled Sage  wrote:
>
> > Hi all,
> >
> > I'd like to change the default value of highAvailabilityMode from
> > DISABLED to AUTO.
> >
> > Currently, if you start two Brooklyn servers pointing at the same
> > persisted state (file-system directory or object store's bucket), then
> > they are independent (because HA is 'disabled' by default). However,
> > they both write to that same persisted state, which will lead to
> > surprising behaviour, particularly when a Brooklyn server is next
> > restarted.
> >
> > Changing to 'AUTO' would (almost entirely) have the same behaviour as we
> > have currently for a single Brooklyn server. In the case of two servers
> > pointing at the same persisted state, the second would come up as
> > 'standby', and will be automatically promoted to 'master' if the first
> > stops or fails.
> >
> > I say "almost entirely":
> > 1. If you run Brooklyn and then kill it (e.g. `kill -9` or turn off the
> > VM), when you start Brooklyn again it will wait to confirm the previous
> > server is really dead. It waits for 30 seconds after the server's last
> > heartbeat, by default.
> > 2. The HA status shows all previous runs of the Brooklyn server (it gets
> > a new node-id each time it restarts). This list will get longer and
> > longer if you keep restarting Brooklyn, pointing at the same persisted
> > state, until you clear out terminates instances from the list (via the
> > UI or the REST api).
> > 3. The logging at startup will be quite different (e.g. "Brooklyn
> > initialisation (part two) complete" now means that the server has
> > finished becoming the 'standby'. If anyone has tools/scripts that
> > search/parse these logs, then they may be affected.
> >
> > ---
> >
> > Note the current behaviour contradicts the docs [1], which say:
> > "Brooklyn will automatically run in HA mode if multiple Brooklyn
> > instances are started pointing at the same persistence store."
> >
> > Thoughts?
> >
> > Aled
> >
> > p.s. another option would be to try to fail-fast when
> > highAvailabilityMode is disabled but there is another Brooklyn using the
> > same persisted state. However, distinguishing that from (1) above is
> > tricky.
> >
> > [1]
> >
> >
> https://github.com/apache/brooklyn-docs/blob/master/guide/ops/high-availability/index.md
> >
> >
> >
>
-- 

Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Re: Brooklyn highAvailabilityMode: default as AUTO?

[Duncan Grant]

Aled,

+1 sounds like a sensible plan

Duncan

On Tue, 22 May 2018 at 13:59 Aled Sage  wrote:

> Hi all,
>
> I'd like to change the default value of highAvailabilityMode from
> DISABLED to AUTO.
>
> Currently, if you start two Brooklyn servers pointing at the same
> persisted state (file-system directory or object store's bucket), then
> they are independent (because HA is 'disabled' by default). However,
> they both write to that same persisted state, which will lead to
> surprising behaviour, particularly when a Brooklyn server is next
> restarted.
>
> Changing to 'AUTO' would (almost entirely) have the same behaviour as we
> have currently for a single Brooklyn server. In the case of two servers
> pointing at the same persisted state, the second would come up as
> 'standby', and will be automatically promoted to 'master' if the first
> stops or fails.
>
> I say "almost entirely":
> 1. If you run Brooklyn and then kill it (e.g. `kill -9` or turn off the
> VM), when you start Brooklyn again it will wait to confirm the previous
> server is really dead. It waits for 30 seconds after the server's last
> heartbeat, by default.
> 2. The HA status shows all previous runs of the Brooklyn server (it gets
> a new node-id each time it restarts). This list will get longer and
> longer if you keep restarting Brooklyn, pointing at the same persisted
> state, until you clear out terminates instances from the list (via the
> UI or the REST api).
> 3. The logging at startup will be quite different (e.g. "Brooklyn
> initialisation (part two) complete" now means that the server has
> finished becoming the 'standby'. If anyone has tools/scripts that
> search/parse these logs, then they may be affected.
>
> ---
>
> Note the current behaviour contradicts the docs [1], which say:
> "Brooklyn will automatically run in HA mode if multiple Brooklyn
> instances are started pointing at the same persistence store."
>
> Thoughts?
>
> Aled
>
> p.s. another option would be to try to fail-fast when
> highAvailabilityMode is disabled but there is another Brooklyn using the
> same persisted state. However, distinguishing that from (1) above is
> tricky.
>
> [1]
>
> https://github.com/apache/brooklyn-docs/blob/master/guide/ops/high-availability/index.md
>
>
>

Brooklyn highAvailabilityMode: default as AUTO?

[Aled Sage]

Hi all,

I'd like to change the default value of highAvailabilityMode from 
DISABLED to AUTO.


Currently, if you start two Brooklyn servers pointing at the same 
persisted state (file-system directory or object store's bucket), then 
they are independent (because HA is 'disabled' by default). However, 
they both write to that same persisted state, which will lead to 
surprising behaviour, particularly when a Brooklyn server is next restarted.


Changing to 'AUTO' would (almost entirely) have the same behaviour as we 
have currently for a single Brooklyn server. In the case of two servers 
pointing at the same persisted state, the second would come up as 
'standby', and will be automatically promoted to 'master' if the first 
stops or fails.


I say "almost entirely":
1. If you run Brooklyn and then kill it (e.g. `kill -9` or turn off the 
VM), when you start Brooklyn again it will wait to confirm the previous 
server is really dead. It waits for 30 seconds after the server's last 
heartbeat, by default.
2. The HA status shows all previous runs of the Brooklyn server (it gets 
a new node-id each time it restarts). This list will get longer and 
longer if you keep restarting Brooklyn, pointing at the same persisted 
state, until you clear out terminates instances from the list (via the 
UI or the REST api).
3. The logging at startup will be quite different (e.g. "Brooklyn 
initialisation (part two) complete" now means that the server has 
finished becoming the 'standby'. If anyone has tools/scripts that 
search/parse these logs, then they may be affected.


---

Note the current behaviour contradicts the docs [1], which say: 
"Brooklyn will automatically run in HA mode if multiple Brooklyn 
instances are started pointing at the same persistence store."


Thoughts?

Aled

p.s. another option would be to try to fail-fast when 
highAvailabilityMode is disabled but there is another Brooklyn using the 
same persisted state. However, distinguishing that from (1) above is tricky.


[1] 
https://github.com/apache/brooklyn-docs/blob/master/guide/ops/high-availability/index.md