Re: Proposal: Add appId optional paramater to deploy api

[Aled Sage]

Thanks Alex,

Yes - we can say it's `Beta` in the rest api description, and in the 
plumming code for however we pass the app-id through.


I'll delete my existing PR, and submit a new one based on this approach.

Aled


On 27/07/2017 08:12, Alex Heneveld wrote:


Thanks Aled -- the inheritance of config from catalog items convinces me.

Can we mark it @Beta / internal in case we need to change the 
approach?  With that I'd be happy with your proposal.


Best
Alex


On 27/07/2017 07:23, Aled Sage wrote:

Hi Alex,

I explored setting a config key in my PR. The downsides of that 
compared to setting the app-id:


1. Code is more complicated - in particular, there are very low-level 
changes to check that the uid is not already in use.
2. Config keys (and tags) are mutable - we can't enforce the proposed 
semantics of the deploymentUid.
3. You can always rely on looking up an entity's id, but config is 
more complicated and can be "resolved" - leads to more complicated 
code (e.g. my PR currently has test-failures for hot-standby).
4. Appropriate search not supported in the REST api (would require 
additional changes - i.e. more work).
5. Server-side searching would either be inefficient or require a new 
index to be exposed (e.g. low-level changes in `EntityManager`).

6. Adds another concept - a new kind of id.

---
By "camp.id", do you mean "camp.plan.id"? (there is also 
"camp.template.id", but I can't find a "camp.id".)


I don't think we should mix this up with that existing concept, which 
is set and used in a completely different way:
* One can already set a camp id on the top-level app, and reference 
it with `$brooklyn:entity(...)` - we don't want to break that.
* The camp id does not have to be unique - changing that would break 
backwards compatibility.
* Existing catalog items can set the camp id, which are used by the 
app instance - e.g. see example in the appendix.


---
What are your main concerns about allowing the id to be set (with the 
regex validation that Mark suggested)? The reason that the is used by 
other internal parts of Brooklyn seems too vague to me. For example, 
is it:
1. Security (e.g. if we didn't validate, then it would risk allowing 
code injection).
2. Makes future changes harder (but the sorts of changes I envisage I 
can also see how we could handle).
3. Point of principle to keep internal things internal wherever 
possible.
4. Risks breaking places that we use the id in strange ways (e.g. if 
an entity uses the id to generate a dns name, then it implies 
case-insensitivity for the uniqueness check).


That last concern is real - I recall that Andrew Kennedy changed our 
ids to be lower case for that reason!


However, I don't think we should have given such "undocumented 
guarantees" on the internal id format. But given some entities rely 
on it, we can be stricter with the validation regex of user-supplied 
ids.


Aled

=== Appendix ===
Example of setting a `camp.plan.id` - we don't want to subtly change 
the semantics of this.


Add this to the catalog:
```
brooklyn.catalog:
  id: app-with-camp-plan-id
  version: 0.0.0-SNAPSHOT
  itemType: template
  item:
services:
  - type: org.apache.brooklyn.entity.stock.BasicApplication
id: myAppCampPlanId
```

Deploy this app:
```
services:
  - type: app-with-camp-plan-id
```

The resulting app instance will have config `camp.plan.id` with value 
`myAppCampPlanId`.




On 27/07/2017 00:40, Alex Heneveld wrote:
The core `id` is a low-level part of `BrooklynObject` used by all 
adjuncts

and entities and persistence.  It feels wrong and risky making this
something that is user- or client- settable.  I gave one example but 
there

are others.

What's wrong with a new config key or reusing `camp.id`?  We already 
use

the latter one if there is a user-specified ID on an entity so it feels
natural to use it, give it special meaning for apps (blocking repeat
deployments), and add support for searching for it. (Apologies if 
this was

explained and I missed it.)

--A


On 26 July 2017 at 22:42, Aled Sage <aled.s...@gmail.com> wrote:


Hi Alex,

Other things get a lot simpler for us if we can just supply the app-id
(e.g. subsequently searching for the app, or ensuring that a 
duplicate app
is not deployed). It also means we're not adding another concept 
that we

need to explain to users.

To me, that simplicity is very compelling.

If we want to support conformance to external id requirements, we 
could

have a config key for a predicate or regex that the supplied id must
satisfy. A given user could thus enforce their id standards in the 
future.

I'd favour deferring that until there is a need to support it (e.g. we
could add it at the same time as adding support for a pluggable id
generator, if we ever do that).

Aled



On 26/07/2017 15:42, Alex Heneveld wrote:

2 feels compelling to me. I want us to have the ability easily to 
change
the

Re: Proposal: Add appId optional paramater to deploy api

[Aled Sage]

// apologies this ended up so long!

Hi Alex, Svet, all,

Interesting suggestion by Alex - sounds useful. However, even with that 
it would require implementing a big chunk of logic in the upstream system.


I keep coming back to thinking that `appId=...` gives the simplest 
semantics for such systems.


---
The caching of `X-Request-ID` approach doesn't work in the scenario when 
the Brooklyn process has restarted, or has failed over in HA mode (I'm 
presuming you don't intend to write them to persisted state).


We'd therefore still end up with an upstream implementation as though 
`X-Request-ID` wasn't supported.


We'd still want either `appId=...` or `deploymentUid=...` so that we can 
subsequently query to find if there is a given orphaned application.


It would be nice if the query was simple/efficient. That's easy if using 
appId, but fiddly to search for an app with a given `deploymentUid` 
config value or tag (I think currently it would require listing all 
apps, then querying each app for this config key; or changing brooklyn 
to expand the rest api; or running an additional custom web-app for such 
bespoke queries).


---
I think it's fine/sensible to focus on the specific problem of 
deploy-app, because that is the current use-case.


Separately, having a general solution that works in some situations is a 
nice addition, but feels like a lower priority for our use-case.


---
I presume if implementing the `X-Request-ID` cache:

 * The cache would be transient (i.e. not persisted for restart/HA).
 * It would record everything with side-effects (POST/PUT/DELETE, but
   not GET/HEAD).
 * We'd record it as soon as the request was received (rather than
   waiting for the request to complete).
 * We'd return a 409 (conflict) response if we received a duplicate.
 * It would be configurable to turn this feature on/off (defaulting to on).
 * The time for keeping these request-ids would be configurable,
   defaulting to something like an hour.

Assuming folk agree it's separate, then let's treat that as a separate 
proposal / email thread.


---

Mark has pointed out that strictly speaking we shouldn't be using query 
parameters with POST requests.


The RESTful way of doing it if supplying a custom app id would be:

`PUT /applications/`

It's not so obvious how best to supply a deploymentUid in the url.

---

To Svet's point of using a tag (if going for the `deploymentUid` 
approach), good point - agree that would probably be better. We'd 
probably add it as the string `org.apache.brooklyn.deploymentUid=`.


It would have been nicer (in this use-case and various others) if tags 
was a map rather than a list, but we could live with adding the uid to 
the list.


---

p.s. To again summarise the requirements/thinking: we want to make it 
easier for upstream systems to use Brooklyn robustly. We want to more 
easily/safely handle retries, and to be able to easily query for and/or 
discover orphaned apps (where a request got through, but response was 
lost). Our primary use-case in the upsteram system is deploying apps. We 
don't want to modify the user's yaml files.


Aled



On 26/07/2017 10:16, Svetoslav Neykov wrote:

I think both proposals make sense. Even if implemented separately.

Wondering what's the right place to put the ID though. Isn't the tags a better 
place for that?  I suppose it depends on whether we want YAML blueprints to 
have access to it.

Svet.



On 25.07.2017 г., at 21:56, Alex Heneveld <alex.henev...@cloudsoftcorp.com> 
wrote:

Aled-

Should this be applicable to all POST/DELETE calls?  Imagine an
`X-caller-request-uid` and a filter which caches them server side for a
short period of time, blocking duplicates.

Solves an overlapping set of problems.  Your way deals with a
"deploy-if-not-present" much later in time.

--A

On 25 July 2017 at 17:44, Aled Sage <aled.s...@gmail.com> wrote:


Hi all,

I've been exploring adding support for `=...` - please see
my work-in-progress PR [1].

Do people think that is a better or worse direction than supporting
`=...` (which would likely be simpler code, but exposes the Brooklyn
internals more).

For `=...`, we could either revert [2] (so we could set the id in
the EntitySpec), or we could inject it via a different (i.e. add a new)
internal way so that it isn't exposedon our Java api classes.

Thoughts?

Aled

[1] https://github.com/apache/brooklyn-server/pull/778

[2] https://github.com/apache/brooklyn-server/pull/687/commits/5
5eb11fa91e9091447d56bb45116ccc3dc6009df



On 07/07/2017 18:28, Aled Sage wrote:


Hi,

Taking a step back to justify why this kind of thing is really
important...

This has come up because we want to call Brooklyn in a robust way from
another system, and to handle a whole load of failure scenarios (e.g. that
Brooklyn is temporarily down, connection fails at some point during the
communication, the client in the other system goes down and another
instance tries to pick up where it left 

Re: Proposal: Add appId optional paramater to deploy api

[Aled Sage]

Hi Mark,

We removed from EntitySpec the ability to set the id for two reasons:

1. there was no use-case at that time; simplifying the code by deleting 
it was therefore sensible - we'd deprecated it for several releases.


2. allowing all uids to be generated/managed internally is simpler to 
reason about, and gives greatest freedom for future refactoring.



I don't see (2) as a compelling reason.  And we now have a use-case, so 
that changes (1).


I'd still be tempted to treat this as an internal part of the api, 
rather than it going on the public `EntitySpec`, but need to look at 
that more to see how feasible it is.


Aled


On 26/07/2017 13:36, Mark Mc Kenna wrote:

Thanks Geoff for the good summary

IMO the path of least resistance that provides the best / most predictable
behaviour is allowing clients to optionally set the app id.

Off the top of my head I cant see this causing any issue, as long as we
sanitise the supplied id something like [a-zA-Z0-9-]{8,}.

Was there a particular reason this was removed in the past?

Cheers
M

On 26 July 2017 at 13:07, Duncan Grant <duncan.gr...@cloudsoftcorp.com>
wrote:


Thanks all for the advice.

I think Geoff's email summarises the issue nicely.  I like Alex's
suggestion but agree that limiting ourselves to deploy in the first is
probably significantly easier.

Personally I don't feel comfortable with using a tag for idempotency and I
do like Aled's suggestion of using PUT with a path with /{id} but would be
happy with either as I think they both fit our use case.

thanks

Duncan

On Wed, 26 Jul 2017 at 11:00 Geoff Macartney <geoff.macart...@cloudsoft.io
wrote:


If I understand correctly this isn't quite what Duncan/Aled are asking

for

though?
Which is not a "request id" but an idempotency token for an application.

It

would
need to work long term, not just cached for a short time, and it would

need

to work
across e.g. HA failover, so it wouldn't be just a matter of caching it on
the server.

For what it's worth, I'd have said this is a good use for tags but maybe
for ease of reading
it's better to have it as a config key as Aled does. As to how to supply
the value
I agree it should just be on the "deploy" operation.

On Tue, 25 Jul 2017 at 19:56 Alex Heneveld <
alex.henev...@cloudsoftcorp.com>
wrote:


Aled-

Should this be applicable to all POST/DELETE calls?  Imagine an
`X-caller-request-uid` and a filter which caches them server side for a
short period of time, blocking duplicates.

Solves an overlapping set of problems.  Your way deals with a
"deploy-if-not-present" much later in time.

--A

On 25 July 2017 at 17:44, Aled Sage <aled.s...@gmail.com> wrote:


Hi all,

I've been exploring adding support for `=...` - please

see

my work-in-progress PR [1].

Do people think that is a better or worse direction than supporting
`=...` (which would likely be simpler code, but exposes the

Brooklyn

internals more).

For `=...`, we could either revert [2] (so we could set the id

in

the EntitySpec), or we could inject it via a different (i.e. add a

new)

internal way so that it isn't exposedon our Java api classes.

Thoughts?

Aled

[1] https://github.com/apache/brooklyn-server/pull/778

[2] https://github.com/apache/brooklyn-server/pull/687/commits/5
5eb11fa91e9091447d56bb45116ccc3dc6009df



On 07/07/2017 18:28, Aled Sage wrote:


Hi,

Taking a step back to justify why this kind of thing is really
important...

This has come up because we want to call Brooklyn in a robust way

from

another system, and to handle a whole load of failure scenarios

(e.g.

that

Brooklyn is temporarily down, connection fails at some point during

the

communication, the client in the other system goes down and another
instance tries to pick up where it left off, etc).

Those kind of thing becomes much easier if you can make certain
assumptions such as an API call being idempotent, or it guaranteeing

to

fail with a given error if that exact request has already been

accepted.

---

I much prefer the semantics of the call failing (with a meaningful

error)

if the app already exists - that will make retry a lot easier to do

safely.

As for config keys on the app, in Duncan's use-case he'd much prefer

to

not mess with the user's YAML (e.g. to inject another config key

before

passing it to Brooklyn). It would be simpler in his case to supply

in

the

url `?appId=...` or `?deploymentId=...`.

For using `deploymentId`, we could but that feels like more work.

We'd

want create a lookup of applications indexed by `deploymentId` as

well

as

`appId`, and to fail if it already exists. Also, what if someone

also

defines a config key called `deploymentId` - would that be

forbidden?

Or

would we name-space the config key with

`org.apache.brooklyn.deploymentId`?

Even with those concerns, I could be persuaded of the
`org.apache.brooklyn.deploymentId` approach.

For "/application's ID is not meant to be user

Re: Proposal: Add appId optional paramater to deploy api

[Aled Sage]

Hi Alex,

Other things get a lot simpler for us if we can just supply the app-id 
(e.g. subsequently searching for the app, or ensuring that a duplicate 
app is not deployed). It also means we're not adding another concept 
that we need to explain to users.


To me, that simplicity is very compelling.

If we want to support conformance to external id requirements, we could 
have a config key for a predicate or regex that the supplied id must 
satisfy. A given user could thus enforce their id standards in the 
future. I'd favour deferring that until there is a need to support it 
(e.g. we could add it at the same time as adding support for a pluggable 
id generator, if we ever do that).


Aled


On 26/07/2017 15:42, Alex Heneveld wrote:

2 feels compelling to me. I want us to have the ability easily to change
the ID generation eg to conform with external reqs such as timestamp or
source.

Go with deploymentUid or similar? Or camp.id?

Best
Alex

On 26 Jul 2017 15:00, "Aled Sage" <aled.s...@gmail.com> wrote:

Hi Mark,

We removed from EntitySpec the ability to set the id for two reasons:

1. there was no use-case at that time; simplifying the code by deleting it
was therefore sensible - we'd deprecated it for several releases.

2. allowing all uids to be generated/managed internally is simpler to
reason about, and gives greatest freedom for future refactoring.


I don't see (2) as a compelling reason.  And we now have a use-case, so
that changes (1).

I'd still be tempted to treat this as an internal part of the api, rather
than it going on the public `EntitySpec`, but need to look at that more to
see how feasible it is.

Aled



On 26/07/2017 13:36, Mark Mc Kenna wrote:


Thanks Geoff for the good summary

IMO the path of least resistance that provides the best / most predictable
behaviour is allowing clients to optionally set the app id.

Off the top of my head I cant see this causing any issue, as long as we
sanitise the supplied id something like [a-zA-Z0-9-]{8,}.

Was there a particular reason this was removed in the past?

Cheers
M

On 26 July 2017 at 13:07, Duncan Grant <duncan.gr...@cloudsoftcorp.com>
wrote:

Thanks all for the advice.

I think Geoff's email summarises the issue nicely.  I like Alex's
suggestion but agree that limiting ourselves to deploy in the first is
probably significantly easier.

Personally I don't feel comfortable with using a tag for idempotency and I
do like Aled's suggestion of using PUT with a path with /{id} but would be
happy with either as I think they both fit our use case.

thanks

Duncan

On Wed, 26 Jul 2017 at 11:00 Geoff Macartney <
geoff.macart...@cloudsoft.io
wrote:

If I understand correctly this isn't quite what Duncan/Aled are asking
for


though?
Which is not a "request id" but an idempotency token for an application.


It


would
need to work long term, not just cached for a short time, and it would


need


to work
across e.g. HA failover, so it wouldn't be just a matter of caching it on
the server.

For what it's worth, I'd have said this is a good use for tags but maybe
for ease of reading
it's better to have it as a config key as Aled does. As to how to supply
the value
I agree it should just be on the "deploy" operation.

On Tue, 25 Jul 2017 at 19:56 Alex Heneveld <
alex.henev...@cloudsoftcorp.com>
wrote:

Aled-

Should this be applicable to all POST/DELETE calls?  Imagine an
`X-caller-request-uid` and a filter which caches them server side for a
short period of time, blocking duplicates.

Solves an overlapping set of problems.  Your way deals with a
"deploy-if-not-present" much later in time.

--A

On 25 July 2017 at 17:44, Aled Sage <aled.s...@gmail.com> wrote:

Hi all,

I've been exploring adding support for `=...` - please


see
my work-in-progress PR [1].

Do people think that is a better or worse direction than supporting
`=...` (which would likely be simpler code, but exposes the


Brooklyn


internals more).

For `=...`, we could either revert [2] (so we could set the id


in

the EntitySpec), or we could inject it via a different (i.e. add a

new)

internal way so that it isn't exposedon our Java api classes.

Thoughts?

Aled

[1] https://github.com/apache/brooklyn-server/pull/778

[2] https://github.com/apache/brooklyn-server/pull/687/commits/5
5eb11fa91e9091447d56bb45116ccc3dc6009df



On 07/07/2017 18:28, Aled Sage wrote:

Hi,

Taking a step back to justify why this kind of thing is really
important...

This has come up because we want to call Brooklyn in a robust way


from

another system, and to handle a whole load of failure scenarios

(e.g.

that

Brooklyn is temporarily down, connection fails at some point during
the

communication, the client in the other system goes down and another

instance tries to pick up where it left off, etc).

Those kind of thing becomes much easier if you can make certain
assumptions such as an API call being idempotent, or it guarantee

Re: Catalog adding unexpected behaviour

[Aled Sage]

Hi Richard,

Thanks for raising this. FYI Alex has created a PR with tests 
demonstrating the behaviour [1] you describe.


---

In the docs [2], we encourage the use of the format below for "entity" 
(because there is only one thing, the `services: ` is unnecessary and 
misleading - it suggests you could have multiple things under `services`):


   brooklyn.catalog:
  version: 0.1.3.SNAPSHOT
  items:
- id: inner
  itemType: entity
  item:
type: server
- id: outer
  itemType: entity
  item:
type: inner

In contrast, if you're adding a "template" (which can consist of 
multiple services), then the format would be:


   brooklyn.catalog:
  version: 0.1.3-SNAPSHOT
  items:
- id: my-app
  itemType: template
  item:
services:
   - type: server

If you use that format for `itemType: entity`, I think it works as expected.

Aled

[1] https://github.com/apache/brooklyn-server/pull/748

[2] http://brooklyn.apache.org/v/0.11.0/ops/catalog/



On 29/06/2017 09:48, Richard Downer wrote:

Hello all,

If I create a catalog.bom with these contents and try to add this to the
catalog with `br catalog add`:

brooklyn.catalog:
   version: 0.1.3.SNAPSHOT
   items:
 - id: inner
   itemType: entity
   item:
 services:
   - type: server
 - id: outer
   itemType: entity
   item:
 services:
   - type: inner

it fails with error `Server error (400): Could not resolve item 'outer'; 2
errors including: Transformer for Brooklyn OASIS CAMP interpreter gave an
error creating this plan: Unable to match plan item:
Service[name=,description=,serviceType=inner,characteristics=[],customAttributes={}]`

If I split this into two different catalog.bom files, and add them one
after the other, it works.

It seems to me that what works with two individual catalog.bom files should
also work when both files are combined into one. What do others think about
this?

Changing `entity` for `template` resolves this problem - as I understand
it, `template` causes in-depth parsing to be deferred until something tries
to use the catalog item. Unfortunately that means that some errors in the
catalog item may not be discovered until deployment time.

Richard.

Re: Faster PR reviews

[Aled Sage]

Hi Alex,

Interesting question. A few initial thoughts:

*YOML*
YOML (PR #363) is an exceptional case - we should not use that as an 
example when discussing this meta-question. The PR is 12,000 lines 
(including comments/notes), and was not discussed on the mailing list 
before it was submitted. I suggest we have a separate email thread 
specifically about merging that PR, as there are certainly very useful 
things we'd get from YOML.


*Small PRs*
We should strongly encourage small focused PRs on a single thing, 
wherever possible. That will make review faster, easier and lower risk. 
For such PRs, we should strive for review+merge within days (7 days 
being an upper bound in normal circumstances, hopefully).


We can add some brief guidelines to this effect at 
http://brooklyn.apache.org/developers/how-to-contribute.html


*Changing low-level Brooklyn*
PRs that change low-level things in Brooklyn (e.g. changes to 
config-inheritance etc) deserve thorough review. They are high-risk as 
the unforeseen consequences of the changes can be very subtle, and break 
downstream blueprints that rely on old ways of doing things.


*Light-weight Review*
I agree with you - where PRs look sensible, low-risk and unit tested we 
should take more risk and merge them sooner (even if there's not been 
time for a thorough review by the community).


Aled


On 02/05/2017 15:50, Duncan Johnston Watt wrote:

Hi Alex

This is probably covered already but I guess there needs to be an impact
assessment (by submitter?) before something is waved through by default.

Best

Duncan

On 2 May 2017 at 06:52, Alex Heneveld 
wrote:


Hi Brooklyners-

As many of you know, my YOML PR #363 [1] has been open for a while. This
sets up a foundation for giving better documentation and feedback and
hugely simplifying how we do our parsing.  However it's a very big PR.  I'm
eager to have people spend some time using it and ideally extending it --
but here I wanted to raise a meta-question:


*W**hat should we do with a PR when we aren't able to review things in as
much depth as we'd like?*


One option -- call it (A) -- is to say if we can't review things
thoroughly in a reasonable timeframe, we do a lighter review and if the PR
looks promising and safe we merge it.

The other option -- call it (B) -- is to leave PRs open for as long as it
takes for us to do the complete review.

I think most people have been approaching this with a mindset of (B), and
while that's great for code quality and shared code understanding, if we
can't deliver on that quickly, it's frankly anti-social.  The contributor
has to deal with merge conflicts (and the rudeness of his or her
contribution being ignored), and Brooklyn loses velocity.  My PR is an
extreme example but many have been affected by slow reviews, and I think
the expectation that reviews have to be so thorough is part of the
problem:  it even discourages reviewers, as if you're not an expert in an
area you probably don't feel qualified to review.

We have good test coverage so product risk of (A) is small, and we have
great coders so I've no worry about us being able to solve problems that
(A) might introduce.  We should be encouraging reviewers to look at any
area, and we need to solve the problem of slow reviews.


*I propose that the**standard we apply is that we quickly either merge PRs
or identify what the contributor needs to resolve.


*I'm all for thorough reviews and shared understanding, but if we can't do
this quickly I suggest we are better to sacrifice those things rather than
block contributions, stifle innovation, and discourage reviews by insisting
on a standards that we struggle to sustain.

As a general rule of thumb, maybe something like:

(1) After 7 days of no activity on a PR we go with an "eyeball test";
unless the following statement is untrue we say:

/I haven't done as much review as I'd like, but the code is clearly
helpful, not risky or obviously wrong  or breaking compatibility, it has
good test coverage, and we can reasonably expect the contributor or
committers to maintain it.  Leaving open a bit longer in case someone else
wants to review more but if nothing further in the next few days, let's
merge.

/(If there are committers who are likely to be specifically interested,
call them out as CC.)

(2) After 3 more days, if no activity, merge it.

And we encourage _anyone_ to review anything.  If the above response is
the baseline, everyone in our community is qualified to do it or better and
we'll be grateful!

Best
Alex


[1]  https://github.com/apache/brooklyn-server/pull/363

Re: [PROPOSAL] Remove java 7 support

[Aled Sage]

Thanks Geoff, much appreciated!

Sent from my iPhone


> On 12 May 2017, at 10:41, Geoff Macartney <geoff.macart...@cloudsoftcorp.com> 
> wrote:
> 
> hi all,
> 
> just picking up this thread again; as mailed back in March (!) [1] our
> Jenkins builds are now using Java 8.
> 
> Has everyone switched to Java 8 and is it now time to update Brooklyn to
> build with 8 rather than 7?
> 
> If no-one objects, I'll try to get some time to do this later next week.
> 
> regards
> Geoff
> 
> 
> [1]
> https://lists.apache.org/thread.html/6d50d242d1ed8e98776eb871c397de1aac06d3baf600edecc53415ff@%3Cdev.brooklyn.apache.org%3E
> 
> On Mon, 6 Mar 2017 at 19:48 Geoff Macartney <
> geoff.macart...@cloudsoftcorp.com> wrote:
> 
>> hi Aled,
>> 
>> sure, I'll have a look at that.
>> 
>> cheers
>> Geoff
>> 
>>> On Mon, 6 Mar 2017 at 18:26 Aled Sage <aled.s...@gmail.com> wrote:
>>> 
>>> Hi all,
>>> 
>>> There have been six +1s (including myself) and no negatives, so let's do
>>> this.
>>> 
>>> ---
>>> 
>>> For Robert's question, I did some investigation for use of lambdas as
>>> config key values. They don't persist correctly unfortunately [1], so we
>>> shouldn't use them anywhere that could end up in persisted state.
>>> 
>>> ---
>>> 
>>> I suggest we separate this into multiple stages (and ensure that
>>> discussion of later stages does not block the earlier stages):
>>> 
>>>  * Stage one: switch to Java 8 as "official target"
>>> 1. Update our jenkins etc to always build/run with Java 8.
>>> 2. All developers/testers switch to Java 8 locally.
>>> 3. Add to the next release notes that Java 7 is no longer supported.
>>>  * Stage two: update our poms so they don't compile/check for Java 7
>>>compatibility [2,3,4].
>>>  * Stage three: update our docs to say not to use lambdas for config
>>>keys (similar to what we already say for anonyous inner classes [5])
>>>  * Stage four: developers free to use Java 8 language features *where
>>>it does not affect APIs or end up in persisted state*
>>>  * Stage five: discuss/improve use of Java 8 (e.g. perhaps fix [1];
>>>should we keep using guava classes that are now in Java 8's
>>>java.util.function.* ?)
>>> 
>>> ---
>>> 
>>> Geoff: if you're volunteering, it would be awesome if you could pick up
>>> stage 1.1 (and then stage 2).
>>> 
>>> Aled
>>> 
>>> [1] https://issues.apache.org/jira/browse/BROOKLYN-448
>>> [2] https://github.com/apache/brooklyn-server/blob/master/pom.xml#L91
>>> [3]
>>> 
>>> https://github.com/apache/brooklyn-server/blob/master/parent/pom.xml#L557-L558
>>> [4]
>>> https://github.com/apache/brooklyn-server/blob/master/parent/pom.xml#L950
>>> [5]
>>> 
>>> https://brooklyn.apache.org/v/latest/ops/persistence/index.html#writing-persistable-code
>>> 
>>> 
>>>> On 06/03/2017 14:35, Geoff Macartney wrote:
>>>> hi all,
>>>> 
>>>> any more thoughts on this?   What concrete steps do we want to take, and
>>>> when?  What can I do to help out?
>>>> 
>>>> Plus, has anyone any views on Robert's question?
>>>> 
>>>> cheers
>>>> Geoff
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Mon, 27 Feb 2017 at 14:21 Robert Moss <robert.m...@cloudsoft.io>
>>> wrote:
>>>> 
>>>>> Aled,
>>>>> 
>>>>> WRT Java 8 features.  Is it still recommended to use named inner
>>> classes
>>>>> over anonymous inner classes? Can we now simplify those areas to use
>>>>> lambdas?  There are a number of classes now built into Java that were
>>>>> previously provided by Guava etc. e.g. Optional, Predicate - should we
>>>>> start using the Java 8 classes instead where possible?
>>>>> 
>>>>> Robert
>>>>> 
>>>>>> On 27 February 2017 at 13:08, Aled Sage <aled.s...@gmail.com> wrote:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> I propose we switch to Java 8 for Apache Brooklyn (i.e. require Java
>>> 8;
>>>>>> remove support for using Java 7 when running Brooklyn).
>>>>>> 
>>>>>> This will make our te

[PROPOSAL] Delete java/groovy example apps that are misleading

[Aled Sage]

Hi all,

I suggest we delete the java/groovy examples in 
brooklyn-library/examples [1] (but not "webapps" in that dir, which is 
used to build WAR files used in demos apps).


These do not follow the new (*) recommended way of writing blueprints. 
Since then, we have written better yaml-based examples in the docs. 
Keeping these Java/Groovy examples is frankly misleading for users.


They are also never tested by anyone anymore, so are unlikely to 
actually work!


Any objections to me delete them?

Aled

(*) The YAML way is not really "new" these days! Should probably just 
say "do not follow the recommended way".

Re: [PROPOSAL] Delete java/groovy example apps that are misleading

[Aled Sage]

I've submitted a PR to delete these: 
https://github.com/apache/brooklyn-library/pull/107



On 10/05/2017 16:04, Thomas Bouron wrote:

Sounds good to me too.

+1

Best

On Wed, 10 May 2017, 14:41 Geoff Macartney, <
geoff.macart...@cloudsoftcorp.com> wrote:


sounds good to me!


On Wed, 10 May 2017 at 14:40 bostko <bos...@gmail.com> wrote:


edit
On 10/05/17 16:37, bostko wrote:

+1

for *deleting* contents in
https://github.com/apache/brooklyn-library/tree/master/examples except
[1]

[1]


https://github.com/apache/brooklyn-library/tree/master/examples/webapps

On 10/05/17 16:30, Aled Sage wrote:

Hi all,

I suggest we delete the java/groovy examples in
brooklyn-library/examples [1] (but not "webapps" in that dir, which
is used to build WAR files used in demos apps).

These do not follow the new (*) recommended way of writing
blueprints. Since then, we have written better yaml-based examples in
the docs. Keeping these Java/Groovy examples is frankly misleading
for users.

They are also never tested by anyone anymore, so are unlikely to
actually work!

Any objections to me delete them?

Aled

(*) The YAML way is not really "new" these days! Should probably just
say "do not follow the recommended way".



--
Kind Regards,
Valentin Aitken!

skype: valentin.aitken
mobile: (+359) 883 453696 <+359%2088%20345%203696>

Re: Code Contribution: Container Service

[Aled Sage]

+1; sounds great.

I think the next steps would be for the code to be made publicly 
viewable, for any 3rd party licenses etc to be checked, and then for us 
to kick off a formal vote?


Richard: can you please guide us on the process?

Aled


On 10/05/2017 17:14, Duncan Johnston-Watt wrote:

+1 (non-binding)

Best
--
Duncan Johnston-Watt
CEO | Cloudsoft Corporation
+44 777 190 2653 | @duncanjw

Sent from my iPhone


On 10 May 2017, at 13:16, Graeme Miller  wrote:

Hello,

We at Cloudsoft Corporation have spent some time developing functionality
on top of Brooklyn that allows a user to deploy to Swarm, Kubernetes and
Openshift. We call this functionality the Cloudsoft Container Service.

We'd like to contribute this code back to Apache Brooklyn. The code comes
complete with documentation and tests and has been well reviewed by people
within Cloudsoft. We view this as mature code, that has been shipped with
our product for some time.

To find more information about how a user would use these locations please
see this documentation[1].

If people are in favour of this, we can provide more deatails about the
code and put it to a vote.

Regards,
Graeme Miller

[1] https://docs.cloudsoft.io/ccs/locations/index.html

Re: [DISCUSS][VOTE] Release Apache Brooklyn 0.11.0 [rc3]

[Aled Sage]

My guess for why it worked previously is that it used to default to centos 6, 
whereas now it's using a centos 7 vm image that has different default config.

I agree it should not block the release.

Aled

Sent from my iPhone


> On 17 May 2017, at 15:25, Geoff Macartney  
> wrote:
> 
> Still it looks bad if one of our out-of-the-box examples doesn't work
> surely?
> 
> I didn't test the templates this time round but did so in previous
> releases, and they all worked then.
> 
> 
> 
>> On Wed, 17 May 2017 at 13:52 Richard Downer  wrote:
>> 
>> IRC:
>> 
>> [13:46:09]   Question @here: if I'm using `openIptables:true`,
>> is it expected that using JcloudsLocationCustomizer should open iptables
>> ports - e.g.
>> 
>> https://github.com/apache/brooklyn-library/blob/master/software/nosql/src/main/java/org/apache/brooklyn/entity/nosql/riak/RiakNodeImpl.java#L137-L161
>> does not open iptables ports, should it?
>> [13:47:06]   richardasf, no it is not expected.
>> [13:48:01]   There is customer requirement for unifying port opening
>> mechanism and translating security  group rules to OS firewall rules.
>> [13:48:28]   I hope I will submit a suggestion for it this week.
>> [13:48:39]   Ok, that clears up that. thanks valio
>> 
>> 
>> So it appears that RiakNode is simply incompatible with iptables, so it
>> should be run with `stopIptables:true` (and therefore run on a cloud which
>> supports security groups or similar). So there's a simple workaround.
>> 
>> This would also appear to NOT be a regression.
>> 
>> In that case I'm comfortable with this not being a release blocker.
>> 
>> Richard.
>> 
>> 
>>> On 17 May 2017 at 13:26, Richard Downer  wrote:
>>> 
>>> It's related to iptables. Setting `stopIptables: true` fixes the problem.
>>> But setting `openIptables: true` does not - I though this odd, since
>>> something is configuring the AWS security group correctly, so I don't
>>> understand why it isn't also configuring iptables with the same data...
>>> 
>>> Richard.
>>> 
>>> 
 On 16 May 2017 at 20:37, Richard Downer  wrote:
 
 Urgh, we'd better investigate. If there's a failure in one of our "try
 this to get started!" blueprints I'd consider that a release blocker.
 Hopefully there's a good reason, or at least a simple workaround...
 
 Richard.
 
 On 16 May 2017 at 17:28, Geoff Macartney  wrote:
 
> I get this too Richard:
> 
> start failed with error:
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$Que
> ueAbortedException:
> Cannot add a task to Task[start]@iEOrS6Mt whose queue has been aborted
> (trying to add Task[Cross-context execution: Invoking effector
> joinCluster
> on RiakNode:d5gt with parameters {nodeName=
> r...@ec2-54-154-176-140.eu-west-1.compute.amazonaws.com}]@U09W94lm)
> 
> Failure running task Cross-context execution: Invoking effector
> joinCluster
> on RiakNode:vrua with parameters {nodeName=
> r...@ec2-54-154-176-140.eu-west-1.compute.amazonaws.com} (lpAS8V4t):
> Error
> invoking joinCluster at RiakNodeImpl{id=vrua9dk6kf}: Execution failed,
> invalid result 1 for joinCluster RiakNodeImpl{id=vrua9dk6kf}
> 
> 
> 
> On Tue, 16 May 2017 at 17:03 Richard Downer 
>> wrote:
> 
>> Hello all,
>> 
>> I'm trying out the rc3 and seeing a problem. If I deploy the
>> "Template
> 3"
>> app (web server + Riak cluster) from the "New Application" window,
> then the
>> individual cluster nodes appear to start, but the cluster as a whole
> goes
>> on fire.
>> 
>> Drilling down, it appears to be a "join cluster" activity which is
> failing.
>> The stdout of the task says:
>> "Node r...@ec2-52-57-99-169.eu-central-1.compute.amazonaws.com is
>> not
>> reachable!"
>> 
>> This is running in AWS EC2 in eu-central-1 - everything is in the
>> same
>> region.
>> 
>> Can anybody else reproduce?
>> 
>> Thanks
>> Richard.
>> 
>> 
>>> On 12 May 2017 at 17:09, Richard Downer  wrote:
>>> 
>>> This thread is for discussions related to the release vote.
>>> 
>>> I should clarify what we are looking for in a release vote.
> Particularly,
>>> we are looking for people to download,validate, and test the
>> release.
>>> Only if you are satisfied that the artifacts are correct and the
> quality
>> is
>>> high enough, should you make a "+1" vote. Alongside your vote you
> should
>>> list
>>> the checks that you made.
>>> 
>>> Here is a good example:
>> http://markmail.org/message/gevsz2pdciraw6jw
>>> 
>>> The vote is not simply about "the master branch contains the
> features I
>>> wanted" -
>>> it is about making sure that *these* 

Re: thoughts on Brooklyn docs

[Aled Sage]

Thanks Mohan,

That's helpful. Really useful to get different people's perspectives for 
how to summarise the key points of Brooklyn!


The diagrams are also interesting - good for communicating the control 
flow to Brooklyn developers and new folk who want to contribute, or to 
power-users who want to hook in at those levels. They feel more detailed 
than a "normal user" of Brooklyn would want, so perhaps would be best 
somewhere under the "developers" section, such as augmenting [1].


Aled

[1] http://brooklyn.apache.org/v/latest/dev/code/structure.html


On 15/05/2017 21:53, mohan kumar Muddana wrote:

This was very high level when I created this when I was in Atos. Let me
know if this is of any help. I know the architecture has changed a bit
after that but basic flow remains the same.
Feel free to copy or let me know how to improve on that.



*https://github.com/mohanjune/brooklynlearnings/tree/master/docs
*
Thanks and regards
Mohan


On Thu, May 11, 2017 at 12:21 AM, Geoff Macartney <
geoff.macart...@cloudsoftcorp.com> wrote:


Was just chatting about docs with some colleagues and remembered this from
Richard, which I thought was a great outline for what the docs should look
like.

How about we make a start on reworking what's there with this as an initial
guide to the endpoint,  and see how it goes?

On Fri, 21 Apr 2017, 21:44 Richard Downer,  wrote:


Hi,

Yes there are some big improvements possible in this area.

As someone who has taken a bit of a back seat in using Apache Brooklyn

for

a while and has recently restarted actively developing blueprints, I've
found that Brooklyn's features in this area have much improved over the
last year or two, but the documentation has failed to keep up - many

great

features available to blueprint writers are either documented too briefly
or in an illogical location, or not documented at all.

I have been thinking that if I was to use my Copious Free Time to write a
book about creating blueprints for Apache Brooklyn, it's table of

contents

would look something like this:

Basic Ingredients
- A simple server example
- What the machine looks like
- The software process lifecycle
- Installing files
- Sensors
- Enrichers
- Effectors
Combining Parts
- Multiple entities in one blueprint
- References between entities
- Latches
- Child entities
- Putting multiple entities on one server
Catalog: the blueprint database
- What is the catalog?
- Looking inside the catalog
- A simple blueprint into the catalog
- Versions and updates
- Parameters and configuration
- Bundling resources with blueprints
- A comprehensive example
The Apache Brooklyn Blueprint Toolkit
- Clusters
- Java-based apps
- [more standard Java entities]
Policies for Active, Automated Management
- Introduction to policies with the Service Restarter policy
- [more policies]
Blueprints for Microsoft Windows Server
- [tbc]
Writing Blueprints in Java
- Depending on Apache Brooklyn: Maven
- A simple Java blueprint
- OSGI for versioning and dependency management
- Enrichers
- Policies
Reference
- Blueprint YAML
- Apache Brooklyn DSL
- Common ConfigKeys
- Common Sensors
- FreeMarker Template

There would also be a separate manual covering the operational side of
Brooklyn: installing and configuring, persistence and HA, configuring
locations, managing the catalog, etc.. Then possibly a final user manual,
which would probably be quite slim: just about how to use the UI and `br`
client to work with blueprints already in the catalog, and examining
running apps.

My 2c.

Richard.


On 21 Apr 2017 17:36, "Geoff Macartney" 

Re: Versions in Brooklyn

[Aled Sage]

+1; sounds sensible.

You said /"I propose we resolve this by recommending a version syntax 
which fits what most things people are doing and which is bi-di mappable 
to OSGi"/.


To clarify, I think you're saying we recommend a version syntax like of 
1.0.0-SNAPSHOT (and discourage 1.0-SNAPSHOT), so that it is a 
bi-directional mapping. Is that right (but examples in the docs link 
include `1.10-rc3-20170619`)?


Would we let users use (non bi-di mappable) 1.0-SNAPSHOT without any 
warnings or deprecation?


Aled


On 22/06/2017 10:28, Svetoslav Neykov wrote:

Makes sense.



On 22.06.2017 г., at 12:27, Alex Heneveld  
wrote:


inline

On 22/06/2017 10:10, Svetoslav Neykov wrote:

+1 to the proposal.

One thing I have reservations about is having a recommended version syntax with 
other formats still supported but deprecated.
As far as I understand the recommended syntax is there so we can guarantee a 
uniqueness of the OSGi versions (when the source version is unique). Instead of 
having a recommended syntax can we document what we consider a unique version 
and let the user decide what format to follow?

yes, we could.  but i think it's nicer in a community setting where blueprints 
are being shared if versions follow the same format.  (we could enforce the 
recommended syntax in the community catalog.)

also i tend to think it's easier for users if we recommend a syntax rather than 
have to explain about uniqueness of osgi bundles. (currently that explanation 
is buried in an advanced section which can safely be ignored.)

--a



Svet.



On 20.06.2017 г., at 14:23, Alex Heneveld  
wrote:


I've drafted the documentation for how this could be explained to users.  This 
may be easier to grok than the email:

https://github.com/apache/brooklyn-docs/pull/198/files#diff-21dacc664dfe4d0a156d65d768a0f0e2R28

Best
Alex



On 19/06/2017 17:39, Alex Heneveld wrote:

Hi All-

TL;DR - I am proposing that we encourage versions in Brooklyn of the form "1.1.0" or "1.2-qualifier" such as 
"1.2-SNAPSHOT", silently mapping when needed to OSGi as "1.1.0" or "1.2.0.qualifier" / 
"1.2.0.SNAPSHOT"


Further to my last mail -- we have a bit of discord between various versioning 
schemes--

* GitHub SemVer - which everyone talks lovingly about (though often not 
knowledgeably, and it's stricter than I realized!)
* OSGi versioning - a precursor to (1), in widespread use but I've never heard 
anyone say anything nice about it
* Maven - allows whatever you want but has recommendations and conventions most 
people kinda follow

They all agree on up to three numbers at the start.  It's what comes after that varies, usually either a 
"-" (semver, maven, conventions) or "." (osgi), followed by qualifiers.  If practice 
almost everyone seems to do "-" followed by qualifiers -- however qualifiers in practice often 
don't follow the strict constraints of semver (no leading zeroes, no underscores) nor some of the maven 
recommendations (use of build number).

(Detailed summary on SemVer and OSGi versioning is included below for 
reference.)

So far, Brooklyn hasn't had an opinion and I liked it that way. However when 
registering OSGi bundles we MUST confirm with OSGi versioning there.  I'm 
pretty sure it's NOT desirable to enforce OSGi versioning on types, given that 
few people use it.  BUT we are moving to a world where I think we want type 
versions (entity versions etc) to align with bundle versions:  there is really 
no point in types having different versions to their defining bundle!  This 
makes for an incompatibility between what people would naturally use and what 
we have to use within OSGi.

With examples, my assumption is that people want to use and see strings like 
"1.1-SNAPSHOT".   But under the covers the OSGi bundle needs to have 
"1.1.0.SNAPSHOT".

I propose we resolve this by recommending a version syntax which fits what most 
things people are doing and which is bi-di mappable to OSGi.  We use this 
version everywhere except where a strict OSGi version is needed.  We WARN if we 
get a non-compliant version in a place which might be ambiguous.  And we 
minimise places where we need to rely on mapping.  (The main place a mapping is 
needed is if we need to create an OSGi version or compare with an OSGi version.)

Specifically I propose that Brooklyn type versions SHOULD be:

 ( "."  ( "."  ")? )? ( "-" ) ?
where qualifier can have letters, numbers, "-" or "_" but NOT additional 
".".

We construct an OSGi version, when needed, by replacing the first "-" with "." and inserting 0's if 
needed for a missing minor/patch.  So "1.1-SNAPSHOT" becomes "1.1.0.SNAPSHOT" when an OSGi version 
is needed.

Note that the above is a SHOULD.  The only strict requirement is the version string MUST 
NOT contain a ":".  (That breaks parsing.)

Where non-compliant versions are supplied, we WARN, but things work.  We apply simple heuristics to create a valid 

Re: Bundles in Brooklyn

[Aled Sage]

+1 to this approach, but I suspect there's a lot more to discuss/plan.


_*Leaking OSGi*_
I agree with Geoff's worry of us "leaking OSGi". However, we should just 
"leak" the semantics of bundle dependencies and versioning, which need 
to be explained to the user. I think we can avoid exposing the OSGi 
syntax anywhere (except for power-users who want it).


_*`brooklyn.libraries` versus `package-imports analogy*_
OSGi recommends the use of package-imports, rather than explicitly 
depending on specific bundles.


The analogy for us would be to depend on other catalog item 
name:versionRange (e.g. in a `brooklyn.imports` section, or whatever). 
We'd then figure out which bundle to get those from using the same rules 
as OSGi package-imports.


Is it worth exploring further that option, rather than going down the 
`brooklyn.libraries` route?


_*References Between Bundles*_
I don't think long-term the `brooklyn.libraries` should have URLs of 
other bundles. It could give the name:versionRange of other bundles 
(which assumes they have somehow already been added to Brooklyn).


(Reasoning is that other smart folk don't embed the URLs, presumably 
because they want people to chose which repo things will come from - 
e.g. enterprise private repo, versus maven central, etc).


_*Deploying Multiple Bundles*_
I lean towards something analogous to the kar file, but maybe lighter 
weight rather than introducing anything like "features".


Really we just need a list of bundle URLs. If someone then deletes one 
of those bundles, which breaks the other bundles deployed along with it, 
then tough!


_*Blueprint Validation*_
Longer term, it would be nice if we further improved our blueprint 
validation to ensure all references in a catalog item being added are 
actually resolvable.


Adding to Brooklyn the ability to add a list-of-bundles will no doubt 
make that harder!


_*Incremental Steps?*_
This might well prove to be a big change under-the-covers. Things that 
scare me include:
* do we need to re-implement for catalog-items the same 
dependency/version rules as OSGi uses for class-loading? (+)
* how much will backwards compatibility with historic persisted state 
make things hard for us.


Are there sensible ways to scope this and de-risk it? Any incremental 
milestones, which give us incremental value or de-risk it?


(+) We could potentially hack something where we synthesise a 
package/resource per catalog item in our auto-generated bundles, and 
then ask OSGi where it would load that from! (But that feels like a bad 
idea, because power-users might add their own bundles directly to karaf.)


_*Business Case (aka value to end-users)*_
It would be good to be clearer on the user-value we deliver from this 
(but please shout if others think that is already clear enough).


We're making "upgrades" easier - focusing specifically on deploying a 
second version of a catalog item, which can then be used (ignoring 
upgrade of existing app instances for now).


We're solving something like the DLL-hell problem (currently, if you 
don't declare explicit versions within your blueprint, then your 
blueprint might break when someone else adds a newer version of your 
dependent blueprint for their own usage). Our current workaround for DLL 
hell is impractical: hardcode explicit version numbers of your 
dependencies in the `type: name:version` (no support for version ranges 
etc, and it would be burried deep within the yaml files).


We're making it easier to deploy different versions of a "group of 
catalog items" (e.g. a big app that is split into several catalog 
items). Currently (if you didn't hardcode versions in every `type: ...` 
reference), deploying a 2.0.0 version would likely break a subsequent 
deployment of the 1.0.0 version, because it would default to the latest 
for each catalog reference. That is, you'd only get the 1.0.0 version of 
the main item, and then the 2.0.0 version of each item it referenced.


We'd move away from `brooklyn.libraries` hard-coding URLs to other 
bundles, which most likely goes against the security policies of many 
enterprises!


Is that a fair summary? Anything to add/change?

Aled


On 21/06/2017 09:41, Geoff Macartney wrote:

hi Alex,

that sounds fair enough.  We'll want to add good documentation around what
it means if *everything* is now a bundle (and how we wrap plain catalogs
into bundles on users' behalf).  We can use the text of your email as a
starting point for the docs; it would be good to explain some of the
_motivation_ around the choice to make everything a bundle, rather than
just say what we do.

I do like the idea of version ranges in brooklyn.libraries, but agree it's
best to proceed in small steps so that would be a later development.

Some more questions on implementation details

- is it part of this plan to move bundle storage into the persistence
location so that HA Brooklyn nodes can find all bundles?
- if a catalog.bom (symbolic name xyz) includes 

Re: ApplicationClusters

[Aled Sage]

Hi Daan,

Sounds interesting! Brooklyn seems like a good fit for what you're 
looking for.


Within Brooklyn, there's a separation of the application definition and 
the location to which it is to be deployed - apps can be written to be 
location-agnostic. From your perspective, the good thing about that is 
someone can write an app without saying about the specific location, and 
then the location is wired in behind the scenes at deploy time (e.g. 
based on the given user's identity, and the implicit CloudStack endpoint 
url).


Brooklyn also includes an application catalog. Users can choose apps 
from the catalog, can compose new apps using the catalog items as 
building blocks, and can add new blueprints to the catalog.


---
A few questions/comments:

Are you planning to extend the existing CloudStack API to support app 
provisioning (and the subsequent lifecycle of the app)? What about 
catalog management?


Could the new API calls take something like the Brooklyn YAML format for 
the app definitions, and for catalog items?


What is your reasoning on pros/cons of embedding Brooklyn versus running 
it as a stand-alone process (which could be automatically deployed as 
part of the CloudStack installation)? If it was a stand-alone app, one 
would interact with it via the REST api.


---
It's also worth mentioning OSGi. Brooklyn currently can be run as a 
vanilla Java process, or in Karaf. We're moving towards making more and 
more use of Karaf (e.g. it helps with versioning of catalog items, where 
those items include Java code or resources such as bash scripts uploaded 
to the Brooklyn server).


Does the place you embed Brooklyn run as an OSGi container? If not, then 
would that mean you'd want to stick with the non-karaf way of running 
the Brooklyn engine long-term?


Aled


On 24/05/2017 14:06, Daan Hoogland wrote:


Hello,

I am an Apache CloudStack developer and PMC member. At the last 
apachecon I have been looking for feedback on a functional 
specification for distributed applications [1]. So far, we have not 
gotten any feedback from outside the project but from within we have 
been searching our way around and have gotten to look at Brooklyn. 
Hope you guys`n’dolls have some patients with me.


What the CloudStack project aims to be is an integrated turnkey 
solution for entry level cloud deployments. That means we don’t want 
to burden users with extra install responsibilities as much as we can. 
Looking at clustered environments we don’t want people to have to 
install aria, Brooklyn or TerraForm to be able to orchestrate groups 
of related VMs or containers. On the other hand, we do not want to 
re-invent the wheel as we kind of did in the Functional Specification 
from the link above [1].


I know that Brooklyn supports CloudStack through a plugin but I am 
looking for a way to integrate the engine, that Brooklyn provides, 
into CloudStack so that writers of specific application plugins like 
ccs, a K8Cluster plugin [2], can write their environment description 
in a rather generic way and call the underlying CloudStack API to have 
the ‘applicationscape’ automagically created.


Do you people have any advice here?

[1] 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/ApplicationCluster+service


[2] https://github.com/shapeblue/ccs



Daan Hoogland
Senior Software Developer
*s:* +31 61400 4545**| *d: *+44(0) 20 3603 0540
*e:* daan.hoogl...@shapeblue.com |*w: *www.shapeblue.com |*t:*  @shapeblue
*a:* 53 Chandos Place, Covent Garden, London WC2N 4HSUK

CloudStack Collab Miami 2017 



Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue 
Services India LLP is a company incorporated in India and is operated 
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda 
is a company incorporated in Brasil and is operated under license from 
Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The 
Republic of South Africa and is traded under license from Shape Blue 
Ltd. ShapeBlue is a registered trademark.This email and any 
attachments to it may be confidential and are intended solely for the 
use of the individual to whom it is addressed. Any views or opinions 
expressed are solely those of the author and do not necessarily 
represent those of Shape Blue Ltd or related companies. If you are not 
the intended recipient of this email, you must neither take any action 
based upon its contents, nor copy or show it to anyone. Please contact 
the sender if you believe you have received this email in error.


Find out more about ShapeBlue and our range of CloudStack related 
services:
IaaS Cloud Design & Build 
 | CSForge – rapid 
IaaS deployment framework 
CloudStack Consulting  | 
CloudStack Software 

Re: Faster PR reviews

[Aled Sage]

k to an "eyeball test" a certain period of time (7 days 
max, less if it's simple?), triage the review to look at:


* clearly helpful & not obviously wrong
* low-risk / doesn't break compatibility
* good test coverage (and passing)
* likely to be maintained

If these can't be confirmed, the reviewer should say what they have 
doubts about, maybe suggest what the contributor could do to help, or 
appeal to other committers more familiar with an area. In any case get 
a discussion going.


If these do seem confirmed, I still suggest we don't merge immediately 
in the absence of a thorough review, but ping specific committers 
likely to be interested.  If no thorough review after a few more days, 
_then_ merge.


I'm not suggesting any heavyweight process, but just enough to put 
healthy forces on us as reviewers.


This is not a theoretical question, nor is it restricted to the YOML 
PR.  We're pretty good with most of our PRs and reviews but there are 
plenty of examples where we've dropped the ball.  Look at [1] which is 
tiny and tests-only and took nine days to get a review.  Or [2] which 
yes combines a few related-but-different things but is by no means a 
hard thing to review.  It would take far more time to split that up 
into 3 branches, test those locally, then babysit each of those PR's 
than it would take for a reviewer to just get on with a review.  It's 
been sitting there for 2 months and doesn't even have a comment.


This is not a good state of affairs.  Turning this discussion in to 
guidelines about PR's misses the point.  If there's any change to our 
docs/process made as a result of this discussion I'd like to see the 
eyeball test added to a review process discussion.


Finally re YOML, there is an ML thread started when the issue was 
raised.  There was chatter beforehand but it wasn't an easy thing to 
discuss until there was prototype code.  The point is for 7 months 
there have been no comments in any of these places, even after I've 
run a public session explaining it and private sessions and the PR 
itself says how it can be tested and how it is insulated from the rest 
of the code (Thomas I think you missed that point).  As there is an ML 
thread and an open issue, either of which would be a fine place to 
comment, but no one is -- the suggestion of a new separate ML thread 
to solve the problem is bizarre.  I say this is _exactly_ the 
situation when we need guidelines for how we handle PR's that are not 
being reviewed in a timely way.


Best
Alex


[1] https://github.com/apache/brooklyn-server/pull/600
[2] https://github.com/apache/brooklyn-server/pull/575



On 02/05/2017 19:21, Aled Sage wrote:

Hi Alex,

Interesting question. A few initial thoughts:

*YOML*
YOML (PR #363) is an exceptional case - we should not use that as an 
example when discussing this meta-question. The PR is 12,000 lines 
(including comments/notes), and was not discussed on the mailing list 
before it was submitted. I suggest we have a separate email thread 
specifically about merging that PR, as there are certainly very 
useful things we'd get from YOML.


*Small PRs*
We should strongly encourage small focused PRs on a single thing, 
wherever possible. That will make review faster, easier and lower 
risk. For such PRs, we should strive for review+merge within days (7 
days being an upper bound in normal circumstances, hopefully).


We can add some brief guidelines to this effect at 
http://brooklyn.apache.org/developers/how-to-contribute.html


*Changing low-level Brooklyn*
PRs that change low-level things in Brooklyn (e.g. changes to 
config-inheritance etc) deserve thorough review. They are high-risk 
as the unforeseen consequences of the changes can be very subtle, and 
break downstream blueprints that rely on old ways of doing things.


*Light-weight Review*
I agree with you - where PRs look sensible, low-risk and unit tested 
we should take more risk and merge them sooner (even if there's not 
been time for a thorough review by the community).


Aled


On 02/05/2017 15:50, Duncan Johnston Watt wrote:

Hi Alex

This is probably covered already but I guess there needs to be an 
impact
assessment (by submitter?) before something is waved through by 
default.


Best

Duncan

On 2 May 2017 at 06:52, Alex Heneveld <alex.henev...@cloudsoftcorp.com>
wrote:


Hi Brooklyners-

As many of you know, my YOML PR #363 [1] has been open for a while. 
This

sets up a foundation for giving better documentation and feedback and
hugely simplifying how we do our parsing.  However it's a very big 
PR.  I'm
eager to have people spend some time using it and ideally extending 
it --

but here I wanted to raise a meta-question:


*W**hat should we do with a PR when we aren't able to review things 
in as

much depth as we'd like?*


One option -- call it (A) -- is to say if we can't review things
thoroughly in a reasonable timeframe, we do a lighter review and if 
the PR

looks promising and safe we merge i

Re: [VOTE] New standards for PR reviewing.

[Aled Sage]

+1 (binding)


On 08/05/2017 11:55, Richard Downer wrote:

There have been recent discussions about how the committers assess PRs for
merging. The discussion is summarised below and the original thread
available at [1].

The consensus of the discussion is to adopt new standards for committers
reviewing PRs, as follows:

--

If a PR has not been reviewed within a certain amount of time - suggested
to be 7 days, or less for smaller PRs - nor has a committer indicated that
they are doing a detailed review, then the PR shall be considered for a
less detailed review, an "eyeball test".

Under an eyeball test, a reviewer will consider if the PR is:
* clearly helpful & not obviously wrong
* low-risk / doesn't break compatibility
* good test coverage (and passing)
* likely to be maintained

If it passes the above criteria, then the reviewer will add a comment to
the PR, and ask if further review is appropriate, possibly tagging specific
committers who may be interested. Then if there is no objection within 72
hours, passive consensus should be assumed, and the PR merged.

If the PR does not pass the above criteria, the reviewer should say what
they have doubts about, suggest what the contributor could do to help,
and/or appeal to other committers more familiar with an area. If
appropriate, move from GitHub onto the mailing list. (The aim here is to
get a discussion going and not give the contributor the impression their PR
is being ignored.)

--

This vote is to decide if we wish to adopt these standards for all PR
reviews going forward, and to document these standards in our website.

This vote will be open for a minimum of 72 hours.

[ ] +1 - adopt this standard
[ ] 0 - no opinion
[ ] -1 - do not adopt this standard, because:

--

Background:

This is related to the recent thread at [1].

Traditionally, this project has had a high bar for reviewing contributions
prior to merging. This dates back to the project's inception, before it was
part of Apache. Reviewers would be expected to inspect the code and
personally test it before allowing it to be merged.

There has been concern expressed that this is holding back Brooklyn
development. Reviewing a PR can be time-consuming; often a detailed review
requires expert knowledge in a particular area of the code which only some
committers possess. The result is that PRs, especially larger ones or ones
in core areas of the project, do not receive timely review, and in some
cases languish far too long. This is bad for the project as it holds back
our velocity, and frustrates contributors who see their changes stuck in
the system for extended lengths of time.

Since we joined the ASF, we have had feedback from others with experience
in Apache that we are too conservative with our code review requirements.
We also recognise the value in automated testing to catch regressions
(although these constantly need work, of course), and in our Git source
control to enable us to revert changes that turn out to be particularly
problematic. We can relax our strict reviewing requirements, which will
increase our velocity, and show our contributors that their work is
receiving attention and getting merged. Should a merge prove to be
problematic, their is still opportunity to do a bug fix (and get it merged
under the same fast process, too), and ultimately the chance to "revert"
the merge if necessary.

So we believe that the quality of the finished product will not be
adversely affected by these changes.


[1]
https://lists.apache.org/thread.html/4398448fd548495a5159016a97afa12dd787ab34786b3bbc0881d5b4@%3Cdev.brooklyn.apache.org%3E

Thanks
Richard.

Re: [VOTE] Accept contribution of Brooklyn Container Service

[Aled Sage]

+1 (binding)


On 31/05/2017 11:56, Andrew Kennedy wrote:

All,

- https://github.com/cloudsoft/brooklyn-container-service

This repository contains Brooklyn Container Service, which is a set of
locations and entities for Apache Brooklyn that allow deployment of
applications to container based locations. The locations provided
use the jclouds Docker provider and the fabric8 Kubernetes client to
give Brooklyn access to Docker Engine, Docker Swarm, Kubernetes and
OpenShift. As well as these locations, entities that represent Docker
containers and Kubernetes pods are included, which can be used to build
applications. Blueprints for containerised applications can also be
developed and deployed using the usual Brooklyn components. This is a
useful addition to Apache Brooklyn and allows the project to extend its
scope to include this essential part of the Cloud Native ecosystem.

Container Service was developed by Cloudsoft Corporation as part of its
AMP product over the last year, based on the earlier open source Clocker
codebase. There have been 14 contributors, making 379 commits in 50
files, with around 3.3 KLOC of Java. The commit being discussed here has
id `975807b89df63833d9b53b50d508cb5d782ad7b0` and tag `apache-brooklyn`,
and the code can be accessed at the following URLs:

- 
https://github.com/cloudsoft/brooklyn-container-service/tree/975807b89df63833d9b53b50d508cb5d782ad7b0
- 
https://github.com/cloudsoft/brooklyn-container-service/releases/tag/apache-brooklyn


This vote is to determine whether the Brooklyn PMC is in favour of
accepting the code contribution. If the vote passes, the PMC and the
authors of the code will work together to complete the ASF IP Clearance
process and merge the Container Service repository into Apache Brooklyn.

[ ] +1 : Accept contribution of Brooklyn Container Service
[ ]  0 : No opinion
[ ] -1 : Reject contribution because...

The vote is open for 72 hours and will close at 11h00 UTC/12h00 BST on
Saturday 03 June 2017 and the results will be announced on this list.

Thanks,
Andrew.
--

Andrew Kennedy ; Founder clocker.io  project ; 
@grkvlt ; Cloudsoft

Deprecating `@Catalog` annotation?

[Aled Sage]

Hi all,

I'd like to deprecate the `@Catalog` annotation [1], and the support for 
`scanFromAnnotations` [2].


Previously, we annotated some entity/policy Java classes with catalog 
information, such as descriptions and icon urls. However, we've moved to 
using .bom files as the way to define catalog items. Therefore the 
`@Catalog` is not normally used.


It is also supported/used by the `./brooklyn list-objects` CLI, to scan 
jars [3]. It find the annotated entities etc, gets the catalog metadata, 
and writes that out in json format. However, I don't think that's what 
we want to do moving forward. We should focus on getting the metadata 
from the .bom files, which is how things should be defined for the catalog.


Any objections to deprecating these?

Aled

[1] 
https://github.com/apache/brooklyn-server/blob/master/api/src/main/java/org/apache/brooklyn/api/catalog/Catalog.java
[2] 
https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/CatalogClasspathDo.java#L199-L209
[3] 
https://github.com/apache/brooklyn-server/blob/master/server-cli/src/main/java/org/apache/brooklyn/cli/ItemLister.java#L88-L94

Fixing BROOKLYN-534: service.state.expected value on error?

[Aled Sage]

Hi all,

TL;DR: low-level discussion of error-handling, and entity state; should 
we change what we set the entity's "expected state" to when there are 
errors?


---
I'm trying to fix https://issues.apache.org/jira/browse/BROOKLYN-534 
(when stopping, ensure the `ServiceRestarter` doesn't restart the 
entity, provisioning a new machine, in the situation where stop fails 
for some reason).


See the failing test I've added in [1].

The problem is that, when the `location.release(machine)` fails, it sets 
the entity's *expected state* to "on-fire".  This means we've lost all 
record of whether we wanted to be stopped or running. The 
ServiceRetarter therefore (sensibly?!) responds to the "on-fire" by 
restarting it!


Background: software prcoess entities normally have an "expected state" 
saying what state they want to be in, and an "actual service state" 
saying what it really is. The latter is inferred from a combination of 
the expected state, whether the service.isUp, and also any explicit 
"problem indicators" that have been set.


---
Options for fixing this include:

1. If the expected state is "on-fire", then don't do anything in such
   policies - instead rely on whatever set that expected state to deal
   with it (e.g. if something else called stop, which failed, then it
   should deal with the error itself).
2. Don't set expected state to on-fire. Instead keep the expected state
   as what we really want it to be, but include an appropriate "problem
   indicator" (e.g. to say that "stop" failed).

Option (1) would probably be easy, but it's unclear where you'd hook in 
logic to deal with such failures. It also feels wrong that we've lost 
all knowledge of what we were trying to do.


Option (2) feels best. However, this stuff is fiddly. We'd need to make 
the sure the "problem indicator" is cleared correctly when an entity 
tries again to start or stop. If it's not cleared, then the entity will 
likely never report itself as recovered from that problem. For example, 
see all the calls to ServiceStateLogic.setExpectedState() [3].


Is it enough to change [2] (to set a "problem indicator", and to set the 
expected state to "stopped"), and to clear that problem indicator in 
start [4] and restart [5]?


Before I go down this road, does anyone have comments, questions, advice 
or strong opinions?


Aled

[1] https://github.com/apache/brooklyn-server/pull/829
[2] 
https://github.com/apache/brooklyn-server/blob/master/software/base/src/main/java/org/apache/brooklyn/entity/software/base/lifecycle/MachineLifecycleEffectorTasks.java#L869 
(in the doStop method's catch block)
[3] 
https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/entity/lifecycle/ServiceStateLogic.java#L158
[4] 
https://github.com/apache/brooklyn-server/blob/master/software/base/src/main/java/org/apache/brooklyn/entity/software/base/lifecycle/MachineLifecycleEffectorTasks.java#L339
[5] 
https://github.com/apache/brooklyn-server/blob/master/software/base/src/main/java/org/apache/brooklyn/entity/software/base/lifecycle/MachineLifecycleEffectorTasks.java#L661

Re: UsageListener: notified of promotion to master?

[Aled Sage]

Hi all,

I finally got round to implementing this! See PR at 
https://github.com/apache/brooklyn-server/pull/832


Aled


On 22/03/2017 19:52, Svetoslav Neykov wrote:

It feels like (2) is a more generally useful feature to add (for programmatic 
support, at least). Does it sound useful for things beyond my use-case?

Yes. I needed this exact functionality last week - for a piece of code shared 
between all applications. I ended up writing a poller thread that monitors the 
state.
I think it will be a useful thing to add.

Svet.



On 22.03.2017 г., at 19:11, Aled Sage <aled.s...@gmail.com> wrote:

Hi all,

I'm writing a UsageListener [1] that is going to record the Brooklyn instance's 
apps in an external inventory. This is mostly straight-forward (except needing 
the fix at [2]).

The tricky bit is when a Brooklyn standby instance is promoted to master - I 
want my listener to be notified so it can record the identity of the new master 
Brooklyn for each of the pre-existing app.

I don't see anywhere I can get a notification of this. Is there?

---

Possible solutions include:

1. Add an additional application event (e.g. state
   "MANAGEMENT_CHANGED") - the UsageListener would receive this event
   for every app when it successfully rebinds.
2. Register a new kind of listener, to be notified of the management
   context's state - on becoming "master", my listener could iterate
   over all the apps.

I don't particularly like either of those options.

For (1), it isn't really to do with app "Usage" like the other events.

For (2), my listener code would be fiddly - I'd need to worry about receiving 
concurrent notifications via the normal UsageListener while I'm iterating over 
the pre-existing apps (assuming the new listener is called asynchronously, 
rather than blocking the management context's startup).

It feels like (2) is a more generally useful feature to add (for programmatic 
support, at least). Does it sound useful for things beyond my use-case?

Thoughts?

Aled

[1] 
https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/mgmt/usage/UsageListener.java
[2] https://github.com/apache/brooklyn-server/pull/606

Re: REST API: bundles + types + subtypes endpoints

[Aled Sage]

riends:

/v2/catalog?category=application
/v2/catalog?category=entity
/v2/catalog?category=policy
/v2/catalog?category=enricher
/v2/catalog?category=location

Doubt 'category' is quite right but something appropriate.

Geoff

On Wed, 20 Sep 2017 at 18:37 Aled Sage <aled.s...@gmail.com> wrote:


Hi all,

For `/subtypes` versus `/types`, it's not really about code 
duplication.

It's that the API is unclear because it's not obvious what the
difference/similarity is. If I saw two endpoints like that (without
looking at the code), I'd assume they were for different things. If I
saw query parameters, I'd immediately understand that it's a filtering
mechanism.

I think the right api name is `/catalog` (for what you've called
`/types`). However, deprecating what's there and adding this at the 
same

endpoint will likely be confusing.

---
There's a pressing need for `/bundles` (so a user can delete a bundle),
whereas `/types` is a nice-to-have. Do you agree?

To me, the `/types` is a version 2 of the catalog (no matter what we
call it).

Thinking of how best to get this into the product in a way we want to
support for a long time, the ideal would be to mark it "beta" and have
it versioned. Other folk (e.g. GCE) do things like `/v2beta/`. I think
that `/v2...` is the right solution from a process, api-design and
understandability perspective.

My feeling is we should defer the `/types`, adding it as part of a
bigger `/v2beta` effort when we're ready to invest that time. This
includes updating the UI and getting sufficient feedback from 
downstream

projects that the new API is right.

---

To Geoff's comment about the type registry being sufficient different
from catalog, I'm not sure about that. Personally I just stretched what
I think of as the catalog, and thus it is a catalog v2 which supports
more things.

I think if "catalog" disappeared, being replaced by "types", it 
would be

more confusing than having a new v2 catalog that does more.

Aled


On 20/09/2017 17:37, Geoff Macartney wrote:

My two cents -

0.  Keep /bundles API PR separate from /type and do /bundles now
1. While I like '/catalog' for being more meaningful than '/type', I

think

what the type registry does is sufficiently different from the classic
catalog that it needs a new name in the API.
1.1. /type would be my preference to /types
2. I'd deprecate catalog when we introduce /v2 (see later)
3. I'd have said do /types now or very soon but put it under a 
(clearly
beta) /v2; however I take Aled's point about clutter-free. Not sure 
how

best otherwise to indicate its beta status; possibly /beta/type?  And

then

migrate to /v2/type at the later time?
4. The only real use of /subtype is to have an easy way to say
'listApplications', 'listEntities', 'listPolicies', 'listEnrichers',
'listLocation'.  I don't think this merits a separate endpoint, and in
particular I would put SubTypeApi.list into /type as
/type/?super=whatever.  As for 'listPolicies' and friends, I would 
have
that as an extra query parameter (not 'super') on /type.  I don't 
really

quite know what to call it! What name should we use to distinguish

between

things that are Applications, Entities, Locations, Policies, and

Enrichers.

G



On Wed, 20 Sep 2017 at 16:31 Robert Moss <robert.m...@cloudsoft.io>

wrote:

1) like the `/catalog` name and agree with Aled about its power to
communicate the idea
2 & 3) run `/v1` and `/v2` in parallel, with the latter being

experimental

and iterative until a 1.0 release
4) no preference

Robert

On 20 September 2017 at 15:45, Alex Heneveld <
alex.henev...@cloudsoftcorp.com> wrote:


Aled, Thomas, Mark - thanks for the good feedback.

All - There are some good suggestions which I agree deserve 
discussion.

Specific points.

(1) Should /types and /bundles be top-level or under a /catalog 
prefix

?

Thomas suggested the latter which also fits.  My reason for doing

top-level

is simply that most REST APIs these days seem to have more top-level
resources.  /catalog is not necessary and in some ways it's cleaner

having
separate endpoints.  On the other hand the /catalog prefix is 
nice to

orient consumers, as Aled says:  `bundles` and `types` on their own

aren't

as self-evident.  And it means we could continue to have `POST

/catalog`

as

the main way to install.

(2) Should we deprecate `/catalog` ?  Thomas suggests maybe not 
yet.  I

think we should as equivalent functionality and more is available at

the

new endpoints.  (Though if we go with (1), obviously we wouldn't

deprecate
the whole endpoint, just the old calls.).  Also worth noting:  I 
don't

think we should remove the deprecated endpoint anytime soon however.

(3) Should we switch to /v2 ?  Aled has suggested we do as the 
generic

`types` support is a significant shift from the old more restrictive
`catalog`.  I don't think we should yet, however:  I'd prefer to 
make

that

move when we are ready to remove all old endpoin

Re: REST API: bundles + types + subtypes endpoints

[Aled Sage]

n

noted

the same functionality as `/subtypes/entity` could be done with
`/types?super=entity` (with appropriate query paramter).  My reasoning

for

making it a separate endpoint is that this is an activity I expect people
will want to do a lot, avoiding the query parameter makes for a slightly
nicer API.  But it is repetition in the code.

(There are some other minor issues but I think I agree with the points

made

there.)

My answers:

(1) slight preference for the `/catalog` prefix
(2) strong pref to deprecate old calls - they are redundant and multiple

is

confusing
(3) strong pref to stay with `/v1` for now
(4) slight pref for explicit `[/catalog]/subtypes` endpoint

Best
Alex


On 20 September 2017 at 12:38, Aled Sage <aled.s...@gmail.com> wrote:


Thanks Alex.

As per my comment in the PR at https://github.com/apache/broo
klyn-server/pull/810#issuecomment-330824643.

---
TL;DR:
Given this is a big API change and given I'm suggesting a `/v2` REST

api

then I wanted to raise it on the list as well.

I propose we split this PR into two. The `/bundles` part we can merge
pretty quickly. However, the `/types` and `/subtypes` is too

controversial

in my opinion - it probably deserves a `/v2/` of the REST api.

We can continue detailed discussion in the PR.

---
I don't want to lose the word "catalog" in the REST api - it's so good

at

getting across the meaning for users! The alternative `/type` is just

not

as good, in my opinion.

The multiple endpoints of `/types` and `/subtypes` is confusing. I'd

model

the latter as just a filter on `/type`. It would be better achieved

with

an

additional query parameter rather than a separate endpoint.

If designing a `/v2` REST api, we could use `/catalog` instead of

`/type`.

However, it will likely take a while to get to a stable and good `/v2`

api.

There are other cleanup/improvements we should probably do to the REST

api

if we're releasing a new version of it (e.g. exclude the deprecated

stuff,

get rid of `/locations` but figure out if we really need to support
locations from brooklyn.properties, find out from the community about

other

inconsistencies or hard-to-understand parts of the api).

The meaning of `GET /subtypes/application` looks completely different

from

`GET /catalog/applications`. The latter retrieves the catalog items

marked

as `template`, but the new api returns everything that implements
`Application`. Perhaps this is an opportunity to get rid of the

"entity"

versus "template" distinction (at least in the REST api). The original
meaning/intent of "template" has entirely changed / been misused! I

believe

it was originally intended as a partially-complete YAML blueprint that
someone would retrieve via the REST api, and then modify. They'd then

POST

their .yaml file to deploy their app. It has now been used as the list

of

apps to include in a "quick launch" view. If designing a new `/v2` api,

I'd

explicitly support a "quick launch" list and would get rid of the
"template" versus "application" versus "entity" distinction in the REST

api

(anywhere you can use an entity, you can use an app; anywhere you need

an

app then a non-app entity will be auto-wrapped in a basic-app).

Thoughts?

Aled


On 07/09/2017 17:26, Alex Heneveld wrote:


Hi team-

As mentioned earlier, I've been working on adding bundle support to

the

REST API, so we can add/remove/query bundles.  And related to this,

and

the

type registry, is the ability to add arbitrary types but until now

there

was no way to query those, so there are endpoints for types/ and
subtypes/.  This is in #810 [1].

In brief you have:

*GET /bundles* - list bundle summaries
*POST /bundles* - add ZIP or BOM YAML
*GET /bundles/com.acme/1.0* - get details on a specific bundle
*DELETE /bundles/com.acme/1.0* - remove a bundle

*GET /types* - list all types (optionally filter by regex or fragment)
*GET /types/acme-entity/1.0* - get details on a specific type
*GET /subtypes**/entity* - list all entities (optionally filter by

regex

or fragment); same for many other categories


A full list including arguments is shown in the PR.

Another good thing about this besides bundle-centric management and
deletion in particular is that it entirely replaces the "catalog/"

endpoint

allowing us to deprecate it.  I expect we'll keep it around for a

while

as

clients (the UI, CLI) still use it but we now have equivalent methods

that

are better aligned to how we do things with bundles.  They're also

quite a

bit faster so if you've gotten bored waiting for catalog to load this
should help (when clients are updated).  And one final benefit, we can

now

register and explore other types eg custom task types, predicates, and

more.

One thing to note is that we have fewer and simpler REST objects using
freeform maps where we return extended type info -- eg config on

entitie

Re: REST API: bundles + types + subtypes endpoints

[Aled Sage]

Thanks Alex.

As per my comment in the PR at 
https://github.com/apache/brooklyn-server/pull/810#issuecomment-330824643.


---
TL;DR:
Given this is a big API change and given I'm suggesting a `/v2` REST api 
then I wanted to raise it on the list as well.


I propose we split this PR into two. The `/bundles` part we can merge 
pretty quickly. However, the `/types` and `/subtypes` is too 
controversial in my opinion - it probably deserves a `/v2/` of the REST api.


We can continue detailed discussion in the PR.

---
I don't want to lose the word "catalog" in the REST api - it's so good 
at getting across the meaning for users! The alternative `/type` is just 
not as good, in my opinion.


The multiple endpoints of `/types` and `/subtypes` is confusing. I'd 
model the latter as just a filter on `/type`. It would be better 
achieved with an additional query parameter rather than a separate endpoint.


If designing a `/v2` REST api, we could use `/catalog` instead of 
`/type`. However, it will likely take a while to get to a stable and 
good `/v2` api. There are other cleanup/improvements we should probably 
do to the REST api if we're releasing a new version of it (e.g. exclude 
the deprecated stuff, get rid of `/locations` but figure out if we 
really need to support locations from brooklyn.properties, find out from 
the community about other inconsistencies or hard-to-understand parts of 
the api).


The meaning of `GET /subtypes/application` looks completely different 
from `GET /catalog/applications`. The latter retrieves the catalog items 
marked as `template`, but the new api returns everything that implements 
`Application`. Perhaps this is an opportunity to get rid of the "entity" 
versus "template" distinction (at least in the REST api). The original 
meaning/intent of "template" has entirely changed / been misused! I 
believe it was originally intended as a partially-complete YAML 
blueprint that someone would retrieve via the REST api, and then modify. 
They'd then POST their .yaml file to deploy their app. It has now been 
used as the list of apps to include in a "quick launch" view. If 
designing a new `/v2` api, I'd explicitly support a "quick launch" list 
and would get rid of the "template" versus "application" versus "entity" 
distinction in the REST api (anywhere you can use an entity, you can use 
an app; anywhere you need an app then a non-app entity will be 
auto-wrapped in a basic-app).


Thoughts?

Aled


On 07/09/2017 17:26, Alex Heneveld wrote:


Hi team-

As mentioned earlier, I've been working on adding bundle support to 
the REST API, so we can add/remove/query bundles.  And related to 
this, and the type registry, is the ability to add arbitrary types but 
until now there was no way to query those, so there are endpoints for 
types/ and subtypes/.  This is in #810 [1].


In brief you have:

*GET /bundles* - list bundle summaries
*POST /bundles* - add ZIP or BOM YAML
*GET /bundles/com.acme/1.0* - get details on a specific bundle
*DELETE /bundles/com.acme/1.0* - remove a bundle

*GET /types* - list all types (optionally filter by regex or fragment)
*GET /types/acme-entity/1.0* - get details on a specific type
*GET /subtypes**/entity* - list all entities (optionally filter by 
regex or fragment); same for many other categories


A full list including arguments is shown in the PR.

Another good thing about this besides bundle-centric management and 
deletion in particular is that it entirely replaces the "catalog/" 
endpoint allowing us to deprecate it.  I expect we'll keep it around 
for a while as clients (the UI, CLI) still use it but we now have 
equivalent methods that are better aligned to how we do things with 
bundles.  They're also quite a bit faster so if you've gotten bored 
waiting for catalog to load this should help (when clients are 
updated).  And one final benefit, we can now register and explore 
other types eg custom task types, predicates, and more.


One thing to note is that we have fewer and simpler REST objects using 
freeform maps where we return extended type info -- eg config on 
entities, policies, etc, sensors and effectors on entities.  I'd like 
to use the same pattern for returning data on adjunct instances so 
that we can support policies, enrichers, and feeds in a consistent way 
(removing duplication there).  This should tie in with Graeme's 
highlights work.


Follow-on work will see the CLI updated to allow `br bundle delete 
com.acme:1.0` and similar.  No immediate plans to put lots of bundle 
info into the UI as bundle devs are probably comfortable with the CLI 
but if anyone would like that speak up.  I have updated UI to _show_ 
the containing bundle ([2], also needs review!).


Best
Alex

[1]  https://github.com/apache/brooklyn-server/pull/810
[2]  https://github.com/apache/brooklyn-ui/pull/48



On 07/09/2017 14:58, Alex Heneveld wrote:


+1 to this, with Thomas's suggestion of a list instead of map and 
Geoff's suggestion of doing it on 

[PROPOSAL] Always load initial catalog, and then add persisted catalog on top

[Aled Sage]

Hi all,

I want to change how Brooklyn populates its catalog at startup. I want 
it to install the "initial catalog", and then add the items from 
persisted state.


See the PR at [1].

---
Currently, if there is any persisted state then we ignore the "initial 
catalog" and use only the persisted state.


However, that causes problems when upgrading Brooklyn. We have to tell 
the user [2] to start Brooklyn and then to manually add the "initial 
catalog" by running:


   br catalog add /opt/brooklyn/catalog/catalog.bom

---
One reason we did it that way was in case the user deliberately deleted 
something from that initial catalog. By relying on only the persisted 
state, we don't re-add what they removed.


However, that feels like optimising for the wrong thing: we make things 
harder for everyone (i.e. they all have to run the extra `br catalog 
add` command on upgrade) in case some people wanted to change what was 
in the initial catalog.


---
Much longer term, I want to separate out the management and upgrade of 
the catalog from that of an AMP server. As an enterprise user, when 
upgrading Brooklyn (e.g. from 0.12.0 to 1.0.0) I would like to get my 
catalog ready for use with the new version before attempting to start up 
all my entities. This is particularly important if the catalog contains 
any Java code compiled against against the old Brooklyn version 
(especially if it's the user's own custom code). However, that is a much 
bigger topic that needs more thought. I plan to write up a proposal for 
that at some point.


Aled

[1] https://github.com/apache/brooklyn-server/pull/852
[1] https://brooklyn.apache.org/v/latest/ops/upgrade.html

Re: Apache Brooklyn 1.0 release

[Aled Sage]

Hi all,

I approach this question as "what is essential for a 1.0 release", 
versus nice-to-have. I think pretty much every feature/change is 
nice-to-have and can be developed incrementally - if it's ready in time 
for a 1.0 then great, otherwise it will go in 1.x or 2.x.


I personally have treated each 0.x Apache Brooklyn release as being 
suitable for production use, because there have been enterprises using 
it in production! As a community, we've been behaving as though it is a 
1.x (with our deprecation policy, our attitude to backards 
compatibility, etc).


---
To answer the individual comments/questions.

**Basic Auth**
I agree we want it, but it's not a blocker. Enterprises have been 
getting by without it, e.g. using the ldap integration [1].


Let's move the basic-auth conversation into a separate "PROPOSAL" 
thread, and see if contributors in the community get it done in time for 
a 1.0 release.


**Feature Complete**
I believe we are "feature complete enough" for a 1.0.0.  There are no 
missing features that block our existing production users, but there are 
certainly things they'd like to see added incrementally.


We will never be "feature complete". We will always want to improve 
Brooklyn, and we should do that incrementally with subsequent 1.x releases.


**Great usability**
I'm sure usability could be improved. However, I think it's good enough 
for a 1.0, and can be improved incrementally.


**freeze our API**
Our deprecation policy and approach to backwards compatibility means we 
are already very careful in how we change our API.


The API will not be "frozen" - but it is stable.

**accept a stricter deprecation policy**
The deprecation policy was discussed in [2], but never formally 
documented on our web-site!


We have been conservative (treating it like a 1.x): we don't delete 
public api stuff without first deprecating, and we wait at least a 
couple of releases before deleting deprecated code.


What constitutes our "public api" can be a little blurred. Some 
downstream projects and power users have hooked into things that were 
not originally thought of as "public" (but not clearly marked as 
"internal" either). We have therefore been conservative about 
deprecating and deleting those as well.


**anything to deprecate before 1.0**
I think that's the same question as we ask for every release - it being 
1.0 is no different.


**removed every deprecated thing that we can**
If something was only deprecated in 0.12, I don't think we should delete 
it in 1.0. We should keep it for a minimum length of time as well so 
that upgrades are relatively pain-free for our users. Some enterprises 
are very conservative about upgrade (e.g. having an internal upgrade 
cycle, only taking new versions every 3 or 6 months).


To me, it's the same question as for every release: have we deleted old 
deprecated code.


**Do we have great documentation**
I'd reword this as "is our documentation good enough?"

We should certainly run through the tutorials, examples etc to make sure 
it's accurate - but that applies to every release.


Docs and website can and should be improved incrementally, rather than 
requiring a big bang for 1.0.


**Do we have a great website**
I'd reword that as "is our website good enough?"

**Will the blueprints in the wider community be compatible**
Yes - based on our existing approach to deprecation and backwards 
compatibility, we should not be breaking them when releasing 1.0!


**media**
Yes!

[1] 
https://github.com/apache/brooklyn-server/blob/master/rest/rest-resources/src/main/java/org/apache/brooklyn/rest/security/provider/LdapSecurityProvider.java
[2] http://markmail.org/message/ogdq2xyio7rjgs56, e-mail thread 
"Deprecation guarantees/policy?"

Re: [VOTE] Release Apache Brooklyn 0.12.0 [rc3]

[Aled Sage]

+1 (binding)

I focused mainly on upgrade and rebind testing:

 * I deployed a simple app to AWS, which I susbsequently used for testing.
 * Tested upgrade from 0.11.0 to 0.12.0-rc3 on CentOS 7.
   (I installed 0.11 and deployed an app, then followed the
   instructions in https://github.com/apache/brooklyn-docs/pull/212 to
   upgrade.)
 * Tested upgrade again, where 0.11's catalog included a .bom that
   referenced a bundle.
 * Restarted Brooklyn a few times (rebinding to its persisted state).

I saw https://issues.apache.org/jira/browse/BROOKLYN-539 (a benign 
exception is logged), but given it is benign we can live with that for 
this release.


Aled


On 22/09/2017 10:33, Duncan Godwin wrote:

This is to call for a vote for the release of Apache Brooklyn 0.12.0.

This release comprises of a source code distribution, and a corresponding
binary distribution, and Maven artifacts.

The source and binary distributions, including signatures, digests, etc. can
be found at:

   https://dist.apache.org/repos/dist/dev/brooklyn/apache-brooklyn-0.12.0-rc3

The artifact SHA-256 checksums are as follows:

   f748ff53ea665a0cabcf42df4ecb346d0f688574286e8a251b37dec282b63a47
*apache-brooklyn-0.12.0-rc3-1.noarch.rpm
   54c85747d3622cac5842adcc738f93da9d160cd8a11f8903583c86363c8a6c4e
*apache-brooklyn-0.12.0-rc3-bin.tar.gz
   f63f6a950d8ea40694988fde3575653653c63808bf9311cbc6bf8c074e28f2b1
*apache-brooklyn-0.12.0-rc3-bin.zip
   de8c2098594edd7e2b60a4a6a44d8b957a7c1b9e7498015e27571e7835054e16
*apache-brooklyn-0.12.0-rc3-classic.tar.gz
   5a913a085e0425f38c50e6a117d0aed7f0624f0f528fc63b5d68266c6036046d
*apache-brooklyn-0.12.0-rc3-classic.zip
   2e3f2acba804b81efa74f71fa48e90965503e8eaf50aecc2b0d14a6f4a08f31a
*apache-brooklyn-0.12.0-rc3-client-cli-linux.tar.gz
   9fef1a73117f568ffeb41da6e4d4cb8ed895b36305df079289f31b3b874d5e4c
*apache-brooklyn-0.12.0-rc3-client-cli-linux.zip
   54004fcec86ff22cea23d460c7e25c243d3a1f67227931adb0067a8611bc1ee3
*apache-brooklyn-0.12.0-rc3-client-cli-macosx.tar.gz
   b2e26523bee04420dadac73989a3265360b30f4041c68ed364c4aaf10e0addee
*apache-brooklyn-0.12.0-rc3-client-cli-macosx.zip
   86a7848f1f1eb81eaa20116df2caa2d80409515a1268b085f6fde828b18b08ac
*apache-brooklyn-0.12.0-rc3-client-cli-windows.tar.gz
   36f14ecbced097561ef9621a28beba6343d96402424974a302560bb429b2685f
*apache-brooklyn-0.12.0-rc3-client-cli-windows.zip
   ed0e89f635501adc8cc347d22a412f29b03b3a4f9967911381e21fd3bc2333e9
*apache-brooklyn-0.12.0-rc3-src.tar.gz
   0764a014905ead1901d9a7074e90b5acde21c98137af25c4ef543f750bc05cf1
*apache-brooklyn-0.12.0-rc3-src.zip
   5fb368d12fb0f2a0a58f112b8bd39465c39827056b3de7d0dc86a40043f30c48
*apache-brooklyn-0.12.0-rc3-vagrant.tar.gz
   518e6eb47a0faaa99f1c2eb74167bfeb107ad8f332adf80fa7bc897990684a8b
*apache-brooklyn-0.12.0-rc3-vagrant.zip

The Nexus staging repository for the Maven artifacts is located at:


https://repository.apache.org/content/repositories/orgapachebrooklyn-1051

All release artifacts are signed with the following key:

 https://people.apache.org/keys/committer/drigodwin.asc

KEYS file available here:

 https://dist.apache.org/repos/dist/release/brooklyn/KEYS

The artifacts were built from git commit IDs:

brooklyn: 473deebda6e214fd1f13187563aeea5219fceef8
brooklyn-client: f31e24a9daa5be3464b62e171603a6c6eae3
brooklyn-dist: 416ebc77473452e26f95a41c771feabf138979db
brooklyn-docs: 6184fe07eaf9d69198934c92ad198a7d76dcbfdd
brooklyn-library: 73b199771e5baa6681afe89d6cc0cdcaad620afb
brooklyn-server: 4268b016191f3a5f049a2c9baa0e1befb8defc22
brooklyn-ui: bc5be4430fd83ebc6819113f1393f1231475ff76

All of the above have been tagged as "apache-brooklyn-0.12.0-rc3"

Please vote on releasing this package as Apache Brooklyn 0.12.0.

The vote will be open for at least 72 hours.
[ ] +1 Release this package as Apache Brooklyn 0.12.0
[ ] +0 no opinion
[ ] -1 Do not release this package because ...


Thanks!

Duncan


CHECKLIST for reference

[ ] Download links work.
[ ] Binaries work.
[ ] Checksums and PGP signatures are valid.
[ ] Expanded source archive matches contents of RC tag.
[ ] Expanded source archive builds and passes tests.
[ ] LICENSE is present and correct.
[ ] NOTICE is present and correct, including copyright date.
[ ] All files have license headers where appropriate.
[ ] All dependencies have compatible licenses.
[ ] No compiled archives bundled in source archive.
[ ] I follow this project’s commits list.

Re: Apache Brooklyn 1.0 release

[Aled Sage]

+1


On 03/10/2017 16:33, Duncan Godwin wrote:

Hi All,

Following the release last week of Apache Brooklyn 0.12.0 I propose that we
make the next version of Apache Brooklyn 1.0.0.

Apache Brooklyn is robust, stable, feature rich and being used in
production by multiple enterprises. Our deprecation policy means we haven't
treated it like a 0.x release in a very long time.

With 0.12.0, we did the last major thing needed before a 1.0 release: we
switched to Karaf as the primary distribution and we deprecated the
"classic mode". What does everyone think?

Many thanks

Duncan

Re: Proposal: delete `--catalogReset`?

[Aled Sage]

p.s. And also *delete* `--catalogAdd` from classic (again it's not 
supported in Karaf).


We sould be encouraging people to use `br catalog add ...` after 
Brooklyn is started, or updating the .bom file at their 
`default.catalog.location`, rather than using this command line option.


Aled


On 26/09/2017 09:53, Aled Sage wrote:

Hi all,

TL;DR: I propose we *delete* support for `--catalogReset`, for the 
next release. It's not supported in karaf mode, and it's dangerous 
(could break rebind of entities)!


Note that I'm not suggesting deprecate. We've moved to karaf as the 
default in 0.12 release, so effectively the functionality has already 
disappeared (in the recommended way of running Brooklyn).


---

In classic mode, we have support for `--catalogReset`. This works in 
combination with `--rebind ...` to tell Brooklyn that it should delete 
the persisted catalog (reseting it to the default initial catalog), 
while keeping the rest of the persisted state as-is.


We don't support this in Karaf mode, and no-one has missed it.

It is dangerous: if you have an entity instance then that will be 
persisted along with a catalogItemId, but resetting the catalog might 
cause that catalog item to disapear. This can cause serious problems 
when we try to rebind that entity instance (because we've lost type 
information).


It is better to explicitly manage the catalog via 
additions/deprecations/removals.


This is yet another example of where we can (slightly) simplify our 
startup/catalog code by removing unnecessary functionality.


---

Slightly longer term, we want to rethink how we load the catalog (from 
"initial" and persisted state): rebind is confusing when upgrading 
Brooklyn, because it keeps the persisted catalog and doesn't 
automatically add any of the catalog items/versions for the new 
Brooklyn version. One has to do that addition manually.


But that will be a topic for a new email thread!

Aled

Proposal: delete `--catalogReset`?

[Aled Sage]

Hi all,

TL;DR: I propose we *delete* support for `--catalogReset`, for the next 
release. It's not supported in karaf mode, and it's dangerous (could 
break rebind of entities)!


Note that I'm not suggesting deprecate. We've moved to karaf as the 
default in 0.12 release, so effectively the functionality has already 
disappeared (in the recommended way of running Brooklyn).


---

In classic mode, we have support for `--catalogReset`. This works in 
combination with `--rebind ...` to tell Brooklyn that it should delete 
the persisted catalog (reseting it to the default initial catalog), 
while keeping the rest of the persisted state as-is.


We don't support this in Karaf mode, and no-one has missed it.

It is dangerous: if you have an entity instance then that will be 
persisted along with a catalogItemId, but resetting the catalog might 
cause that catalog item to disapear. This can cause serious problems 
when we try to rebind that entity instance (because we've lost type 
information).


It is better to explicitly manage the catalog via 
additions/deprecations/removals.


This is yet another example of where we can (slightly) simplify our 
startup/catalog code by removing unnecessary functionality.


---

Slightly longer term, we want to rethink how we load the catalog (from 
"initial" and persisted state): rebind is confusing when upgrading 
Brooklyn, because it keeps the persisted catalog and doesn't 
automatically add any of the catalog items/versions for the new Brooklyn 
version. One has to do that addition manually.


But that will be a topic for a new email thread!

Aled

Re: Apache Brooklyn Architecture

[Aled Sage]

Hi Taylor, thanks for getting in touch.

Improving the docs would be awesome!

Is there a use-case you have in mind as well, or is it to just get a 
better understanding of how Brooklyn works?


---
See 
http://brooklyn.apache.org/v/latest/blueprints/java/entities.html#softwareprocess-lifecycle 
for a description of the steps.


The description at that page could do with updating - e.g. it is missing 
the "latches".


Also, the pages could do with a re-vamp to ensure people are pointed to 
the yaml way whenever possible, with the underlying Java being described 
for those who want to look under the covers or write entities in Java 
for whatever reason.


Note that those using YAML don't need to know most of this, but they do 
need to know the order in which their commands will be executed.


---
I can give you some additional pointers/info now.

Let's start with a simpler entity type - e.g. 
org.apache.brooklyn.entity.stock.BasicStartable - which will illustrate 
how things wire up.


   location: localhost
   services:
  - type: org.apache.brooklyn.entity.stock.BasicStartable
    brooklyn.config:
  foo: bar
    brooklyn.enrichers:
  - type: org.apache.brooklyn.enricher.stock.Transformer
    brooklyn.config:
  ...

Brooklyn looks up "org.apache.brooklyn.entity.stock.BasicStartable" in 
the catalog, and instantiates that entity. Under the covers, the YAML 
parser creates an EntitySpec, then calls 
managementContext.getEntityManager().createEntity(spec).


The keys and values under brooklyn.config get turned into 
org.apache.brooklyn.config.ConfigKeys (which are set via the spec). In 
the subsequent entity, these values are then available in the Java model 
via entity.config().get(...).


When the entity is instantiated (after its config has been populated 
etc), the entity.init() method is called. After that, the entity is 
offiically "managed" by the management context.


The enrichers listed above are instantiated in a similar way (i.e. the 
yaml parser creates an EnricherSpec, and then calls 
managementContext.getEntityManager().createEnricher(spec). The enricher 
then has its enricher.setEntity(entity) method called automatically, 
which is where it wires itself up (e.g. subscribes to sensors, etc).


The location is instantiated (see the code 
org.apache.brooklyn.api.location.LocationRegistry, LocationSpec, and 
managementContext.getLocationManager().createLocation(spec)).


When this yaml blueprint is deployed, it automatically calls the 
start(locations) effector, if it exists. Effectors can either be 
declared in Java entities or can be defined in the YAML. Where the given 
effector is a simple annotated method, that method will be called 
(wrapped in a "task" so that it is shown in the activity view).


---
Now let's switch to your VanillaSoftwareProcess. e.g.:

   location: localhost
   services:
  - type:
   org.apache.brooklyn.entity.software.base.VanillaSoftwareProcess
    brooklyn.config:
  install.command: echo installing
  launch.command: echo launching
  checkRunning.command: echo checking

As with the simpler entity example, the start(locations) effector is 
called. This delegates to 
softwareProcessDriverLifecycleEffectorTasks.start(), which executes a 
sequence the sequence of actions as sub-tasks (based on the config you 
have supplied).


---
You asked above the choice of driver. See 
http://brooklyn.apache.org/v/latest/blueprints/java/entity.html#key-steps


In hindsight, this is definitely architectural/plugability overkill! 
Almost everything uses just one driver implementation. One can configure 
it explicitly, but the "convention over configuration" is to implement 
getDriverInterface() to return the driver interface (e.g. 
VanillaSoftwareProcessDriver.class), and then for it to reflectively 
instantiate the expected driver class name by changing he suffix to 
SshDriver (e.g. VanillaSoftwareProcessSshDriver).


Aled


On 23/08/2017 16:49, Taylor wrote:

Hello,


For the last few weeks I have been reading the documentation and source code 
trying to build a mental model for how brooklyn works. This started when I 
began looking at writing a custom entity and the VanillaSoftwareService. I want 
to understand how the tiny bit of YAML gets translated into the java objects on 
the brooklyn host or remote host. For example, where is the main entry point, 
how is the driver selected for an entity, when an enricher is invoked how does 
that reach a method on the driver.


 From looking at the source code I have discovered various objects and processes but do 
not yet have a "start to finish" idea of what goes on. I have read through the 
CAMP specs and the brooklyn site and not found a diagram or comments in code.


I would like to work with some experienced developers to fill in my blanks and 
possibly add some comments/content/diagrams to the site.


Is anyone knowledgeable in this subject?


Thanks,

[PROPOSAL] Delete "classic-mode" (always use Karaf)

[Aled Sage]

Hi all,

I propose that we *delete* Brooklyn classic-mode from master now, in 
preparation for the 1.0.0 release.


---

In 0.12.0, we switched the main distro to be the karaf-mode. We also 
built the classic-mode distro - the intent being for users to have a 
usable classic mode, rather than being forced immediately to switch to 
karaf without advanced warning.


However, we unfortunately did not deprecate classic-mode as clearly as 
we should have (e.g. didn't explicitly say that it will be deleted in an 
upcoming release, and didn't deprecate the `Main` class).


I think it's still ok to delete it for several reasons:

1. This is a "mode" of running Brooklyn, rather than underlying
   Brooklyn functionality.
2. The `Main` class [1] etc should not be considered part of our
   *public* API (even though we didn't explicitly tell people that it
   was internal).
3. As part of 0.12.0 release, we added to the docs instructions for
   upgrading to Karaf [2].
4. Supporting classic to the same standard as Karaf will become
   increasingly painful as we do more and more with Bundles!
5. It's a major release, so if we are going to delete it then 1.0.0 is
   the perfect time!

Aled

[1] 
https://github.com/apache/brooklyn-server/blob/master/server-cli/src/main/java/org/apache/brooklyn/cli/Main.java


[2] 
http://brooklyn.apache.org/v/0.12.0/ops/upgrade.html#upgrade-from-apache-brooklyn-0110-and-below

Re: Deprecating `@Catalog` annotation?

[Aled Sage]

Hi all,

I've finally deprecated this - see 
https://github.com/apache/brooklyn-server/pull/857.


---
However, I haven't changed the use of `--jar` in `ItemLister`: that will 
still look for Java classes that implement Entity etc and/or classes 
annotated with `@Catalog`; it is run via the non-karaf CLI.


It should really look for a `catalog.bom` in the jar, and should re-use 
as much code as possible from `BrooklynCatalog` and `BrooklynTypeRegistry`.


I think that would involve a re-think / re-write. We could achieve the 
same functionality by querying the catalog of a running Brooklyn's 
catalog, rather than as a separate offline tool.


I suggest we ignore the existing `ItemLister` for now, and deprecate it 
at the same time as we deprecate the classic (non-karaf) mode of Brooklyn.


Aled


On 15/09/2017 15:33, Thomas Bouron wrote:

+1

On Fri, 15 Sep 2017 at 15:04 Richard Downer <rich...@apache.org> wrote:


+1

On 15 September 2017 at 14:12, Aled Sage <aled.s...@gmail.com> wrote:


Hi all,

I'd like to deprecate the `@Catalog` annotation [1], and the support for
`scanFromAnnotations` [2].

Previously, we annotated some entity/policy Java classes with catalog
information, such as descriptions and icon urls. However, we've moved to
using .bom files as the way to define catalog items. Therefore the
`@Catalog` is not normally used.

It is also supported/used by the `./brooklyn list-objects` CLI, to scan
jars [3]. It find the annotated entities etc, gets the catalog metadata,
and writes that out in json format. However, I don't think that's what we
want to do moving forward. We should focus on getting the metadata from

the

.bom files, which is how things should be defined for the catalog.

Any objections to deprecating these?

Aled

[1] https://github.com/apache/brooklyn-server/blob/master/api/
src/main/java/org/apache/brooklyn/api/catalog/Catalog.java
[2] https://github.com/apache/brooklyn-server/blob/master/core/
src/main/java/org/apache/brooklyn/core/catalog/internal/
CatalogClasspathDo.java#L199-L209
[3] https://github.com/apache/brooklyn-server/blob/master/server
-cli/src/main/java/org/apache/brooklyn/cli/ItemLister.java#L88-L94

Brooklyn highAvailabilityMode: default as AUTO?

[Aled Sage]

Hi all,

I'd like to change the default value of highAvailabilityMode from 
DISABLED to AUTO.


Currently, if you start two Brooklyn servers pointing at the same 
persisted state (file-system directory or object store's bucket), then 
they are independent (because HA is 'disabled' by default). However, 
they both write to that same persisted state, which will lead to 
surprising behaviour, particularly when a Brooklyn server is next restarted.


Changing to 'AUTO' would (almost entirely) have the same behaviour as we 
have currently for a single Brooklyn server. In the case of two servers 
pointing at the same persisted state, the second would come up as 
'standby', and will be automatically promoted to 'master' if the first 
stops or fails.


I say "almost entirely":
1. If you run Brooklyn and then kill it (e.g. `kill -9` or turn off the 
VM), when you start Brooklyn again it will wait to confirm the previous 
server is really dead. It waits for 30 seconds after the server's last 
heartbeat, by default.
2. The HA status shows all previous runs of the Brooklyn server (it gets 
a new node-id each time it restarts). This list will get longer and 
longer if you keep restarting Brooklyn, pointing at the same persisted 
state, until you clear out terminates instances from the list (via the 
UI or the REST api).
3. The logging at startup will be quite different (e.g. "Brooklyn 
initialisation (part two) complete" now means that the server has 
finished becoming the 'standby'. If anyone has tools/scripts that 
search/parse these logs, then they may be affected.


---

Note the current behaviour contradicts the docs [1], which say: 
"Brooklyn will automatically run in HA mode if multiple Brooklyn 
instances are started pointing at the same persistence store."


Thoughts?

Aled

p.s. another option would be to try to fail-fast when 
highAvailabilityMode is disabled but there is another Brooklyn using the 
same persisted state. However, distinguishing that from (1) above is tricky.


[1] 
https://github.com/apache/brooklyn-docs/blob/master/guide/ops/high-availability/index.md

Re: Improving persistence efficiency

[Aled Sage]

For these changes, please review:

 * https://github.com/apache/brooklyn-server/pull/964
 * https://github.com/apache/brooklyn-server/pull/966



On 24/05/2018 15:22, Aled Sage wrote:

Hi all,

I'd like to improve the efficiency of Brooklyn persistence - depending 
on your entities, a lot of files/objects can be written a lot of 
times. This results in overly high traffic to the persistence 
directory (be it an object store like S3, or the file system).


Below are three big improvements I plan to make - any 
objections/comments?

*

1. When an entity changes and is persisted, its 'adjuncts' (e.g. 
policies and enrichers) are also re-persisted.*


I've created a PR to persist only the entity when it changes: 
https://github.com/apache/brooklyn-server/pull/964.


The old behaviour was for legacy reasons. A long long time ago, some 
policies/locations had been written such that they did not notify any 
persistence-listeners when they changed. They piggybacked on the 
entity's persistence.


There is a risk that some users still have such legacy 
entities/policies/locations in use. I've therefore included a 
feature-flag (`referencedObjectsPersistence`) to re-enable the old 
behaviour.


---
*2. When an attribute is re-published with the same value, don't 
re-persist*


Currently, when `entity.sensors().set(attribute, val)` is called with 
the same value as it already has, it still calls `onAttributeChange` 
which causes the entity to be re-persisted.


We could guard this, to only trigger re-persistence when the value has 
actually changed.


Note you can currently prevent an enricher re-publishing the attribute 
by using the config `enricher.suppressDuplicates`. You can can do the 
same for a sensor feed using `suppressDuplicates`.


---
*3. When any of the entity's attributes change, it is re-persisted 
(capped at max once per second, configurable).*


We should mark some attributes as non-persisted. If their value 
changes, it will not cause the entity to be re-persisted.


This would be especially useful for attributes like cpu-usage, 
requests-per-second, etc.


In the Java code, there is already support for marking an attribute as 
non-persisted. It will require some work to wire this up properly 
(e.g. currently an attribute-change still triggers re-persistence, but 
that attribute is then excluded).


Aled

Re: Brooklyn highAvailabilityMode: default as AUTO?

[Aled Sage]

Geoff, all,

I've added some docs (about behaviour when there is a single server): 
https://github.com/apache/brooklyn-docs/pull/254


Aled


On 24/05/2018 16:37, Aled Sage wrote:

Thanks all,

Here's the PR to change the default behaviour: 
https://github.com/apache/brooklyn-server/pull/965


I'll add more info to the docs shortly.

Aled


On 22/05/2018 21:12, Geoff Macartney wrote:

+1 sounds like a change that's needed to match the intended behaviour
anyway, as described in the docs.  We should update the docs as part of
this to include your explanation above, Aled, of the details of the
behaviour.

regards
Geoff

On Tue, 22 May 2018 at 15:27 Thomas Bouron 
<thomas.bou...@cloudsoftcorp.com>

wrote:


+1, sounds sensible to me

Best.

On Tue, 22 May 2018 at 14:51 Duncan Grant <duncan.gr...@cloudsoft.io>
wrote:


Aled,

+1 sounds like a sensible plan

Duncan

On Tue, 22 May 2018 at 13:59 Aled Sage <aled.s...@gmail.com> wrote:


Hi all,

I'd like to change the default value of highAvailabilityMode from
DISABLED to AUTO.

Currently, if you start two Brooklyn servers pointing at the same
persisted state (file-system directory or object store's bucket), 
then

they are independent (because HA is 'disabled' by default). However,
they both write to that same persisted state, which will lead to
surprising behaviour, particularly when a Brooklyn server is next
restarted.

Changing to 'AUTO' would (almost entirely) have the same behaviour as

we
have currently for a single Brooklyn server. In the case of two 
servers

pointing at the same persisted state, the second would come up as
'standby', and will be automatically promoted to 'master' if the 
first

stops or fails.

I say "almost entirely":
1. If you run Brooklyn and then kill it (e.g. `kill -9` or turn 
off the
VM), when you start Brooklyn again it will wait to confirm the 
previous
server is really dead. It waits for 30 seconds after the server's 
last

heartbeat, by default.
2. The HA status shows all previous runs of the Brooklyn server (it

gets

a new node-id each time it restarts). This list will get longer and
longer if you keep restarting Brooklyn, pointing at the same 
persisted
state, until you clear out terminates instances from the list (via 
the

UI or the REST api).
3. The logging at startup will be quite different (e.g. "Brooklyn
initialisation (part two) complete" now means that the server has
finished becoming the 'standby'. If anyone has tools/scripts that
search/parse these logs, then they may be affected.

---

Note the current behaviour contradicts the docs [1], which say:
"Brooklyn will automatically run in HA mode if multiple Brooklyn
instances are started pointing at the same persistence store."

Thoughts?

Aled

p.s. another option would be to try to fail-fast when
highAvailabilityMode is disabled but there is another Brooklyn using

the

same persisted state. However, distinguishing that from (1) above is
tricky.

[1]


https://github.com/apache/brooklyn-docs/blob/master/guide/ops/high-availability/index.md 





--

Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Re: Brooklyn highAvailabilityMode: default as AUTO?

[Aled Sage]

Thanks all,

Here's the PR to change the default behaviour: 
https://github.com/apache/brooklyn-server/pull/965


I'll add more info to the docs shortly.

Aled


On 22/05/2018 21:12, Geoff Macartney wrote:

+1 sounds like a change that's needed to match the intended behaviour
anyway, as described in the docs.  We should update the docs as part of
this to include your explanation above, Aled, of the details of the
behaviour.

regards
Geoff

On Tue, 22 May 2018 at 15:27 Thomas Bouron <thomas.bou...@cloudsoftcorp.com>
wrote:


+1, sounds sensible to me

Best.

On Tue, 22 May 2018 at 14:51 Duncan Grant <duncan.gr...@cloudsoft.io>
wrote:


Aled,

+1 sounds like a sensible plan

Duncan

On Tue, 22 May 2018 at 13:59 Aled Sage <aled.s...@gmail.com> wrote:


Hi all,

I'd like to change the default value of highAvailabilityMode from
DISABLED to AUTO.

Currently, if you start two Brooklyn servers pointing at the same
persisted state (file-system directory or object store's bucket), then
they are independent (because HA is 'disabled' by default). However,
they both write to that same persisted state, which will lead to
surprising behaviour, particularly when a Brooklyn server is next
restarted.

Changing to 'AUTO' would (almost entirely) have the same behaviour as

we

have currently for a single Brooklyn server. In the case of two servers
pointing at the same persisted state, the second would come up as
'standby', and will be automatically promoted to 'master' if the first
stops or fails.

I say "almost entirely":
1. If you run Brooklyn and then kill it (e.g. `kill -9` or turn off the
VM), when you start Brooklyn again it will wait to confirm the previous
server is really dead. It waits for 30 seconds after the server's last
heartbeat, by default.
2. The HA status shows all previous runs of the Brooklyn server (it

gets

a new node-id each time it restarts). This list will get longer and
longer if you keep restarting Brooklyn, pointing at the same persisted
state, until you clear out terminates instances from the list (via the
UI or the REST api).
3. The logging at startup will be quite different (e.g. "Brooklyn
initialisation (part two) complete" now means that the server has
finished becoming the 'standby'. If anyone has tools/scripts that
search/parse these logs, then they may be affected.

---

Note the current behaviour contradicts the docs [1], which say:
"Brooklyn will automatically run in HA mode if multiple Brooklyn
instances are started pointing at the same persistence store."

Thoughts?

Aled

p.s. another option would be to try to fail-fast when
highAvailabilityMode is disabled but there is another Brooklyn using

the

same persisted state. However, distinguishing that from (1) above is
tricky.

[1]



https://github.com/apache/brooklyn-docs/blob/master/guide/ops/high-availability/index.md




--

Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Improving persistence efficiency

[Aled Sage]

Hi all,

I'd like to improve the efficiency of Brooklyn persistence - depending 
on your entities, a lot of files/objects can be written a lot of times. 
This results in overly high traffic to the persistence directory (be it 
an object store like S3, or the file system).


Below are three big improvements I plan to make - any objections/comments?
*

1. When an entity changes and is persisted, its 'adjuncts' (e.g. 
policies and enrichers) are also re-persisted.*


I've created a PR to persist only the entity when it changes: 
https://github.com/apache/brooklyn-server/pull/964.


The old behaviour was for legacy reasons. A long long time ago, some 
policies/locations had been written such that they did not notify any 
persistence-listeners when they changed. They piggybacked on the 
entity's persistence.


There is a risk that some users still have such legacy 
entities/policies/locations in use. I've therefore included a 
feature-flag (`referencedObjectsPersistence`) to re-enable the old 
behaviour.


---
*2. When an attribute is re-published with the same value, don't re-persist*

Currently, when `entity.sensors().set(attribute, val)` is called with 
the same value as it already has, it still calls `onAttributeChange` 
which causes the entity to be re-persisted.


We could guard this, to only trigger re-persistence when the value has 
actually changed.


Note you can currently prevent an enricher re-publishing the attribute 
by using the config `enricher.suppressDuplicates`. You can can do the 
same for a sensor feed using `suppressDuplicates`.


---
*3. When any of the entity's attributes change, it is re-persisted 
(capped at max once per second, configurable).*


We should mark some attributes as non-persisted. If their value changes, 
it will not cause the entity to be re-persisted.


This would be especially useful for attributes like cpu-usage, 
requests-per-second, etc.


In the Java code, there is already support for marking an attribute as 
non-persisted. It will require some work to wire this up properly (e.g. 
currently an attribute-change still triggers re-persistence, but that 
attribute is then excluded).


Aled

Re: Add support for br --endpoint?

[Aled Sage]

Hi all,

TL;DR: we can achieve what we want with the existing $BRCLI_HOME 
environment variable. Therefore suggest we defer this!


---
It turns out we already support the alternative suggestion of "custom 
configuration file" by using the environment variable $BRCLI_HOME. We 
discovered that our problem was just some new code had been added that 
didn't set this environment variable!


If you set $BRCLI_HOME, then it will use the configuration file 
$BRCLI_HOME/.brooklyn_cli. We can get jenkins to do exactly what we want 
by including the line (in all the scripts):


    export BRCLI_HOME=${WORKSPACE}

---

Much longer term, I agree that 'profile' is a nice way of thinking about 
it. However, to me it doesn't fit with idea of 'login' and 'logout' 
(e.g. aws CLI uses `aws configure --profile ...` to add a new profile).


But unless anyone is keen to work on that in the near future, I suggest 
we defer that conversation.


Aled


On 12/06/2018 10:12, Mark McKenna wrote:

​Aled,

I think i prefer the idea of a profile as its the same pattern that kubectl
sorta follow with contexts.
I assume we would have a way to delete a profile maybe `br logout --profile
staging`

For use within build / test pipelines I can see the `.brooklyn_cli` being
supplied so to give maximum flexibility it may be better to add both
`--profile` & `--config`

M

On Mon, 11 Jun 2018 at 20:20, Duncan Grant 
wrote:


Aled,

I like the endpoint option but I think it would imply that the username and
password parameters of br login should be optional in the case that they
have already been stored.

Possibly an alternative would be an init command a bit like the gcloud init
command but non-interactive.  br --config /path/to/.brooklyn_cli init
https://localhost:8443 admin pa55w0rd.  To me init better describes what
we
are doing anyway given there is no option to log out.

Regards

Duncan



On Mon, 11 Jun 2018 at 17:49 Aled Sage  wrote:


Hi all,

TL;DR: I want to add `--endpoint` to the br command, because `br login`
has global scope.

We at Cloudsoft use `br` in our qa jenkins jobs. However, the `br login`
command has global scope. This means that two jobs run on the same
jenkins slave (both running as user `jenkins`) will interfere if their
`br login ...; br catalog add ...` commands become interleaved.

I propose that we add support for a `--endpoint` argument in br
commands. For example:

  br login https://localhost:8443 admin pa55w0rd
  br --endpoint https://localhost:8443 catalog add /path/to/mybundle/

Note that `br login` stores the credentials in ~/.brooklyn_cli, along
with a "target" for the endpoint that is currently active.

---
Alternative approaches include:

1. Custom configuration file, such as:

  br --config /path/to/.brooklyn_cli login https://localhost:8443
admin pa55w0rd
  br --config /path/to/.brooklyn_cli catalog add /path/to/mybundle/

2. Introduce the notion of "profiles", such as:

  br login --profile staging https://localhost:8443 admin pa55w0rd
  br --profile staging catalog add /path/to/mybundle/

But profiles would be more complex to implement - it implies support for
multiple logins to the same endpoint with different user:credentials.

Thoughts?

Aled

Add support for br --endpoint?

[Aled Sage]

Hi all,

TL;DR: I want to add `--endpoint` to the br command, because `br login` 
has global scope.


We at Cloudsoft use `br` in our qa jenkins jobs. However, the `br login` 
command has global scope. This means that two jobs run on the same 
jenkins slave (both running as user `jenkins`) will interfere if their 
`br login ...; br catalog add ...` commands become interleaved.


I propose that we add support for a `--endpoint` argument in br 
commands. For example:


    br login https://localhost:8443 admin pa55w0rd
    br --endpoint https://localhost:8443 catalog add /path/to/mybundle/

Note that `br login` stores the credentials in ~/.brooklyn_cli, along 
with a "target" for the endpoint that is currently active.


---
Alternative approaches include:

1. Custom configuration file, such as:

    br --config /path/to/.brooklyn_cli login https://localhost:8443 
admin pa55w0rd

    br --config /path/to/.brooklyn_cli catalog add /path/to/mybundle/

2. Introduce the notion of "profiles", such as:

    br login --profile staging https://localhost:8443 admin pa55w0rd
    br --profile staging catalog add /path/to/mybundle/

But profiles would be more complex to implement - it implies support for 
multiple logins to the same endpoint with different user:credentials.


Thoughts?

Aled

Re: New Angular UI for Brooklyn [DISCUSS]

[Aled Sage]

+1


On 29/05/2018 18:23, Mark McKenna wrote:

+1 Will be a great addition



On Mon, 28 May 2018 at 19:04, John McCabe  wrote:


Awesome +1

On Mon, May 28, 2018 at 2:05 PM Andrea Turli  wrote:


+1

On 28 May 2018 at 14:51, Duncan Grant  wrote:


+1 (non-binding)
This will make it much easier for new users to start using Apache

Brooklyn.

On Mon, 28 May 2018 at 13:47 Thomas Bouron


com>
wrote:


+1 it's a great news!

On Mon, 28 May 2018 at 13:40 Geoff Macartney <

geoff.macart...@gmail.com>

wrote:


+1 exciting news

On Mon, 28 May 2018, 13:25 Duncan Johnston-Watt, <

dun...@blockchaintp.com>

wrote:


+1 Non-binding

On Mon, May 28, 2018 at 12:46 PM, Alex Heneveld <
alex.henev...@cloudsoftcorp.com> wrote:


Dear Brooklyners,

Our users at Fujitsu, UShareSoft, and Cloudsoft have generously

sponsored

the contribution of a new UI for Apache Brooklyn.  This is

based

on

the

previously-proprietary Cloudsoft AMP UI, for those of you

familiar

with

that.

The proposed newly contributed UI has all the functionality of

the

existing UI including an inspector, groovy console, and online

REST

docs.

It is much more recent (angular, webpack), modular, easy to

develop

against, and lovely to look at, and so would be a great

contribution

based

solely on that.

But even better, it provides a lot of new features:

*  A visual blueprint composer:  drag-and-drop elements from

the

catalog

onto a canvas, with a bi-directional YAML editor

* More live activity update:  a kilt view for activities,

tailing

output

from SSH commands

* A bundle-oriented catalog:  with search, bundle- or type-

view,

delete

bundles

* An extensible, skinnable, and reusable modular architecture:

embed

angular directives and components from this project in others,

build

a

branded version of the UI, and/or add your own modules (e.g. to

accompany

specific blueprints)

The last point in particular I think will be very valuable:  it

will

allow

people to use Brooklyn in many more good ways!  There are plans

to

make

the

Composer embeddable and able to work with other input libraries

(think

e.g.

of pointing it at a Docker repo or an image catalog), and with

widgets

for

configuring items, all ultimately generating Brooklyn

blueprints.

Note that this is proposed to replace the existing UI, and as

we

have

already deprecated the non-OSGi build, it is proposed to make

this

compatible only with the OSGi build.

It is also worth pointing out that the main authors on this UI

are

already

Brooklyn contributors, so there is enough experience among

active

project

members to maintain, explain, and extend this.

Assuming this proposal finds favour, we will open a repo for

review

purposes (but it will not be a merged via PR, with the actual

contribution

to come via the IP clearance process [1]), followed by

associated

PRs

in

other projects so that everything works seamlessly (which as

minor

changes

to existing code is more suited to PRs than the IP clearance

process).

Specifically we will:

* Ensure it builds and runs with the new UI in place of the old

(note

below on the Karaf switch)

* Ensure all tests are passing (esp UI tests)

* Ensure there are effective dev/test pathways and that

documentation

is

updated (in particular for testing the UI and with the UI; this

should

be

much simpler as the new UI can run separately, point at a REST

endpoint,

and can do incremental updates for UI code changes made while

running!)

* Ensure we have IP clearance, license, and are duly diligent

in

the

approval (as this is a large contribution we recognise this

will

need

special attention)

Are there any objections at this point, or any suggestions for

other

tasks

we should do to ensure its smooth integration?  Note that this

is

purely

advisory at this stage but we would very much appreciate early

sight

of

any

potential obstacles.

Once the above list is complete we will commence the IP

clearance

process

including formal vote.

Best,
Alex


[1]

https://incubator.apache.org/ip-clearance/ip-clearance-template.html




--
Duncan Johnston-Watt
CEO, Blockchain Technology Partners 

Twitter: @duncanjw 
Mob: +44 777 190 2653 <+44%207771%20902653> <+44%207771%20902653>

<+44%207771%20902653>

LinkedIn: https://linkedin.com/in/duncanjohnstonwatt


--

Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Re: Coercing instance types

[Aled Sage]

Hi Duncan,

Agree, this can be an issue for folk not specifying the explicit 
instance type - either it not working in newer regions, or it choosing a 
sub-optimal instance type.


It is tracked in:

 * https://issues.apache.org/jira/browse/BROOKLYN-582
 * https://issues.apache.org/jira/browse/JCLOUDS-1379

Unfortunately it's not as simple as updating globally. Different regions 
support different instance types. Also, the closest match for minRam and 
minCores will be different in different regions, and will sometimes be 
an older instance type (e.g. there is no `m4.medium`, to replace the 
`m3.medium` - and the `t2` instances have different behaviour with their 
CPU credits).


Also, the algorithm for minRam / minCores makes it hard to choose the 
most price-effective instance. For example, m5.large has more CPU than 
m4.large, but is also cheaper. The metadata in jclouds doesn't take this 
into account.


But saying that, there may be some quick wins to make mismatches much 
less likely. And a way to improve jclouds so that the valid instances 
per region could be supplied/retrieved.


Aled


On 30/06/2018 19:16, Duncan Johnston-Watt wrote:

Apologies - context is AWS where it would be good to avoid matching
requirements to hardwareId[1] not available in a given region.

[1]
https://brooklyn.apache.org/v/latest/locations/index.html#common-configuration-options

Best

Duncan

On Sat, Jun 30, 2018 at 7:09 PM, Duncan Johnston-Watt <
dun...@blockchaintp.com> wrote:


I appreciate that this is probably a jclouds question but the m3.
defaults don't work in London or Paris. Is it worth updating this globally?

Best
--
Duncan Johnston-Watt
CEO, Blockchain Technology Partners 

Twitter: @duncanjw 
Mob: +44 777 190 2653 <+44%207771%20902653>
LinkedIn: https://linkedin.com/in/duncanjohnstonwatt

[PROPOSAL] catalog bundle id:version mandatory in v1.0.0

[Aled Sage]

Hi all,

I propose we make it mandatory to specify the bundle id and version 
(currently it is optional).


I think we should do this asap, before 1.0.0 is released.

This is a breaking change - it will require users to add a `bundle: ...` 
line to their .bom files (if they are not supplying the metadata another 
way, such as building their project as an OSGi bundle).


TL;DR: for backwards compatibility, we let users have a different mental 
model from what Brooklyn now actually does, which can lead to confusing 
behaviour when upgrading or working with snapshots.


*## Current Behaviour*

Currently, we'll auto-wrap things as a bundle, generating a unique 
anonymous bundle name:version if one is not supplied.


This is important for users doing `br catalog add myfile.bom` or `br 
catalog add mydir/`. In both cases we automatically create an OSGi 
bundle with those contents. For that bundle's name:version, one can 
explicitly supply it (e.g. via `bundle: ...` in the .bom file). But, for 
backwards compatibility, we support .bom files that have no such metadata.


*## Old Behaviour (0.11.0 and Earlier)*

Before we added full support and use of bundles in the catalog, the 
user's .bom file was parsed and its items added to the catalog. The raw 
.bom file was discarded.


For upgrade/dependencies/versioning/usability reasons described in 
[1,2,3], this was a bad idea.



*## Reason the Current Behaviour is Bad!*

By auto-generating bundle names, we allow users to have a completely 
different mental model from what is actually happening in Brooklyn.


For simple cases (e.g. their .bom file only ever contains one item), 
that's fine.


However, it leads to surprising behaviour (and more complicated Brooklyn 
code), particularly when using snapshots or forced upgrade.


Consider a (developer) user writing a .bom file, with version 
1.0.0-SNAPSHOT containing entities A and B. If the user modifies it and 
runs `br catalog add myfile.bom` then we successfully replace the old 
auto-generated bundle with this new one. However, if the user then 
deletes B (so the .bom contains only A) and does `catalog add` again, 
it's unclear to Brooklyn whether this is meant to be the same .bom file. 
You could argue that it should be forbidden (because if we kept the old 
.bom we'd end up with two different versions of A, and deleting B would 
be wrong if it was a different .bom).


The right thing in my opinion is to force the user to have the same 
mental model as Brooklyn does: they need to include a `bundle: ` line in 
their .bom file so they are explicit about what they are doing.


Note this does not force the user to understand OSGi/java; it just 
requires them to think in terms of a "versioned bundle" of catalog items.


Aled

[1] "Uploading ZIPs for a better dev workflow" dev@brooklyn email thread
[2] "Bundles in Brooklyn" dev@brooklyn email thread
[3] "Bundles of fun" dev@brooklyn email thread

Re: [PROPOSAL] catalog bundle id:version mandatory in v1.0.0

[Aled Sage]

Alex,

I say we break it - force the user to have the correct mental model for 
1.0.0!


It's a simple one-line addition to fix a given .bom file. We should 
obviously ensure that historic persisted state continues to work.


---

Also note that our deprecation warnings are too hidden. For example, if 
you do `br catalog add ...` then deprecation warnings will only be 
visible via the Brooklyn log (rather than being returned to the user of 
`br` or via the REST api).


But that's a separate topic!

Aled


On 26/10/2017 12:58, Alex Heneveld wrote:


Absolutely.

Question is whether we should deprecate or make it a breaking change.

Deprecating feels right, though I think it would mean we can't 
actually remove until 2.0 ?


Best
Alex


On 26/10/2017 12:06, Aled Sage wrote:

Hi all,

I propose we make it mandatory to specify the bundle id and version 
(currently it is optional).


I think we should do this asap, before 1.0.0 is released.

This is a breaking change - it will require users to add a `bundle: 
...` line to their .bom files (if they are not supplying the metadata 
another way, such as building their project as an OSGi bundle).


TL;DR: for backwards compatibility, we let users have a different 
mental model from what Brooklyn now actually does, which can lead to 
confusing behaviour when upgrading or working with snapshots.


*## Current Behaviour*

Currently, we'll auto-wrap things as a bundle, generating a unique 
anonymous bundle name:version if one is not supplied.


This is important for users doing `br catalog add myfile.bom` or `br 
catalog add mydir/`. In both cases we automatically create an OSGi 
bundle with those contents. For that bundle's name:version, one can 
explicitly supply it (e.g. via `bundle: ...` in the .bom file). But, 
for backwards compatibility, we support .bom files that have no such 
metadata.


*## Old Behaviour (0.11.0 and Earlier)*

Before we added full support and use of bundles in the catalog, the 
user's .bom file was parsed and its items added to the catalog. The 
raw .bom file was discarded.


For upgrade/dependencies/versioning/usability reasons described in 
[1,2,3], this was a bad idea.



*## Reason the Current Behaviour is Bad!*

By auto-generating bundle names, we allow users to have a completely 
different mental model from what is actually happening in Brooklyn.


For simple cases (e.g. their .bom file only ever contains one item), 
that's fine.


However, it leads to surprising behaviour (and more complicated 
Brooklyn code), particularly when using snapshots or forced upgrade.


Consider a (developer) user writing a .bom file, with version 
1.0.0-SNAPSHOT containing entities A and B. If the user modifies it 
and runs `br catalog add myfile.bom` then we successfully replace the 
old auto-generated bundle with this new one. However, if the user 
then deletes B (so the .bom contains only A) and does `catalog add` 
again, it's unclear to Brooklyn whether this is meant to be the same 
.bom file. You could argue that it should be forbidden (because if we 
kept the old .bom we'd end up with two different versions of A, and 
deleting B would be wrong if it was a different .bom).


The right thing in my opinion is to force the user to have the same 
mental model as Brooklyn does: they need to include a `bundle: ` line 
in their .bom file so they are explicit about what they are doing.


Note this does not force the user to understand OSGi/java; it just 
requires them to think in terms of a "versioned bundle" of catalog 
items.


Aled

[1] "Uploading ZIPs for a better dev workflow" dev@brooklyn email thread
[2] "Bundles in Brooklyn" dev@brooklyn email thread
[3] "Bundles of fun" dev@brooklyn email thread

Re: Blueprint project structure

[Aled Sage]

Hi all,

I agree with Geoff: documenting the different approaches as you've done 
here sounds like the best way forward.


---
Note that:
    br catalog add 
https://github.com/brooklyncentral/brooklyn-dns/archive/master.zip

is not the same as:
    br catalog add target/brooklyn-dns-0.2.0-SNAPSHOT.jar

For the latter, the bundle has symbolic name 
"io.brooklyn.dns.brooklyn-dns" (which it gets from the way the pom.xml 
builds it as an OSGi bundle). The former has no bundle name, so gets an 
auto-generated anonymous bundle name.


I was assuming we would not tell people they must include `bundle: ` in 
the .bom file if they are explicitly building it as a bundle anyway. 
That is a violation of DRY (Don't Repeat Yourself). However, it would be 
necessary if we want to support adding via the zip and the jar.


Aled


On 25/10/2017 16:03, Geoff Macartney wrote:

The phrase "win - win" springs to mind.

I'm not sure we need to go quite so far as saying that 2 is always best
practice.  I'd tend toward using your description as the basis for a new
page in the docs that explains how projects can be structured and the
significance of having the `catalog.bom` in the root position, and leaves
it up to people to use what suits them.  If someone wants to use approach 1
they shouldn't feel they're deviating from best practice.





On Wed, 25 Oct 2017 at 15:58 Alex Heneveld 
wrote:


Hi All-

With the move towards bundles, people have been experimenting with
project structure, and there seem to be two schools:

(1) use maven, include a `pom.xml` in root, and put the `catalog.bom`
and blueprints into either `src/main/resources/` or a custom resources
directory eg `catalog/`

(2) put `catalog.bom` in root, and have yaml blueprint files, and
scripts and other resources, in whatever structure underneath makes sense

Both have benefits.  For instance with (2) you can deploy straight from
the root, or even `br catalog add
https://github.com/brooklyncentral/brooklyn-dns/archive/master.zip`
, and
it's obvious where the entry point is.  With (1) you can build and
install to maven, and access the dependency as `mvn:...` URLs in karaf;
and it of course becomes very useful when including java.

I've done some experimenting at [1] on a pom.xml configuration that
works with (2).   This means a blueprint developer can do what they like
without ever touching maven, not having a pom.xml, etc.  Then if they
want maven at some point it can be added on without having to change
anything else in the project.  It also means community blueprints can be
used as (1) or as (2), and people looking at them just look at the
subset of bom and yaml files that are the blueprint and it will be
sensible, without any odd structure imposed by maven.  And we can
encourage putting a `catalog.bom` in the root of every project on the
planet so there is never ever any question about how to deploy software
ever again.  :)

Is this structure something we should converge on as a recommendation
for blueprints projects?  Basically I'm proposing we say (2) is _always_
best practice, even when is needed (1), and we have (one so far and more
to follow if we like this) exemplars for how to structure projects.

Best
Alex


[1]  https://github.com/brooklyncentral/brooklyn-dns/pull/7

Re: Time to make a release?

[Aled Sage]

Thanks Richard.

Yesterday evening, I fixed a regression (in detecting recursive 
references in entity config). The PR has now been merged [1]. It would 
be a nice-to-have to include this in the milestone release, but it's not 
a blocker.


If you haven't kicked off the build, could you cherry-pick this please 
(assuming you agree).


But if you have already built it, then let's only pull in this PR if we 
need to produce a second release candidate for other reasons.


Aled

[1] https://github.com/apache/brooklyn-server/pull/987



On 07/09/2018 11:18, Richard Downer wrote:

Ok, the stars have aligned for a release - I've got a smoke test build
passing and all the requested PRs have been merged.

Here's what the first release candidate will be built from:

brooklyn-server: e601350d75eb61f0056f4900eca898426348ced9
https://github.com/apache/brooklyn-server/commits/e601350d75eb61f0056f4900eca898426348ced9

brooklyn-library: a94f7d84ebcd984ec7c44d62add2f00d5e663717
https://github.com/apache/brooklyn-library/commits/a94f7d84ebcd984ec7c44d62add2f00d5e663717


brooklyn-docs: 2b06e9605e1ee23f27f0d81d249074a60350cff6
https://github.com/apache/brooklyn-docs/commits/2b06e9605e1ee23f27f0d81d249074a60350cff6

brooklyn-client: 05031a79d4724b867db76b7c4afe55190666c7af
https://github.com/apache/brooklyn-client/commits/05031a79d4724b867db76b7c4afe55190666c7af

brooklyn-ui: 503d74eb20bc5206daaa0cfcf1f55b2e0cd0d5c9
https://github.com/apache/brooklyn-ui/commits/503d74eb20bc5206daaa0cfcf1f55b2e0cd0d5c9

brooklyn-dist: 3a30944e22052bb7b9c406e285a73f0e9236d0f7
https://github.com/apache/brooklyn-dist/commits/3a30944e22052bb7b9c406e285a73f0e9236d0f7

More updates to follow.

Richard.


On Tue, 4 Sep 2018 at 10:25, Richard Downer  wrote:


Looks like we have a consensus.

**KLAXON**

Last call for PRs before the release. Please let me know TODAY if there's
anything that needs to go in. Tomorrow morning I'll assess the requested
PRs and get a release started.

Richard.


On Tue, 4 Sep 2018 at 09:41, Thomas Bouron <
thomas.bou...@cloudsoftcorp.com> wrote:


+1

On Tue, 4 Sep 2018 at 09:34 Geoff Macartney 
wrote:


+1 to an interim release

On Mon, 3 Sep 2018 at 13:31 Alex Heneveld <
alex.henev...@cloudsoftcorp.com>
wrote:


+1

We might want to make an effort to get these merged (but I don't think
it's critical):

* https://github.com/apache/brooklyn-server/pulls
  #982 - better type coercion
  #971 - location DSL

There are a few others in library, docs, and dist, which we should take
care of as general housekeeping but not ahead of an M1 IMO.

Best
Alex


On 03/09/2018 13:24, Aled Sage wrote:

+1

Assuming there's consensus, shall we give 24 hours for folk to get

any

outstanding PRs submitted/merged, before we kick an RC1 etc?

Aled


On 03/09/2018 13:17, Richard Downer wrote:

All,

It's probably about time we made a release - shockingly, it's been
nearly a
year since the last one.

We want to start the release train towards a 1.0 release, but I

think we

could benefit from an interim release first, given that the paint is
still
drying on the new UI. Therefore I propose a "milestone 1" release,
1.0.0-M1. I'm happy to volunteer to be release manager on this one.

Thoughts, comments, +1 or -1?

Cheers
Richard




--

Thomas Bouron
Senior Software Engineer

*Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud

GitHub: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Need a hand with AWS? Get a Free Consultation.

Re: [PROPOSAL] Deprecated JBoss 7 entity

[Aled Sage]

+1 to deprecating it Thomas.

Alex, I agree with you in general. In this case, we don't particularly 
care about a yaml equivalent of jboss 7 because it's so old. We would 
like a yaml blueprint for wildfly [1] in the community at some point. I 
thought that someone had done that in the past, but I couldn't find it 
when I had a quick look. Also, Wildfly is now at version 12, so likely 
the blueprint done previously would need updated anyway.


Aled

[1] http://wildfly.org


On 09/03/2018 11:16, Alex Heneveld wrote:


go for it.  is there a yaml entity in the community we can point 
people to as part of the deprecation?


wherever possible i'd like to replace the system-specific java 
entities like jboss with yaml equivalents in the community (ie outside 
apache brooklyn).


--a


On 09/03/2018 10:28, Thomas Bouron wrote:

Hi Brooklyners

I made a PR yesterday[1] (which as been merged, thanks Geoff) to fix an
issue with JBoss 7 entity. As it turns out, JBoss 7 is now EOL and 
does not

work with java 7u171 onward. This is due to the `jboss-module.jar` not
being compatible with the newest versions of Java.

The patch I made is a trick really but it works: the version of
`jboss-module.jar` shipped with JBoss 7.1.1 is `1.1.1.GA`. However, this
particular jar has been updated to `1.1.5.GA` and using this one 
fixes the
issue. While this works, it is still a hacky thing to do therefore I 
would

like to deprecate this entity.

Any objection before I do this?

Best.

[1] https://github.com/apache/brooklyn-library/pull/148

Failing Go build: NodePrime/jsonpath missing?

[Aled Sage]

Hi all,

The Apache Brooklyn master build [1] has failed the last couple of times 
for the Go build of brooklyn-client (see end of email for the error text).


It looks to me like our dependency github.com/NodePrime/jsonpath has 
been deleted! It definitely used to be there (lots of google hits 
pointing at that [2]), but 404 when visiting the page.


The build presumably only works for me locally because I have it cached 
locally:


   [exec] [INFO]  Downloading dependencies. Please wait...
   [exec] [INFO]  --> Found desired version locally
   github.com/NodePrime/jsonpath 84403ded544328c99be3472727667179eef23a91!

Can someone else please double-check my findings?

I couldn't find any record of jsonpath having moved. Anyone have any 
opinions how best to fix this?


---
The error shown in jenkins console is:

   all:
 [exec] Starting build.sh (brooklyn-client go build script)
 [exec] Installing glide
 [exec] Installing dependencies
 [exec] [WARN]    The name listed in the config file
   (github.com/apache/brooklyn-client) does not match the current
   location (.)
 [exec] [INFO]    Downloading dependencies. Please wait...
 [exec] [INFO]    --> Fetching golang.org/x/crypto
 [exec] [INFO]    --> Fetching github.com/NodePrime/jsonpath
 [exec] [INFO]    --> Fetching github.com/urfave/cli
 [exec] [WARN]    Unable to checkout github.com/NodePrime/jsonpath
 [exec] [ERROR]    Update failed for
   github.com/NodePrime/jsonpath: Unable to get repository: Cloning
   into '/root/.glide/cache/src/https-github.com-NodePrime-jsonpath'...
 [exec] fatal: could not read Username for
   'https://github.com': No such device or address
 [exec] : exit status 128
 [exec] [ERROR]    Failed to install: Unable to get repository:
   Cloning into
   '/root/.glide/cache/src/https-github.com-NodePrime-jsonpath'...
 [exec] fatal: could not read Username for
   'https://github.com': No such device or address
 [exec] : exit status 128
 [exec] Building br for default OS/ARCH:
 [exec] darwin/amd64
 [exec] ../models/catalog.go:28:2: cannot find package
   "github.com/NodePrime/jsonpath" in any of:
 [exec]
   
/usr/build/brooklyn-client/cli/target/src/github.com/apache/brooklyn-client/cli/vendor/github.com/NodePrime/jsonpath
   (vendor tree)
 [exec] /usr/lib/go-1.7/src/github.com/NodePrime/jsonpath (from
   $GOROOT)
 [exec]
   /usr/build/brooklyn-client/cli/target/src/github.com/NodePrime/jsonpath
   (from $GOPATH)
 [exec] ../command_metadata/command_metadata.go:21:8: cannot
   find package "github.com/urfave/cli" in any of:
 [exec]
   
/usr/build/brooklyn-client/cli/target/src/github.com/apache/brooklyn-client/cli/vendor/github.com/urfave/cli
   (vendor tree)
 [exec]     /usr/lib/go-1.7/src/github.com/urfave/cli (from
   $GOROOT)
 [exec]
   /usr/build/brooklyn-client/cli/target/src/github.com/urfave/cli
   (from $GOPATH)
 [exec] ../commands/login.go:37:2: cannot find package
   "golang.org/x/crypto/ssh/terminal" in any of:
 [exec]
   
/usr/build/brooklyn-client/cli/target/src/github.com/apache/brooklyn-client/cli/vendor/golang.org/x/crypto/ssh/terminal
   (vendor tree)
 [exec] /usr/lib/go-1.7/src/golang.org/x/crypto/ssh/terminal
   (from $GOROOT)
 [exec]
   /usr/build/brooklyn-client/cli/target/src/golang.org/x/crypto/ssh/terminal
   (from $GOPATH)

Aled

[1] https://builds.apache.org/job/brooklyn-master-build-docker/
[2] https://github.com/NodePrime/jsonpath

jenkins PR failure in `git fetch`: delete workspace before build starts?

[Aled Sage]

Hi all,

Our brooklyn-ui PR builds (e.g. [1]) were failing with the same error as 
is given in INFRA-17055 [2].


```
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress git://github.com/apache/brooklyn-ui.git 
+refs/pull/*:refs/remotes/origin/pr/*" returned status code 128:

stdout:
stderr: error: Could not read 0f7365b818627ae0fa4c1cdf0fb6a65062d23968
```

I've therefore applied the same fix as described in INFRA-17055: enabled 
'Delete workspace before build starts'.


And I've kicked off my PR build again for [3].

---
Longer term, I think it makes sense to delete the workspace for PR 
builds, because PRs could vary massively. If this works for brooklyn-ui, 
is it sensible to roll that change across all the other PR build 
configurations as well?


(I think I've seen that error a few times - I'd previously put it down 
to bad networks, interrupted during git fetch.)


Aled

[1] https://builds.apache.org/job/brooklyn-ui-pull-requests/167/console
[2] https://issues.apache.org/jira/browse/INFRA-17055
[3] https://github.com/apache/brooklyn-ui/pull/76

Re: Time to make a release?

[Aled Sage]

+1

Assuming there's consensus, shall we give 24 hours for folk to get any 
outstanding PRs submitted/merged, before we kick an RC1 etc?


Aled


On 03/09/2018 13:17, Richard Downer wrote:

All,

It's probably about time we made a release - shockingly, it's been nearly a
year since the last one.

We want to start the release train towards a 1.0 release, but I think we
could benefit from an interim release first, given that the paint is still
drying on the new UI. Therefore I propose a "milestone 1" release,
1.0.0-M1. I'm happy to volunteer to be release manager on this one.

Thoughts, comments, +1 or -1?

Cheers
Richard

Re: Empire-db build problems <-- brooklyn-master-build-docker_ws-cleanup workspace left behind

[Aled Sage]

Hi all,

Do any brooklyn folk know what the 
`brooklyn-master-build-docker_ws-cleanup_.*` workspaces are for, and why 
they are not being cleaned up?


It sounds like this bug in ws-cleanup-plugin [1]. It was fixed, but then 
folk reported that they were still experiencing the same problem [2].


A workaround described in [3] is:

  ws-cleanup is leaking workspaces because the dynamic nodes are 
unavailable at the time
  the dispose is ran. I've installed a cron which is run every 1/2/h 
which is removing
  ws-cleanup folders that aren't modified the last hour on ovh docker 
hosts.


I'm not sure what we could do on the Brooklyn side.

---
Looking in the console output of our brooklyn-master-build-docker jobs, 
I see #700 [4] says at the very start:


  Building remotely on H24 (ubuntu xenial) in workspace 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker

  [WS-CLEANUP] Deleting project workspace...
  [WS-CLEANUP] Done

In contrast, builds #702 [5] doesn't have the line `[WS-CLEANUP] Done`. 
Does that mean it will have left behind one of these directories perhaps?


Looking at [6], it looks like we only keep a history of 5 builds in 
jenkins. We therefore should really not have 28 workspace directories on 
the server H24!


If it might be a problem with ws-cleanup having permissions to delete 
the workspace, could it be related to the way we mount the workspace as 
a volume in the docker container?


  docker run -i --rm -v /home/jenkins:/usr/home
  -v 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker/.m2:/root/.m2 
\
  -v 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker:/usr/build 
\
-w /usr/build brooklyn:latest mvn clean install -s 
/usr/home/.m2/settings.xml


Aled

[1] https://issues.jenkins-ci.org/browse/JENKINS-24824
[2] https://issues.jenkins-ci.org/browse/JENKINS-53579
[3] https://jira.nuxeo.com/browse/NXBT-2205
[4] 
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
[5] 
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull

[6] https://builds.apache.org/job/brooklyn-master-build-docker/


On 20/09/2018 20:20, Chris Lambertus wrote:






On Sep 20, 2018, at 11:51 AM, Dave Brondsema  wrote:

Moving to builds@

I'm seeing this type of disk error on Allura builds that have used H27 and H24
today.

Here’s the raw usage output from H24. Not sure what’s going on with those 
Brooklyn jobs. CC to dev@brooklyn so they can have a look.



root@asf924:/home/jenkins/jenkins-slave/workspace# du -sh -- *
4.3GActiveMQ-Java8
2.5GActiveMQ-Java8-All-UnitTests
1.2GActiveMQ-PR-Build
4.5GAmbari-branch-2.7
8.0GAmbari-trunk-Commit
127MAnt-Build-Matrix-1.9.x-Linux
129MAnt-Build-Matrix-master-Linux
88M Ant-Master SonarQube
4.7MApache%20Jackrabbit%20Oak%20matrix
15G Apache Jackrabbit Oak matrix
12M Aries jax-rs whiteboard
14M Aries-rsa master deploy
816Maurora-packaging-nightly
4.0K
aven-assembly-plugin_master-SP4DXUL4GVN3Q5KJHI5ED6WPWUEC2L2PUW2RCT2SFVGTCXHUIFDQ@tmp
4.0K
aven-box_maven-doxia_master-4SHBBM33BQC2SGN65AA3AW6GPXIK6RA462YQND5AN3YPZ5U2VGSQ@tmp
4.0K
aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@2@tmp
4.0K
aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@tmp
4.0K
aven-enforcer_MENFORCER-312-H6BWWMKU3O4RIVLVOSDDUVD7CT73H5PSEGRJICFTH5FH5BR7L3XQ@tmp
122MBigtop-trunk-testartifacts
924Mbookkeeper_postcommit_master_java8
261Mbookkeeper_precommit_bookie_tests
302Mbookkeeper_precommit_client_tests
754Mbookkeeper_precommit_pullrequest_java8
758Mbookkeeper_precommit_pullrequest_java9
272Mbookkeeper_precommit_pullrequest_validation
400Mbookkeeper_precommit_pullrequest_validation@2
334Mbookkeeper_precommit_remaining_tests
257Mbookkeeper_precommit_replication_tests
259Mbookkeeper_precommit_tls_tests
907Mbookkeeper_release_branch_48_integrationtests
886Mbookkeeper_release_branch_48_java8
4.0K
box_maven-ant-plugin_master-N743QKOWXTZ36LZCK2YK3A6N4ZY3UG7X5RV3MH7PEH646A2IMDHA@tmp
4.0K
-box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@2@tmp
4.0K
-box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@tmp
4.0K
box_maven-ejb-plugin_master-D5MGOBWWYZCBR23TA7FG7MUJQK233O27B5TSPA7YUOAEWLP5Z72Q@tmp
4.0K
-box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@2@tmp
4.0K
-box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@tmp
5.7M
-box_maven_pre-reset-master-AF72GWR2EBH4F7AYOLQPRDHKOEZISK7V6BEF33K5DPJKW2VE4ATA
4.0K
box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@2@tmp
4.0K
box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@tmp
4.0K

Re: PROPOSAL: conditional config constraints

[Aled Sage]

Hi Geoff, all,

You're example is correct - that is what I'd write. I don't think 
there's a short way to convey this info without using two such 
predicates, or implementing a full-blown DSL.


I wondered about including an "iff" (i.e. "if and only if"), but that 
doesn't add much. For example, e.g. "forbiddenIff("Y")` doesn't tell us 
it's required if Y is not defined. Simiarly, `requiredIffNot("Y")` would 
not tell us it's forbidden if Y is defined.


I'll push ahead with the proposal from my original email, starting with 
the server-side.


Aled


On 21/09/2018 09:08, Geoff Macartney wrote:

Hi Aled,

I'd say go for it, that looks like something that could be valuable in
various cases.
I take it your example of "exactly one of config X or config Y" would be
expressed
along the lines of a parallel set of constraints between each config -

On X:
  constraints:
- requiredUnless("Y")
- forbiddenIf("Y")

On Y:
  constraints:
- requiredUnless("X")
- forbiddenIf("X")

Geoff


On Tue, 18 Sep 2018 at 17:05 Aled Sage  wrote:


Hi all,

I'd like to add support for more sophisticated config constraints, where
there are inter-dependencies between config keys. I'd like the blueprint
composer web-console to understand (a small number of) these, and thus
to give feedback to the user about what is required and whether their
blueprint is valid.

For example, a blueprint that requires exactly one of config X or config
Y. Another example: config X2 is required if and only if config X1 is
supplied.

There are a few questions / decision points:

   1. What constraints should we support out-of-the-box?
   2. What naming convention do we use, so that the UI can parse +
understand these?
   3. Should we support multiple constraints (we do in the REST api, but
not currently in the Java API or in the Blueprint Composer UI)

---
I suggest we support things like:

  constraints:
- requiredUnless("Y")
- forbiddenIf("Y")

and:

  constraints:
- requiredIf("X1")
- forbiddenUnless("X1")

The structure of this string would be:

  {required,forbidden}{If,Unless}("")

  requiredIf("X1"):  value is required if config X1 is set;
otherwise optional.
  forbiddenUnless("X1"): value must be null if config X1 is not set;
otherwise optional.
  requiredUnless("Y"):   value is required if config Y is not set;
otherwise optional.
  forbiddenIf("Y"):  value must be null if config Y is set;
otherwise optional.

I don't think we want to get too sophisticated. For example, do *not*
support "must match regex '[a-z]+' unless config Y is present". I don't
think we should create a full-blown DSL for this!

---
Implementation notes:

We already have the basis of this in our Java code. We support a
predicate of type
`org.apache.brooklyn.core.objs.BrooklynObjectPredicate`, which has the
additional method `boolean apply(T input, BrooklynObject context)`. The
`BrooklynObject` could be an entity, or location, etc. An implementation
of this predicate can therefore lookup other config key's values, when
validating the value.

For the UI, the Blueprint Composer calls:

http://localhost:8081/v1/catalog/bundles/
//types//latest

This returns things like:

  "config": [
{
  "name": "myPredicate",
  "type": "java.lang.String",
  "reconfigurable": false,
  "label": "myPredicate",
  "pinned": false,
  "constraints": [
"MyPredicateToString()"
  ],
  "links": {}
},
...

The constraint returned here is the toString() of the predicate.

In the UI [1], there is currently some very simple logic to interpret
this string for particular types of constraint.

Aled

[1]

brooklyn-ui/ui-modules/blueprint-composer/app/components/providers/blueprint-service.provider.js

Re: Empire-db build problems <-- brooklyn-master-build-docker_ws-cleanup workspace left behind

[Aled Sage]

Hi Chris, all,

Thomas has just pointed me at 
https://issues.apache.org/jira/browse/INFRA-16417, in which you recommend:


   "The problem is the bind mount for .m2 and permissions conflicts
   between the docker
   image and jenkins. Please don't use a bind mount for filesystems you
   intend to write to.
   It will make the build run a bit longer since it has to download the
   artifacts within the
   container, but it will prevent you from having problems and
   conflicts with the
   jenkins .m2 directory."

Apologies for us not running with this advice previously - I'm looking 
into how we could do this.


Are there any good example configurations of other Apache projects 
building and running tests (ideally using maven) in docker containers, 
on apache jenkins slaves?


Aled


On 21/09/2018 10:03, Aled Sage wrote:

Hi all,

Do any brooklyn folk know what the 
`brooklyn-master-build-docker_ws-cleanup_.*` workspaces are for, and 
why they are not being cleaned up?


It sounds like this bug in ws-cleanup-plugin [1]. It was fixed, but 
then folk reported that they were still experiencing the same problem 
[2].


A workaround described in [3] is:

  ws-cleanup is leaking workspaces because the dynamic nodes are 
unavailable at the time
  the dispose is ran. I've installed a cron which is run every 1/2/h 
which is removing
  ws-cleanup folders that aren't modified the last hour on ovh docker 
hosts.


I'm not sure what we could do on the Brooklyn side.

---
Looking in the console output of our brooklyn-master-build-docker 
jobs, I see #700 [4] says at the very start:


  Building remotely on H24 (ubuntu xenial) in workspace 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker

  [WS-CLEANUP] Deleting project workspace...
  [WS-CLEANUP] Done

In contrast, builds #702 [5] doesn't have the line `[WS-CLEANUP] 
Done`. Does that mean it will have left behind one of these 
directories perhaps?


Looking at [6], it looks like we only keep a history of 5 builds in 
jenkins. We therefore should really not have 28 workspace directories 
on the server H24!


If it might be a problem with ws-cleanup having permissions to delete 
the workspace, could it be related to the way we mount the workspace 
as a volume in the docker container?


  docker run -i --rm -v /home/jenkins:/usr/home
  -v 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker/.m2:/root/.m2 
\
  -v 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker:/usr/build 
\
-w /usr/build brooklyn:latest mvn clean install -s 
/usr/home/.m2/settings.xml


Aled

[1] https://issues.jenkins-ci.org/browse/JENKINS-24824
[2] https://issues.jenkins-ci.org/browse/JENKINS-53579
[3] https://jira.nuxeo.com/browse/NXBT-2205
[4] 
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
[5] 
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull

[6] https://builds.apache.org/job/brooklyn-master-build-docker/


On 20/09/2018 20:20, Chris Lambertus wrote:





On Sep 20, 2018, at 11:51 AM, Dave Brondsema  
wrote:


Moving to builds@

I'm seeing this type of disk error on Allura builds that have used 
H27 and H24

today.
Here’s the raw usage output from H24. Not sure what’s going on with 
those Brooklyn jobs. CC to dev@brooklyn so they can have a look.




root@asf924:/home/jenkins/jenkins-slave/workspace# du -sh -- *
4.3G    ActiveMQ-Java8
2.5G    ActiveMQ-Java8-All-UnitTests
1.2G    ActiveMQ-PR-Build
4.5G    Ambari-branch-2.7
8.0G    Ambari-trunk-Commit
127M    Ant-Build-Matrix-1.9.x-Linux
129M    Ant-Build-Matrix-master-Linux
88M    Ant-Master SonarQube
4.7M    Apache%20Jackrabbit%20Oak%20matrix
15G    Apache Jackrabbit Oak matrix
12M    Aries jax-rs whiteboard
14M    Aries-rsa master deploy
816M    aurora-packaging-nightly
4.0K 
aven-assembly-plugin_master-SP4DXUL4GVN3Q5KJHI5ED6WPWUEC2L2PUW2RCT2SFVGTCXHUIFDQ@tmp
4.0K 
aven-box_maven-doxia_master-4SHBBM33BQC2SGN65AA3AW6GPXIK6RA462YQND5AN3YPZ5U2VGSQ@tmp
4.0K 
aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@2@tmp
4.0K 
aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@tmp
4.0K 
aven-enforcer_MENFORCER-312-H6BWWMKU3O4RIVLVOSDDUVD7CT73H5PSEGRJICFTH5FH5BR7L3XQ@tmp

122M    Bigtop-trunk-testartifacts
924M    bookkeeper_postcommit_master_java8
261M    bookkeeper_precommit_bookie_tests
302M    bookkeeper_precommit_client_tests
754M    bookkeeper_precommit_pullrequest_java8
758M    bookkeeper_precommit_pullrequest_java9
272M    bookkeeper_precommit_pullrequest_validation
400M    bookkeeper_precommit_pullrequest_validation@2
334M    bookkeeper_precommit_remaining_tests
257M    bookkeeper_precommit_replication_tests
259M    bookkeeper_precommit_tls_tests
907M    bookkeeper_release_branch_48_integrationtests
886M    bookkeeper_release_branch_48_java8
4.0K 
box_maven-ant-plugin_master-N743QKOWXTZ36LZCK2YK3A6N4ZY3UG7X5RV3MH7PEH646A2IMDH

Rebind historic persisted state: referencing wrap:mvn bundles

[Aled Sage]

Hi all,

TL;DR: I've hit a problem with rebinding to historic persisted state, 
when wrap:mvn has been used for OSGi bundles - the symbolic name 
changed, so classloading didn't work on rebind. Which of the solutions 
should we go for?



_*PROBLEM*_
The persisted state refers to a class in aws-java-sdk 1.11.245, but in 
my newer brooklyn I've upgraded this bundle to 1.11.411 (the old bundle 
is not there). Because this jar was added using wrap-mvn, the two 
versions of the bundle have different symbolic names! Brooklyn therefore 
doesn't know to look in the newer aws-java-sdk bundle.


The bundle was added via a feature.xml containing:

wrap:mvn:com.amazonaws/aws-java-sdk-bundle/${aws.java.sdk.version}

When built locally, this produced a bundle with the very strange 
symbolic name:


wrap_file__Users_aledsage_amp_cloudsoft-amp-karaf-5.2.0_system_com_amazonaws_aws-java-sdk-bundle_1.11.245_aws-java-sdk-bundle-1.11.245.jar

_*EXISTING FUNCTIONALITY*_
Brooklyn currently supports a couple of related features:

1. The persisted state will by default not include bundle versions. We
   are therefore willing to use newer versions of a given bundle.
2. When adding your own bundle, you can include metadata in it's
   MANIFST.mf to say what bundle/version it replaces.
   See brooklyn-docs guide/ops/upgrades/_blueprints.md

However, I don't want to use (2) because that would involve fiddling 
with the wrap:mvn to add more metadata.



_*POSSIBLE SOLUTIONS*_
There are no doubt many ways to solve this problem. I describe a few of 
them below.


I think I favour the class-renames approach because of its simplicity.

*_Add support to class-renames_*
Our deserializingClassRenames.properties allows rebind to handle class 
renames, or a specific class moving from one bundle to another. This is 
useful for historic persisted state.


We could add support for bundle wildcards, to say that all classes in 
one bundle can be loaded from another bundle.


For example:

    wrap_blah_aws-java-sdk-bundle-1.11.245.jar\:*  : 
wrap_blah_aws-java-sdk-bundle-1.11.411.jar\:*


*_Support a bundle-upgrade configuration file_*
We could add support for a config file that gives explicit named bundle 
replacements. This would augment the existing functionality (2 above), 
but instead of the replacement bundle being defined in the new bundle's 
metadata, it could also be defined in this configuration file.


For example, $BROOKLYN_HOME/etc/org.apache.brooklyn.bundleupgrade.cfg 
could contain something like:


    wrap_blah_aws-java-sdk-bundle-1.11.411.jar:
  Brooklyn-Catalog-Upgrade-For-Bundles: 
"wrap_blah_aws-java-sdk-bundle-1.11.245.jar: *"


(would need to figure out the cfg versus yaml format here, obviously!)

*_Recognise 'wrap' bundles, and allow newer versions_*
When loading the class, we could recognise the "wrap_" prefix. We could 
figure out from the symbolic name what it was built from, and be willing 
to use bundles that are newer versions.


However, playing with wrap:mvn, the bundle naming is not as obvious as 
one would expect. The symbolic name below is a very different structure 
from that above:


wrap_mvn_com.example.brooklyn.test.resources.osgi_brooklyn-test-osgi-com-example-plainoldjar_1.0.0

This example was created in karaf client by running:

    bundle:install 
wrap:mvn:com.example.brooklyn.test.resources.osgi/brooklyn-test-osgi-com-example-plainoldjar/1.0.0


See brooklyn-server's 
org.apache.brooklyn.util.core.ClassLoaderUtils.tryLoadFromBundle.


*_Require users to set the symbolic name, when using wrap:mvn_*
We could require users to *not* use the simple "wrap:mvn", and instead 
force them to ensure a more predictable symbolic name + version is used.


However, that sounds harder for users. It also doesn't solve the problem 
for anyone with such historic persisted state.



_*LONG TERM / DOCS
*_We should warn people about this in our docs, including a description 
of how to work around it.


We should discourage the use of complex types in config keys and 
sensors, where we (or the user) don't explicitly control the versioning 
and schema of those types.


**_**_

Re: Empire-db build problems <-- brooklyn-master-build-docker_ws-cleanup workspace left behind

[Aled Sage]

// removing bui...@apache.org to save them this chatter

Hi all,

I've reconfigured the builds for brooklyn-library so that it does not 
bind-mount [1] the jenkins ${HOME}/.m2 directory from the jenkins host, 
when running the build/tests in the container.


I've also switched it to use Jenkins pipeline, so it actually uses the 
Jenkinsfile [2] that Thomas added to the repo back in November! There 
are two new builds configured:


https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker-pipeline/
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-pull-requests-pipeline

Once we're happy with these, we should disable (or perhaps delete) the 
old build configurations:


https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-pull-requests/

My next steps are:

1. Disable the old build configurations.
2. Reconfigure the other brooklyn repos (and build configurations) to
   follow the pattern used in brooklyn-library.
3. Monitor the builds, to see if this is enough to fix the problems we
   were seeing.
   (however, we still bind-mount the writeable workspace directory in
   the container, so there is still room for improvement I think).

Please shout if you have any comments, or think we should take this in a 
different direction instead.


Aled

[1] https://docs.docker.com/storage/bind-mounts/#use-a-read-only-bind-mount
[2] https://github.com/apache/brooklyn-library/blob/master/Jenkinsfile


On 24/09/2018 15:20, Aled Sage wrote:

Hi Chris, all,

Thomas has just pointed me at 
https://issues.apache.org/jira/browse/INFRA-16417, in which you recommend:


"The problem is the bind mount for .m2 and permissions conflicts
between the docker
image and jenkins. Please don't use a bind mount for filesystems
you intend to write to.
It will make the build run a bit longer since it has to download
the artifacts within the
container, but it will prevent you from having problems and
conflicts with the
jenkins .m2 directory."

Apologies for us not running with this advice previously - I'm looking 
into how we could do this.


Are there any good example configurations of other Apache projects 
building and running tests (ideally using maven) in docker containers, 
on apache jenkins slaves?


Aled


On 21/09/2018 10:03, Aled Sage wrote:

Hi all,

Do any brooklyn folk know what the 
`brooklyn-master-build-docker_ws-cleanup_.*` workspaces are for, and 
why they are not being cleaned up?


It sounds like this bug in ws-cleanup-plugin [1]. It was fixed, but 
then folk reported that they were still experiencing the same problem 
[2].


A workaround described in [3] is:

  ws-cleanup is leaking workspaces because the dynamic nodes are 
unavailable at the time
  the dispose is ran. I've installed a cron which is run every 1/2/h 
which is removing
  ws-cleanup folders that aren't modified the last hour on ovh docker 
hosts.


I'm not sure what we could do on the Brooklyn side.

---
Looking in the console output of our brooklyn-master-build-docker 
jobs, I see #700 [4] says at the very start:


  Building remotely on H24 (ubuntu xenial) in workspace 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker

  [WS-CLEANUP] Deleting project workspace...
  [WS-CLEANUP] Done

In contrast, builds #702 [5] doesn't have the line `[WS-CLEANUP] 
Done`. Does that mean it will have left behind one of these 
directories perhaps?


Looking at [6], it looks like we only keep a history of 5 builds in 
jenkins. We therefore should really not have 28 workspace directories 
on the server H24!


If it might be a problem with ws-cleanup having permissions to delete 
the workspace, could it be related to the way we mount the workspace 
as a volume in the docker container?


  docker run -i --rm -v /home/jenkins:/usr/home
  -v 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker/.m2:/root/.m2 
\
  -v 
/home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker:/usr/build 
\
-w /usr/build brooklyn:latest mvn clean install -s 
/usr/home/.m2/settings.xml


Aled

[1] https://issues.jenkins-ci.org/browse/JENKINS-24824
[2] https://issues.jenkins-ci.org/browse/JENKINS-53579
[3] https://jira.nuxeo.com/browse/NXBT-2205
[4] 
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
[5] 
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull

[6] https://builds.apache.org/job/brooklyn-master-build-docker/


On 20/09/2018 20:20, Chris Lambertus wrote:





On Sep 20, 2018, at 11:51 AM, Dave Brondsema  
wrote:


Moving to builds@

I'm seeing this type of disk error on Allura builds that have used 
H27 and H24

today.
Here’s the raw usage output from H24. Not sure what’s going on with 
those Brooklyn jobs. CC to dev@brooklyn so they can have a look.




root@asf924:/ho

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Aled Sage]

+1 from me.

We need "Consensus in the project (documented via the mailing list)" - I 
interpreted that as us needing a formal vote, but if an informal email 
thread will do then that's fine with me


Aled


On 12/12/2018 16:41, Richard Downer wrote:

Brooklyn team,

Apart from myself, I don't think anyone has clearly come out in favour or
opposed to this. I'd rather we got consensus and moved to gitbox early - so
that if some people do object, we have time to work out the objections with
infra, before we are involuntarily moved.

Thoughts, anyone? Does this need a formal vote to get people responding?

Richard.


On Fri, 7 Dec 2018 at 16:54, Daniel Gruno  wrote:


[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
   DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]

Hello Apache projects,

I am writing to you because you may have git repositories on the
git-wip-us server, which is slated to be decommissioned in the coming
months. All repositories will be moved to the new gitbox service which
includes direct write access on github as well as the standard ASF
commit access via gitbox.apache.org.

## Why this move? ##
The move comes as a result of retiring the git-wip service, as the
hardware it runs on is longing for retirement. In lieu of this, we
have decided to consolidate the two services (git-wip and gitbox), to
ease the management of our repository systems and future-proof the
underlying hardware. The move is fully automated, and ideally, nothing
will change in your workflow other than added features and access to
GitHub.

## Timeframe for relocation ##
Initially, we are asking that projects voluntarily request to move
their repositories to gitbox, hence this email. The voluntary
timeframe is between now and January 9th 2019, during which projects
are free to either move over to gitbox or stay put on git-wip. After
this phase, we will be requiring the remaining projects to move within
one month, after which we will move the remaining projects over.

To have your project moved in this initial phase, you will need:

- Consensus in the project (documented via the mailing list)
- File a JIRA ticket with INFRA to voluntarily move your project repos
over to gitbox (as stated, this is highly automated and will take
between a minute and an hour, depending on the size and number of
your repositories)

To sum up the preliminary timeline;

- December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
relocation
- January 9th -> February 6th: Mandated (coordinated) relocation
- February 7th: All remaining repositories are mass migrated.

This timeline may change to accommodate various scenarios.

## Using GitHub with ASF repositories ##
When your project has moved, you are free to use either the ASF
repository system (gitbox.apache.org) OR GitHub for your development
and code pushes. To be able to use GitHub, please follow the primer
at: https://reference.apache.org/committer/github


We appreciate your understanding of this issue, and hope that your
project can coordinate voluntarily moving your repositories in a
timely manner.

All settings, such as commit mail targets, issue linking, PR
notification schemes etc will automatically be migrated to gitbox as
well.

With regards, Daniel on behalf of ASF Infra.

PS:For inquiries, please reply to us...@infra.apache.org, not your
project's dev list :-).

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Aled Sage]

Hi Brooklyn community members,

We use https://git-wip-us.apache.org/repos/asf/brooklyn, so this impacts us.

We have the choice of waiting a month or two, or volunteering to move to 
either gitbox.apache.org OR github [1] for our development and code pushes.


Being able to merge pull requests directly in github would be really 
convenient! However, I've not looked into the details of each option.


Anyone have strong opinions?

Aled

[1] https://reference.apache.org/committer/github


On 07/12/2018 16:52, Daniel Gruno wrote:

[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
 DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]

Hello Apache projects,

I am writing to you because you may have git repositories on the
git-wip-us server, which is slated to be decommissioned in the coming
months. All repositories will be moved to the new gitbox service which
includes direct write access on github as well as the standard ASF
commit access via gitbox.apache.org.

## Why this move? ##
The move comes as a result of retiring the git-wip service, as the
hardware it runs on is longing for retirement. In lieu of this, we
have decided to consolidate the two services (git-wip and gitbox), to
ease the management of our repository systems and future-proof the
underlying hardware. The move is fully automated, and ideally, nothing
will change in your workflow other than added features and access to
GitHub.

## Timeframe for relocation ##
Initially, we are asking that projects voluntarily request to move
their repositories to gitbox, hence this email. The voluntary
timeframe is between now and January 9th 2019, during which projects
are free to either move over to gitbox or stay put on git-wip. After
this phase, we will be requiring the remaining projects to move within
one month, after which we will move the remaining projects over.

To have your project moved in this initial phase, you will need:

- Consensus in the project (documented via the mailing list)
- File a JIRA ticket with INFRA to voluntarily move your project repos
  over to gitbox (as stated, this is highly automated and will take
  between a minute and an hour, depending on the size and number of
  your repositories)

To sum up the preliminary timeline;

- December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
  relocation
- January 9th -> February 6th: Mandated (coordinated) relocation
- February 7th: All remaining repositories are mass migrated.

This timeline may change to accommodate various scenarios.

## Using GitHub with ASF repositories ##
When your project has moved, you are free to use either the ASF
repository system (gitbox.apache.org) OR GitHub for your development
and code pushes. To be able to use GitHub, please follow the primer
at: https://reference.apache.org/committer/github


We appreciate your understanding of this issue, and hope that your
project can coordinate voluntarily moving your repositories in a
timely manner.

All settings, such as commit mail targets, issue linking, PR
notification schemes etc will automatically be migrated to gitbox as
well.

With regards, Daniel on behalf of ASF Infra.

PS:For inquiries, please reply to us...@infra.apache.org, not your 
project's dev list :-).

brooklyn-ui: daily build?

[Aled Sage]

Hi all,

TL;DR: we should reconfigure the `brooklyn-ui` build (and probably 
others) to be triggered daily and/or when `brooklyn-server` builds.

_*
Problem*_
Our brooklyn-ui build is configured to only run when the repo changes 
[1]. However, its parent pom is `org.apache.brooklyn:brooklyn-parent`, 
which is in `brooklyn-server` repo.


A change was made to the `snakeyaml.version` property in 
brooklyn-parent. However, it took a couple of weeks before the next 
(unrelated) change to the `brooklyn-ui` repo was merged [2], so we were 
left with a stale reference. This caused problems for downstream 
projects that build/test against snapshot.


_*Solution*_
Looking at brooklyn-library-master-docker [3], it is also configured to 
run daily. I propose we do the same for `brooklyn-ui`, and for all our 
other repos.

_*
More Solution*_
We could also trigger `brooklyn-ui` and the other repos when the 
upstream repo (i.e. `brooklyn-server`) is successfully built. Or is that 
too much load on the apache infrastructure for minor benefit?


Thoughts?

If no-one objects or has a better idea, I'll reconfigure each build to 
also run daily.


Aled

[1] 
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/configure

[2] https://github.com/apache/brooklyn-ui/commits
[3] 
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/configure

Re: [DISCUSS] Release Apache Brooklyn 1.0.0-M1 [rc1]

[Aled Sage]

+1 from me.

Here's what I tested:

1. installed/ran brooklyn (tar.gz) from local machine (os x); used it
   to spin up a blueprint for...
2. install/run brooklyn (rpm) on CentOS machine in AWS
3. deployed each of the 4 templates to aws:
1. server-template
2. bash-web-server-template
3. bash-web-and-riak-template
4. resilient-bash-web-cluster-template
4. deploy mysql (using https://github.com/brooklyncentral/brooklyn-mysql)
5. deployed a dynamic cluster of 100 'server' entities to localhost.
6. restarted Brooklyn by running `systemctl restart brooklyn` (to
   confirm that persistence/rebind works).

---
The one caveat is for (3.3) above: the riak cluster gave an error. Each 
node came up ok, but when it tried to run `joinCluster` for one of them 
it gave the error:


   2018-09-13T12:01:18,843 - DEBUG 128 o.a.b.SSH [Thread-1521]
   [co2yobqk8m@3.120.132.118:stdout] Node
   r...@ec2-18-185-97-126.eu-central-1.compute.amazonaws.com is not
   reachable!
   2018-09-13T12:01:18,859 - DEBUG 128 o.a.b.SSH [Thread-1521]
   [co2yobqk8m@3.120.132.118:stdout] Executed
   /tmp/brooklyn-20180913-120117508-g2Xj-joinCluster_RiakNodeImpl_id_co.sh,
   result 1

I tried it twice - same error each time.

I propose that we just create a jira issue to track this, rather than it 
blocking a milestone release.


Aled


On 12/09/2018 16:37, Thomas Bouron wrote:

Hi all

It's a solid +1 to me.

Here is the summary of my tests:
1. downloaded all artifacts
2. verified all signatures
3. build source with tests
4. launched Brooklyn vagrant and did sanity checks, i.e. up and running
5. launched Brooklyn (zip) and did sanity checks, i.e. up and running
6. used CLI to add locations
7. deployed the 4th quick launch template (Resilient Load-Balanced 
Bash Web Cluster (Brooklyn Example) in AWS and verify it was working


Item 1, 2, 3 and 4 were done automatically with the attached script.

On Wed, 12 Sep 2018 at 12:20 Richard Downer > wrote:


This thread is for discussions related to the release vote.

I should clarify what we are looking for in a release vote.
Particularly,
we are looking for people to download,validate, and test the release.
Only if you are satisfied that the artifacts are correct and the
quality is
high enough, should you make a "+1" vote. Alongside your vote you
should
list
the checks that you made.

Here is a good example: http://markmail.org/message/gevsz2pdciraw6jw

The vote is not simply about "the master branch contains the
features I
wanted" -
it is about making sure that *these* artifacts are *correct* (e.g.
they are
not corrupted, hashes and signatures pass) and are of
*sufficiently high
quality* to be stamped as an official release of The Apache Software
Foundation.

Why test the artifacts when master is looking good? Here are some
reasons:

- somebody could have made a commit that broke it, since you last
git pulled
- the release branch could have been made at the wrong point, or
inconsistently
  between all of the submodules
- something in the release process could have broken it
- I could have made a mistake and corrupted the files
- a problem with the Apache infrastructure could mean that the release
files are
  unobtainable or corrupted

This is why the release manager needs you to download the actual
release
artifacts and try them out.

The way Apache works can be a bit arcane sometimes, but it's all
done with
a reason. If the vote passes then the contents of the email and
its links
become "endorsed" by The Apache Software Foundation, and the
Foundation will
take on legal liability for them, forever.

And of course we want the best possible experience for our users -
so we
need
the actual release files to be tested manually to make sure that a
mistake
does
not ruin the experience for users.

So if you can spare an hour or more to download some of the
artifacts and
try
them out, then it will be *very* useful! The vote lasts for three
days so
there's no need to rush to get a vote in.

Thanks!
Richard

--

Thomas Bouron
Senior Software Engineer

*Cloudsoft  *| Bringing Business to the Cloud

GitHub: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Need a hand with AWS? Get a Free Consultation.

Re: Fwd: Build failed in Jenkins: brooklyn-master-build-docker #676

[Aled Sage]

Karaf 4.1.6 release notes [1] says it upgrades Jetty to 9.3.24 (and Pax 
Web to 6.0.11) [2], to fix the CVEs mentioned previously [3].


I suggest we try upgrading to karaf 4.1.6 (we're currently on 4.1.2 [4]).

I'll give that a go now.

Aled

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311140=12342748

[2] https://issues.apache.org/jira/browse/KARAF-5860
[3] https://securitytracker.com/id/1041194
[4] https://github.com/apache/brooklyn-server/blob/master/pom.xml#L173


On 15/09/2018 10:55, Aled Sage wrote:
The build is failing because karaf has jetty 9.3.14, but we tried to 
upgrade our dependency to 9.3.24 [1].


At least I think that is the cause, based on the clue in [2].

I’m not at my computer just now so can’t check properly what versions 
of jetty is included in karaf / pax web versions.


If we want to upgrade jetty version, we may need to upgrade karaf version.

Thoughts?

Aled

[1] https://github.com/apache/brooklyn-server/pull/991/files

[2] https://issues.apache.org/jira/plugins/servlet/mobile#issue/KARAF-5471


-- Forwarded message -
From: *Apache Jenkins Server* <mailto:jenk...@builds.apache.org>>

Date: Sat, 15 Sep 2018 at 07:55
Subject: Build failed in Jenkins: brooklyn-master-build-docker #676
To: mailto:dev@brooklyn.apache.org>>


See 
<https://builds.apache.org/job/brooklyn-master-build-docker/676/display/redirect>


--
[...truncated 12.10 MB...]
[ERROR]  mvn:org.apache.cxf/cxf-rt-transports-http-hc/3.1.10
[ERROR]  mvn:org.apache.cxf/cxf-rt-transports-http/3.1.10
[ERROR] mvn:org.apache.cxf/cxf-rt-ws-addr/3.1.10
[ERROR] mvn:org.apache.cxf/cxf-rt-ws-policy/3.1.10
[ERROR] mvn:org.apache.cxf/cxf-rt-wsdl/3.1.10
[ERROR]  mvn:org.apache.felix/org.apache.felix.configadmin/1.8.14
[ERROR]  mvn:org.apache.felix/org.apache.felix.fileinstall/3.6.0
[ERROR]  mvn:org.apache.felix/org.apache.felix.inventory/1.0.4
[ERROR]  mvn:org.apache.felix/org.apache.felix.metatype/1.1.2
[ERROR]  mvn:org.apache.felix/org.apache.felix.scr.compat/1.0.4
[ERROR]  mvn:org.apache.felix/org.apache.felix.scr/2.0.12
[ERROR]  mvn:org.apache.felix/org.apache.felix.webconsole.plugins.ds/2.0.6
[ERROR] 
 mvn:org.apache.felix/org.apache.felix.webconsole.plugins.event/1.1.6

[ERROR]  mvn:org.apache.geronimo.specs/geronimo-jaspic_1.0_spec/1.1
[ERROR]  mvn:org.apache.geronimo.specs/geronimo-jta_1.1_spec/1.1.1
[ERROR]  mvn:org.apache.geronimo.specs/geronimo-osgi-registry/1.1
[ERROR]  mvn:org.apache.httpcomponents/httpasyncclient-osgi/4.1.2
[ERROR]  mvn:org.apache.httpcomponents/httpclient-osgi/4.5.2
[ERROR]  mvn:org.apache.httpcomponents/httpcore-osgi/4.4.4
[ERROR] mvn:org.apache.jclouds.api/atmos/2.1.0
[ERROR] mvn:org.apache.jclouds.api/byon/2.1.0
[ERROR] mvn:org.apache.jclouds.api/cloudstack/2.1.0
[ERROR] mvn:org.apache.jclouds.api/docker/2.1.0
[ERROR] mvn:org.apache.jclouds.api/ec2/2.1.0
[ERROR] mvn:org.apache.jclouds.api/elasticstack/2.1.0
[ERROR] mvn:org.apache.jclouds.api/oauth/2.1.0
[ERROR]  mvn:org.apache.jclouds.api/openstack-keystone/2.1.0
[ERROR]  mvn:org.apache.jclouds.api/openstack-neutron/2.1.0
[ERROR]  mvn:org.apache.jclouds.api/openstack-nova-ec2/2.1.0
[ERROR]  mvn:org.apache.jclouds.api/openstack-nova/2.1.0
[ERROR]  mvn:org.apache.jclouds.api/openstack-swift/2.1.0
[ERROR]  mvn:org.apache.jclouds.api/rackspace-cloudfiles/2.1.0
[ERROR]  mvn:org.apache.jclouds.api/rackspace-cloudidentity/2.1.0
[ERROR] mvn:org.apache.jclouds.api/s3/2.1.0
[ERROR] mvn:org.apache.jclouds.api/sts/2.1.0
[ERROR]  mvn:org.apache.jclouds.common/googlecloud/2.1.0
[ERROR]  mvn:org.apache.jclouds.common/openstack-common/2.1.0
[ERROR]  mvn:org.apache.jclouds.driver/jclouds-bouncycastle/2.1.0
[ERROR]  mvn:org.apache.jclouds.driver/jclouds-jsch/2.1.0
[ERROR]  mvn:org.apache.jclouds.driver/jclouds-log4j/2.1.0
[ERROR]  mvn:org.apache.jclouds.driver/jclouds-okhttp/2.1.0
[ERROR]  mvn:org.apache.jclouds.driver/jclouds-slf4j/2.1.0
[ERROR]  mvn:org.apache.jclouds.driver/jclouds-sshj/2.1.0
[ERROR]  mvn:org.apache.jclouds.karaf.bundles/jsch-agentproxy-jsch/2.1.0
[ERROR] mvn:org.apache.jclouds.karaf/cache/2.1.0
[ERROR] mvn:org.apache.jclouds.karaf/core/2.1.0
[ERROR] mvn:org.apache.jclouds.karaf/recipe/2.1.0
[ERROR] mvn:org.apache.jclouds.karaf/services/2.1.0
[ERROR] mvn:org.apache.jclouds.labs/azurecompute/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/aws-ec2/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/aws-s3/2.1.0
[ERROR]  mvn:org.apache.jclouds.provider/azureblob/2.1.0
[ERROR]  mvn:org.apache.jclouds.provider/azurecompute-arm/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/b2/2.1.0
[ERROR]  mvn:org.apache.jclouds.provider/digitalocean2/2.1.0
[ERROR]  mvn:org.apache.jclouds.provider/elastichosts-lon-b/2.1.0
[ERROR]  mvn:org.apache.jclouds.provider/elastichosts-lon-p/2.1.0
[ERROR]  mvn:org.ap

Fwd: Build failed in Jenkins: brooklyn-master-build-docker #676

[Aled Sage]

The build is failing because karaf has jetty 9.3.14, but we tried to
upgrade our dependency to 9.3.24 [1].

At least I think that is the cause, based on the clue in [2].

I’m not at my computer just now so can’t check properly what versions of
jetty is included in karaf / pax web versions.

If we want to upgrade jetty version, we may need to upgrade karaf version.

Thoughts?

Aled

[1] https://github.com/apache/brooklyn-server/pull/991/files

[2] https://issues.apache.org/jira/plugins/servlet/mobile#issue/KARAF-5471


-- Forwarded message -
From: Apache Jenkins Server 
Date: Sat, 15 Sep 2018 at 07:55
Subject: Build failed in Jenkins: brooklyn-master-build-docker #676
To: 


See <
https://builds.apache.org/job/brooklyn-master-build-docker/676/display/redirect
>

--
[...truncated 12.10 MB...]
[ERROR] mvn:org.apache.cxf/cxf-rt-transports-http-hc/3.1.10
[ERROR] mvn:org.apache.cxf/cxf-rt-transports-http/3.1.10
[ERROR] mvn:org.apache.cxf/cxf-rt-ws-addr/3.1.10
[ERROR] mvn:org.apache.cxf/cxf-rt-ws-policy/3.1.10
[ERROR] mvn:org.apache.cxf/cxf-rt-wsdl/3.1.10
[ERROR] mvn:org.apache.felix/org.apache.felix.configadmin/1.8.14
[ERROR] mvn:org.apache.felix/org.apache.felix.fileinstall/3.6.0
[ERROR] mvn:org.apache.felix/org.apache.felix.inventory/1.0.4
[ERROR] mvn:org.apache.felix/org.apache.felix.metatype/1.1.2
[ERROR] mvn:org.apache.felix/org.apache.felix.scr.compat/1.0.4
[ERROR] mvn:org.apache.felix/org.apache.felix.scr/2.0.12
[ERROR]
 mvn:org.apache.felix/org.apache.felix.webconsole.plugins.ds/2.0.6
[ERROR]
 mvn:org.apache.felix/org.apache.felix.webconsole.plugins.event/1.1.6
[ERROR] mvn:org.apache.geronimo.specs/geronimo-jaspic_1.0_spec/1.1
[ERROR] mvn:org.apache.geronimo.specs/geronimo-jta_1.1_spec/1.1.1
[ERROR] mvn:org.apache.geronimo.specs/geronimo-osgi-registry/1.1
[ERROR] mvn:org.apache.httpcomponents/httpasyncclient-osgi/4.1.2
[ERROR] mvn:org.apache.httpcomponents/httpclient-osgi/4.5.2
[ERROR] mvn:org.apache.httpcomponents/httpcore-osgi/4.4.4
[ERROR] mvn:org.apache.jclouds.api/atmos/2.1.0
[ERROR] mvn:org.apache.jclouds.api/byon/2.1.0
[ERROR] mvn:org.apache.jclouds.api/cloudstack/2.1.0
[ERROR] mvn:org.apache.jclouds.api/docker/2.1.0
[ERROR] mvn:org.apache.jclouds.api/ec2/2.1.0
[ERROR] mvn:org.apache.jclouds.api/elasticstack/2.1.0
[ERROR] mvn:org.apache.jclouds.api/oauth/2.1.0
[ERROR] mvn:org.apache.jclouds.api/openstack-keystone/2.1.0
[ERROR] mvn:org.apache.jclouds.api/openstack-neutron/2.1.0
[ERROR] mvn:org.apache.jclouds.api/openstack-nova-ec2/2.1.0
[ERROR] mvn:org.apache.jclouds.api/openstack-nova/2.1.0
[ERROR] mvn:org.apache.jclouds.api/openstack-swift/2.1.0
[ERROR] mvn:org.apache.jclouds.api/rackspace-cloudfiles/2.1.0
[ERROR] mvn:org.apache.jclouds.api/rackspace-cloudidentity/2.1.0
[ERROR] mvn:org.apache.jclouds.api/s3/2.1.0
[ERROR] mvn:org.apache.jclouds.api/sts/2.1.0
[ERROR] mvn:org.apache.jclouds.common/googlecloud/2.1.0
[ERROR] mvn:org.apache.jclouds.common/openstack-common/2.1.0
[ERROR] mvn:org.apache.jclouds.driver/jclouds-bouncycastle/2.1.0
[ERROR] mvn:org.apache.jclouds.driver/jclouds-jsch/2.1.0
[ERROR] mvn:org.apache.jclouds.driver/jclouds-log4j/2.1.0
[ERROR] mvn:org.apache.jclouds.driver/jclouds-okhttp/2.1.0
[ERROR] mvn:org.apache.jclouds.driver/jclouds-slf4j/2.1.0
[ERROR] mvn:org.apache.jclouds.driver/jclouds-sshj/2.1.0
[ERROR]
 mvn:org.apache.jclouds.karaf.bundles/jsch-agentproxy-jsch/2.1.0
[ERROR] mvn:org.apache.jclouds.karaf/cache/2.1.0
[ERROR] mvn:org.apache.jclouds.karaf/core/2.1.0
[ERROR] mvn:org.apache.jclouds.karaf/recipe/2.1.0
[ERROR] mvn:org.apache.jclouds.karaf/services/2.1.0
[ERROR] mvn:org.apache.jclouds.labs/azurecompute/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/aws-ec2/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/aws-s3/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/azureblob/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/azurecompute-arm/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/b2/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/digitalocean2/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/elastichosts-lon-b/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/elastichosts-lon-p/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/elastichosts-sat-p/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/go2cloud-jhb1/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/gogrid/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/google-cloud-storage/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/google-compute-engine/2.1.0
[ERROR] mvn:org.apache.jclouds.provider/openhosting-east1/2.1.0

PROPOSAL: conditional config constraints

[Aled Sage]

Hi all,

I'd like to add support for more sophisticated config constraints, where 
there are inter-dependencies between config keys. I'd like the blueprint 
composer web-console to understand (a small number of) these, and thus 
to give feedback to the user about what is required and whether their 
blueprint is valid.


For example, a blueprint that requires exactly one of config X or config 
Y. Another example: config X2 is required if and only if config X1 is 
supplied.


There are a few questions / decision points:

 1. What constraints should we support out-of-the-box?
 2. What naming convention do we use, so that the UI can parse + 
understand these?
 3. Should we support multiple constraints (we do in the REST api, but 
not currently in the Java API or in the Blueprint Composer UI)


---
I suggest we support things like:

    constraints:
  - requiredUnless("Y")
  - forbiddenIf("Y")

and:

    constraints:
  - requiredIf("X1")
  - forbiddenUnless("X1")

The structure of this string would be:

    {required,forbidden}{If,Unless}("")

    requiredIf("X1"):  value is required if config X1 is set; 
otherwise optional.
    forbiddenUnless("X1"): value must be null if config X1 is not set; 
otherwise optional.
    requiredUnless("Y"):   value is required if config Y is not set; 
otherwise optional.
    forbiddenIf("Y"):  value must be null if config Y is set; 
otherwise optional.


I don't think we want to get too sophisticated. For example, do *not* 
support "must match regex '[a-z]+' unless config Y is present". I don't 
think we should create a full-blown DSL for this!


---
Implementation notes:

We already have the basis of this in our Java code. We support a 
predicate of type 
`org.apache.brooklyn.core.objs.BrooklynObjectPredicate`, which has the 
additional method `boolean apply(T input, BrooklynObject context)`. The 
`BrooklynObject` could be an entity, or location, etc. An implementation 
of this predicate can therefore lookup other config key's values, when 
validating the value.


For the UI, the Blueprint Composer calls:

http://localhost:8081/v1/catalog/bundles///types//latest

This returns things like:

    "config": [
  {
    "name": "myPredicate",
    "type": "java.lang.String",
    "reconfigurable": false,
    "label": "myPredicate",
    "pinned": false,
    "constraints": [
  "MyPredicateToString()"
    ],
    "links": {}
  },
  ...

The constraint returned here is the toString() of the predicate.

In the UI [1], there is currently some very simple logic to interpret 
this string for particular types of constraint.


Aled

[1] 
brooklyn-ui/ui-modules/blueprint-composer/app/components/providers/blueprint-service.provider.js

Re: Brooklyn 1.0 is shaping up <-- karaf version bump

[Aled Sage]

nt level with this latest Karaf.


Cheers
Geoff



On Mon, 2 Dec 2019 at 13:28, Aled Sage <mailto:aled.s...@gmail.com>> wrote:


Hi all,

TL;DR: I'm looking at upgading CXF from 3.2.8 to 3.3.2.

---

For dependency:
 system//javax/mail/mail: [1.4.7, 1.4.4]

Exploring this one shows that v1.4.4 is coming from the old cxf-specs
3.2.8. The Karaf 4.2.7 announcement [1] gives a table of versions
- it
shows CXF at 3.3.2. We should look up updating CXF from 3.2.8 to
3.3.2.

That will also require an upgrade of mvn:io.cloudsoft.windows/winrm4j
(as v0.7.0 is built against CXF 3.2.8).

I'm looking into this, to see what else breaks when we switch to
CXF 3.3.2.

---

I suggest we ignore:
system//org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle:

[1.2.2, 1.2]

Feature 'jetty' brings in v1.2.2, whereas feature 'pax-jetty'
brings in
v1.2, so it's out of our control.

Aled


On 02/12/2019 10:38, Aled Sage wrote:
> Hi all,
>
> I see there was great progress with the karaf version bump last
week
> and over the weekend, e.g. [1, 2].
>
> Looking at brooklyn master karaf dependency versions using `find
> ${BROOKLYN_HOME}/system/ -name "*.jar"` and running a little
program
> to find multiple versions, I see the following concerning
duplicates:
>
>
system//org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle:

> [1.2.2, 1.2]
> system//javax/mail/mail: [1.4.7, 1.4.4]
>
> The full list is:
>
> system//org/ow2/asm/asm: [5.2, 7.1]
> system//org/ow2/asm/asm-util: [5.2, 7.1]
> system//org/ow2/asm/asm-tree: [5.2, 7.1]
> system//org/ow2/asm/asm-analysis: [5.2, 7.1]
> system//org/ow2/asm/asm-commons: [5.2, 7.1]
> system//org/yaml/snakeyaml: [1.23, 1.17]
>
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.jzlib:

> [1.1.3_2, 1.0.7_1]
>
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okhttp:

> [3.4.1_1, 2.2.0_1]
>
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okio:

> [1.9.0_1, 1.2.0_1]
>
system//org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle:

> [1.2.2, 1.2]
> system//javax/mail/mail: [1.4.7, 1.4.4]
> system//javax/ws/rs/javax.ws <http://javax.ws>.rs-api: [2.0.1,
2.1.1]
>
> Compare this with Brooklyn version 1.0.0-M1:
>
> system//org/ow2/asm/asm-all: [5.2, 5.0.2]
> system//org/yaml/snakeyaml: [1.17, 1.22]
>

system//org/apache/servicemix/specs/org.apache.servicemix.specs.activation-api-1.1:

> [2.5.0, 2.6.0]
>
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.jzlib:

> [1.1.3_2, 1.0.7_1]
>
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okhttp:

> [3.4.1_1, 2.2.0_1]
>
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okio:

> [1.9.0_1, 1.2.0_1]
> system//org/apache/aries/org.apache.aries.util: [1.1.3, 1.1.0]
> system//com/google/code/gson/gson: [2.5, 2.3]
>
> I'll take a look at these and see if they really are a problem
or not.
>
> Aled
>
> [1] https://github.com/apache/brooklyn-server/pull/1068
>
> [2] https://github.com/apache/brooklyn-server/pull/1069
>
>
>
>
> On 02/12/2019 09:27, Aled Sage wrote:
>> Hi,
>>
>> Thanks Ludo, all: - great list, and +1 from me.
>>
>> I think most of those sound fairly small and manageable.
>>
>> The one that stands out as potentially hard is the "Karaf version
>> bump" (which would fix the "Two different versions of
jetty-server").
>> We've had problems in the past getting all versions consistent,
and
>> karaf to start up fast and cleanly (without it restarting a
bunch of
>> bundles). Changing the version is really easy - getting all
versions
>> of all bundles to be consistent can be a pain! But it's worth
doing.
>>
>> Aled
>>
>>
>> On 01/12/2019 14:47, Geoff Macartney wrote:
>>> We should definitely include
>>> https://issues.apache.org/jira/browse/BROOKLYN-597 "Remove MD5
and
>>> SHA-1
>>> checksums" as part of 1.0.0.  I'd be happy to do this but I
see it's
>>> got
>>> you assigned to it Richard. Do you have work in progress on
this, or
>>> would
>>> you like me to look into it? I have a notion that I might
polish up the
>>>

Re: Brooklyn 1.0 is shaping up <-- karaf version bump

[Aled Sage]

Hi all,

Good news - I have it working. I need to do some more testing, and need 
to look at releasing winrm4j 0.8.0.


I got some great pointers from the CXF mailing list (thanks Alexey 
Markevich!) [1].


The solution is for winrm4j to include:

  
    org.apache.felix
maven-bundle-plugin
    
  
  
    javax.xml.bind*;version=!,
    *
  
  
    
  

This prevents the package-import for javax.xml.bind.annotation to not 
define a specific version range.


---

The missing understanding for me was what dependency=true means [2].

For example, in cxf-specs:

    dependency="true">mvn:jakarta.xml.bind/jakarta.xml.bind-api/${cxf.jaxb.version}


It means that the bundle will only be installed if it is really needed - 
i.e. it provides packages that are not being exported by other bundles.


Because winrm4j now had:

    javax.xml.bind.annotation;version="[2.3,3)"

It meant that the javax.xml.bind.annotation from 
org.apache.felix.framework didn't match. It needed jakarta.xml.bind-api 
to meet the package import dependency.


By excluding the version range for javax.xml.bind.annotation, it was 
happy to just use org.apache.felix.framework.


Aled

[1] 
http://mail-archives.apache.org/mod_mbox/cxf-users/201912.mbox/%3C9a1d1ab6-8d00-234d-29b0-c86c31506c87%40gmail.com%3E


[2] 
http://karaf.922171.n3.nabble.com/features-xml-dependency-quot-true-quot-td3286359.html




On 04/12/2019 20:28, Aled Sage wrote:


*T**_L;DR_*

I was trying to upgrade to CXF 3.3.2 (from 3.2.8) because 
https://karaf.apache.org/news.html announcement about Karaf 4.2.7 
lists CXF version 3.3.2.


But I've hit a big problem.

winrm4j fails in Karaf with CXF 3.3.2: JAXB service construction fails 
when parsing annotations.


Should we abandon the CXF upgrade and stick with 3.2.8?

I'll report it to the CXF mailing list, so perhaps reserve judgement 
to see what they say.


_*DETAILS OF ERROR*_

The upgrade involved upgrading winrm4j to use CXF 3.3.2, which seemed 
to go fine (all tests pass outside of Karaf).


However, when I test use of WinRM inside Brooklyn Karaf, it fails with 
the error below:


```
2019-12-03T11:44:16,468 wmhGA3U2-[ut059oi7tk,ltajipcuun] WARN  103 
i.c.w.c.WinRmClient [ager-Ak6FHXbO-32] Error creating WinRm service 
with mbean strategy (trying other strategies): 
org.apache.cxf.service.factory.ServiceConstructionException

org.apache.cxf.service.factory.ServiceConstructionException: null
    at 
org.apache.cxf.jaxb.JAXBDataBinding.initialize(JAXBDataBinding.java:355) 
~[!/:3.3.2]
    at 
org.apache.cxf.service.factory.AbstractServiceFactoryBean.initializeDataBindings(AbstractServiceFactoryBean.java:86) 
~[!/:3.3.2]
    at 
org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL(ReflectionServiceFactoryBean.java:426) 
~[!/:3.3.2]
    at 
org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:528) 
~[!/:3.3.2]
    at 
org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:263) 
~[!/:3.3.2]
    at 
org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:199) 
~[?:?]
    at 
org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:103) 
~[!/:3.3.2]
    at 
org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java:91) 
~[!/:3.3.2]
    at 
org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean.java:159) 
~[!/:3.3.2]
    at 
org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBean.java:142) 
~[?:?]
    at 
org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean.java:95) 
~[!/:3.3.2]
    at 
io.cloudsoft.winrm4j.client.WinRmFactory.doCreateServiceWithBean(WinRmFactory.java:97) 
~[!/:?]
    at 
io.cloudsoft.winrm4j.client.WinRmFactory.createService(WinRmFactory.java:33) 
[!/:?]
    at 
io.cloudsoft.winrm4j.client.WinRmFactory.newInstance(WinRmFactory.java:21) 
[!/:?]
    at 
io.cloudsoft.winrm4j.client.WinRmClient.getService(WinRmClient.java:191) 
[!/:?]
    at 
io.cloudsoft.winrm4j.client.WinRmClient.(WinRmClient.java:172) 
[!/:?]
    at 
io.cloudsoft.winrm4j.client.WinRmClientBuilder.build(WinRmClientBuilder.java:194) 
[!/:?]
    at 
io.cloudsoft.winrm4j.winrm.WinRmTool.executeCommand(WinRmTool.java:298) 
[!/:?]
    at 
io.cloudsoft.winrm4j.winrm.WinRmTool.executeCommand(WinRmTool.java:216) 
[!/:?]
    at 
org.apache.brooklyn.util.core.internal.winrm.winrm4j.Winrm4jTool$1.apply(Winrm4jTool.java:119) 
[!/:1.0.0-SNAPSHOT]
    at 
org.apache.brooklyn.util.core.internal.winrm.winrm4j.Winrm4jTool$1.apply(Winrm4jTool.java:117) 
[!/:1.0.0-SNAPSHOT]
    at 
org.apache.brooklyn.util.core.internal.winrm.winrm4j.Winrm4jTool.exec(Winrm4jTo

Re: Brooklyn 1.0 is shaping up <-- karaf version bump

[Aled Sage]

Hi all,

I see there was great progress with the karaf version bump last week and 
over the weekend, e.g. [1, 2].


Looking at brooklyn master karaf dependency versions using `find 
${BROOKLYN_HOME}/system/ -name "*.jar"` and running a little program to 
find multiple versions, I see the following concerning duplicates:


system//org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle: 
[1.2.2, 1.2]

system//javax/mail/mail: [1.4.7, 1.4.4]

The full list is:

system//org/ow2/asm/asm: [5.2, 7.1]
system//org/ow2/asm/asm-util: [5.2, 7.1]
system//org/ow2/asm/asm-tree: [5.2, 7.1]
system//org/ow2/asm/asm-analysis: [5.2, 7.1]
system//org/ow2/asm/asm-commons: [5.2, 7.1]
system//org/yaml/snakeyaml: [1.23, 1.17]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.jzlib: 
[1.1.3_2, 1.0.7_1]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okhttp: 
[3.4.1_1, 2.2.0_1]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okio: 
[1.9.0_1, 1.2.0_1]
system//org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle: 
[1.2.2, 1.2]

system//javax/mail/mail: [1.4.7, 1.4.4]
system//javax/ws/rs/javax.ws.rs-api: [2.0.1, 2.1.1]

Compare this with Brooklyn version 1.0.0-M1:

system//org/ow2/asm/asm-all: [5.2, 5.0.2]
system//org/yaml/snakeyaml: [1.17, 1.22]
system//org/apache/servicemix/specs/org.apache.servicemix.specs.activation-api-1.1: 
[2.5.0, 2.6.0]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.jzlib: 
[1.1.3_2, 1.0.7_1]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okhttp: 
[3.4.1_1, 2.2.0_1]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okio: 
[1.9.0_1, 1.2.0_1]

system//org/apache/aries/org.apache.aries.util: [1.1.3, 1.1.0]
system//com/google/code/gson/gson: [2.5, 2.3]

I'll take a look at these and see if they really are a problem or not.

Aled

[1] https://github.com/apache/brooklyn-server/pull/1068

[2] https://github.com/apache/brooklyn-server/pull/1069




On 02/12/2019 09:27, Aled Sage wrote:

Hi,

Thanks Ludo, all: - great list, and +1 from me.

I think most of those sound fairly small and manageable.

The one that stands out as potentially hard is the "Karaf version 
bump" (which would fix the "Two different versions of jetty-server"). 
We've had problems in the past getting all versions consistent, and 
karaf to start up fast and cleanly (without it restarting a bunch of 
bundles). Changing the version is really easy - getting all versions 
of all bundles to be consistent can be a pain! But it's worth doing.


Aled


On 01/12/2019 14:47, Geoff Macartney wrote:

We should definitely include
https://issues.apache.org/jira/browse/BROOKLYN-597 "Remove MD5 and SHA-1
checksums" as part of 1.0.0.  I'd be happy to do this but I see it's got
you assigned to it Richard. Do you have work in progress on this, or 
would

you like me to look into it? I have a notion that I might polish up the
signing procedure a bit.

Cheers
Geoff

On Sun, 1 Dec 2019 at 14:13, Geoff Macartney 
wrote:


Hi Ludo,

I am indeed pleased to see this!  I had been thinking of mailing the
community again on the matter after Christmas was out of the way, 
let's let

2020 be the year of Brooklyn 1.0.0!

That's a great list of things to address. One thing to add would be
removing some deprecated code. Do you think they are all small 
changes, or

do any require extensive work or have a wide impact?

The reason I ask is that I think it would be good to restrict 
ourselves to
relatively easy and limited impact changes between now and 1.0.0. My 
fear
is that larger changes will take a longer time and push back a 
release even
further. It seems to me things are quite stable now; I'd say let's 
polish

it up and release it rather than try to make significant improvements.

What does everyone think?

Cheers
Geoff




On Thu, 28 Nov 2019 at 11:04, Ludovic Farine  wrote:


Hi everyone,

Hope you are all doing well!

Christmas is fast approaching and well hopefully a gift we would 
all like

to have: Brooklyn 1.0.0 release :) Geoff would be pleased to hear!

Working towards this goal of first official release, these are a 
few bug

fixes and improvements that I believe are worth considering
(non-exhaustive
and not ordered wishlist!) :

Bug fixes:

    -

    Broken icon links for ELK and DNS entities
    -

    REST API Swagger error
    -

    Two different versions of jetty-server
    -

    Karaf version bump


Improvements:

    -

    Session Management: introduce expiry settings for inactivity or
repeated
    background API calls
    -

    User Experience: blueprints ordered by date deployed, cookies to
    remember preferred sort criteria and palette preferences


Clean-up:

    -

    Review existing open Pull Requests
    -

    Fix most of the Jenkins tests


What are your thoughts on this scope? Any other suggestions you 
would like

to deliver in this next release?

Looking forward t

Re: Brooklyn 1.0 is shaping up

[Aled Sage]

Hi,

Thanks Ludo, all: - great list, and +1 from me.

I think most of those sound fairly small and manageable.

The one that stands out as potentially hard is the "Karaf version bump" 
(which would fix the "Two different versions of jetty-server"). We've 
had problems in the past getting all versions consistent, and karaf to 
start up fast and cleanly (without it restarting a bunch of bundles). 
Changing the version is really easy - getting all versions of all 
bundles to be consistent can be a pain! But it's worth doing.


Aled


On 01/12/2019 14:47, Geoff Macartney wrote:

We should definitely include
https://issues.apache.org/jira/browse/BROOKLYN-597 "Remove MD5 and SHA-1
checksums" as part of 1.0.0.  I'd be happy to do this but I see it's got
you assigned to it Richard. Do you have work in progress on this, or would
you like me to look into it? I have a notion that I might polish up the
signing procedure a bit.

Cheers
Geoff

On Sun, 1 Dec 2019 at 14:13, Geoff Macartney 
wrote:


Hi Ludo,

I am indeed pleased to see this!  I had been thinking of mailing the
community again on the matter after Christmas was out of the way, let's let
2020 be the year of Brooklyn 1.0.0!

That's a great list of things to address. One thing to add would be
removing some deprecated code. Do you think they are all small changes, or
do any require extensive work or have a wide impact?

The reason I ask is that I think it would be good to restrict ourselves to
relatively easy and limited impact changes between now and 1.0.0. My fear
is that larger changes will take a longer time and push back a release even
further. It seems to me things are quite stable now; I'd say let's polish
it up and release it rather than try to make significant improvements.

What does everyone think?

Cheers
Geoff




On Thu, 28 Nov 2019 at 11:04, Ludovic Farine  wrote:


Hi everyone,

Hope you are all doing well!

Christmas is fast approaching and well hopefully a gift we would all like
to have: Brooklyn 1.0.0 release :) Geoff would be pleased to hear!

Working towards this goal of first official release, these are a few bug
fixes and improvements that I believe are worth considering
(non-exhaustive
and not ordered wishlist!) :

Bug fixes:

-

Broken icon links for ELK and DNS entities
-

REST API Swagger error
-

Two different versions of jetty-server
-

Karaf version bump


Improvements:

-

Session Management: introduce expiry settings for inactivity or
repeated
background API calls
-

User Experience: blueprints ordered by date deployed, cookies to
remember preferred sort criteria and palette preferences


Clean-up:

-

Review existing open Pull Requests
-

Fix most of the Jenkins tests


What are your thoughts on this scope? Any other suggestions you would like
to deliver in this next release?

Looking forward to get the community engaging in the next weeks and
months.

Remember you can participate with others on the official Slack channel
*#brooklyn* on the official Apache group. Sign up here
https://s.apache.org/slack-invite to join the discussion!

Thanks

Ludo


--

Ludovic Farine

Senior Technical Project Manager


Cloudsoft  | Bringing Business to the Cloud

E: l...@cloudsoft.io

T: +44 7584 748013

L: https://www.linkedin.com/in/ludovicfarine/

Re: Brooklyn 1.0 is shaping up <-- karaf version bump

[Aled Sage]

Hi all,

TL;DR: I'm looking at upgading CXF from 3.2.8 to 3.3.2.

---

For dependency:
    system//javax/mail/mail: [1.4.7, 1.4.4]

Exploring this one shows that v1.4.4 is coming from the old cxf-specs 
3.2.8. The Karaf 4.2.7 announcement [1] gives a table of versions - it 
shows CXF at 3.3.2. We should look up updating CXF from 3.2.8 to 3.3.2.


That will also require an upgrade of mvn:io.cloudsoft.windows/winrm4j 
(as v0.7.0 is built against CXF 3.2.8).


I'm looking into this, to see what else breaks when we switch to CXF 3.3.2.

---

I suggest we ignore:
system//org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle: 
[1.2.2, 1.2]


Feature 'jetty' brings in v1.2.2, whereas feature 'pax-jetty' brings in 
v1.2, so it's out of our control.


Aled


On 02/12/2019 10:38, Aled Sage wrote:

Hi all,

I see there was great progress with the karaf version bump last week 
and over the weekend, e.g. [1, 2].


Looking at brooklyn master karaf dependency versions using `find 
${BROOKLYN_HOME}/system/ -name "*.jar"` and running a little program 
to find multiple versions, I see the following concerning duplicates:


system//org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle: 
[1.2.2, 1.2]

system//javax/mail/mail: [1.4.7, 1.4.4]

The full list is:

system//org/ow2/asm/asm: [5.2, 7.1]
system//org/ow2/asm/asm-util: [5.2, 7.1]
system//org/ow2/asm/asm-tree: [5.2, 7.1]
system//org/ow2/asm/asm-analysis: [5.2, 7.1]
system//org/ow2/asm/asm-commons: [5.2, 7.1]
system//org/yaml/snakeyaml: [1.23, 1.17]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.jzlib: 
[1.1.3_2, 1.0.7_1]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okhttp: 
[3.4.1_1, 2.2.0_1]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okio: 
[1.9.0_1, 1.2.0_1]
system//org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle: 
[1.2.2, 1.2]

system//javax/mail/mail: [1.4.7, 1.4.4]
system//javax/ws/rs/javax.ws.rs-api: [2.0.1, 2.1.1]

Compare this with Brooklyn version 1.0.0-M1:

system//org/ow2/asm/asm-all: [5.2, 5.0.2]
system//org/yaml/snakeyaml: [1.17, 1.22]
system//org/apache/servicemix/specs/org.apache.servicemix.specs.activation-api-1.1: 
[2.5.0, 2.6.0]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.jzlib: 
[1.1.3_2, 1.0.7_1]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okhttp: 
[3.4.1_1, 2.2.0_1]
system//org/apache/servicemix/bundles/org.apache.servicemix.bundles.okio: 
[1.9.0_1, 1.2.0_1]

system//org/apache/aries/org.apache.aries.util: [1.1.3, 1.1.0]
system//com/google/code/gson/gson: [2.5, 2.3]

I'll take a look at these and see if they really are a problem or not.

Aled

[1] https://github.com/apache/brooklyn-server/pull/1068

[2] https://github.com/apache/brooklyn-server/pull/1069




On 02/12/2019 09:27, Aled Sage wrote:

Hi,

Thanks Ludo, all: - great list, and +1 from me.

I think most of those sound fairly small and manageable.

The one that stands out as potentially hard is the "Karaf version 
bump" (which would fix the "Two different versions of jetty-server"). 
We've had problems in the past getting all versions consistent, and 
karaf to start up fast and cleanly (without it restarting a bunch of 
bundles). Changing the version is really easy - getting all versions 
of all bundles to be consistent can be a pain! But it's worth doing.


Aled


On 01/12/2019 14:47, Geoff Macartney wrote:

We should definitely include
https://issues.apache.org/jira/browse/BROOKLYN-597 "Remove MD5 and 
SHA-1
checksums" as part of 1.0.0.  I'd be happy to do this but I see it's 
got
you assigned to it Richard. Do you have work in progress on this, or 
would

you like me to look into it? I have a notion that I might polish up the
signing procedure a bit.

Cheers
Geoff

On Sun, 1 Dec 2019 at 14:13, Geoff Macartney 


wrote:


Hi Ludo,

I am indeed pleased to see this!  I had been thinking of mailing the
community again on the matter after Christmas was out of the way, 
let's let

2020 be the year of Brooklyn 1.0.0!

That's a great list of things to address. One thing to add would be
removing some deprecated code. Do you think they are all small 
changes, or

do any require extensive work or have a wide impact?

The reason I ask is that I think it would be good to restrict 
ourselves to
relatively easy and limited impact changes between now and 1.0.0. 
My fear
is that larger changes will take a longer time and push back a 
release even
further. It seems to me things are quite stable now; I'd say let's 
polish

it up and release it rather than try to make significant improvements.

What does everyone think?

Cheers
Geoff




On Thu, 28 Nov 2019 at 11:04, Ludovic Farine  
wrote:



Hi everyone,

Hope you are all doing well!

Christmas is fast approaching and well hopefully a gift we would 
all like

to have: Brooklyn 1.0.0 release :) Geoff would be pleased to hear!

Working towards this goal of first offic

Re: PAX Exam Issue with Brooklyn Build

[Aled Sage]

Thanks Martin,

Which environment(s) is it passing and failing in?

If this works in our apache jenkins and for contributors locally, and 
especially if the environment it fails in is "niche" then a +1 from me 
for skipping the test so we can get the release candidate built.


Aled

p.s. the test passes for me locally on my mac - I tested it with:

    mvn test 
-Dtest=org.apache.brooklyn.core.dsl.external.ExternalConfigBrooklynPropertiesOsgiTest



On 24/01/2020 16:35, Martin Harris wrote:

Hi Folks,

In building the first release candidate for Brooklyn 1.0.0, we've hit 
an issue when trying to run the build. The 
`org.apache.brooklyn.AssemblyTest` and 
`org.apache.brooklyn.core.dsl.external.ExternalConfigBrooklynPropertiesOsgiTest` 
tests are failing in certain environments, but are not reproducible in 
other environments


So far, we have been unable to determine the specific cause, and have 
not been able to find a resolution


The exact failures and relevant section of the log are attached

As the issue seem environment specific, and no other issues have been 
identified in the built artifacts, we are considering simply disabling 
the tests


If you think we should not disable the tests, or if you can provide 
any feedback on the cause of the issue, please let us know. Otherwise, 
we will disable the tests and continue with the build early next week. 
A lazy consensus applies


Thanks

--
Martin Harris
Lead Software Engineer

*Cloudsoft  *| Bringing Business to the Cloud

E: mar...@cloudsoft.io 
M: 07989 047855

Need a hand with AWS? Get a Free Consultation. 

Re: [VOTE] Release Apache Brooklyn 1.0.0 [rc3]

[Aled Sage]

Hi Geoff, all,

For using vagrant, Iuliana said:

    "Successfully started unpacked and started 
apache-brooklyn-1.0.0-rc3-vagrant.zip. All nodes were correctly created."


Geoff, did your steps match the vagrant instructions at 
http://brooklyn.apache.org/v/latest/start/running.html?


*Iuliana*, can you add anything about how you ran it?

--

The error is interesting, it is trying to use:

https://www.apache.org/dyn/closer.lua?action=download=brooklyn/apache-brooklyn-1.0.0/apache-brooklyn-1.0.0.noarch.rpm

That wouldn't work because we haven't released 1.0.0 yet.

Comparing that against the 0.12.0 release path, it's similar:

https://www.apache.org/dyn/closer.lua?action=download=brooklyn/apache-brooklyn-0.12.0/apache-brooklyn-0.12.0-1.noarch.rpm

Note the need for the "-1" in the file name though.

Aled


On 20/02/2020 23:13, Geoff Macartney wrote:

My current appraisal of the RC is:

verify_brooklyn_rc.sh looks ok, tail of output (tidied):

[✓] Build from sources successful
~/brooklyn_releases/apache-brooklyn-1.0.0-rc3 ~/brooklyn_releases

---

Additional steps requiring manual intervention (execute in source
distribution folder apache-brooklyn-1.0.0-src:
...etc

Checks successfully completed:
[✓] Download links work.
[✓] Checksums and PGP signatures are valid.
[✓] Expanded source archive matches contents of RC tag.
[✓] Expanded source archive builds and passes tests.
[✓] LICENSE is present and correct.
[✓] NOTICE is present and correct, including copyright date.
[✓] No compiled archives bundled in source archive.

Checks left to do manually with the help of above instructions:
[✓] All files have license headers where appropriate.
[✓] All dependencies have compatible licenses.

Remaning items from checklist:
[✓] Binaries work.
[✓] I follow this project’s commits list.


So far so good. Also did:

[✓] Was able to use UI to create and install simple apps (on AWS - a
server, and Tomcat 8 with demo war).
[✓] br cli works (OSX binary, haven't tried the others)

On the other hand, I get a failure running the Vagrant release. I am not
sure I'm doing this right though; it's a long time since I've used this. I
just unpacked that tarball, cd-ed to that directory, and did "vagrant up".
It churns away for a bit, then I get a 404 at the "download release
archive" step. I tweaked the script to output the URL it's downloading and
it says

 brooklyn: Downloading Brooklyn release archive
https://www.apache.org/dyn/closer.lua?action=download=brooklyn/apache-brooklyn-1.0.0/apache-brooklyn-1.0.0.noarch.rpm
 brooklyn: curl: (22) The requested URL returned error: 404 Not Found
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.


Does anyone recognise this, or know what's up?

regards
Geoff








On Thu, 20 Feb 2020 at 18:33, Geoff Macartney 
wrote:


Note that's different from the situation with the Docker image above. If
we are distributing source with a Dockerfile, I'd really expect that to
"Just Work". I'm tending towards thinking this will mean a -1 from me -
open to debate the matter though, what do you all think?

Geoff


On Thu, 20 Feb 2020 at 18:30, Geoff Macartney 
wrote:


Hi Paul,


Given Debian Buster, the current version doesn't provide a Java 8 JRE,

the

Debian package of Apache Brooklyn won't work on that version.
Getting this to work would require the user to install a system-wide

JRE 8

on their machine.

That's perfectly OK, we require a Java 8 runtime but we don't make any
stipulations about how you get that. It's quite a normal option to consider
getting a machine, installing Java, and then running Brooklyn on it.

Cheers
Geoff





On Thu, 20 Feb 2020 at 17:02, Duncan Grant 
wrote:


+1 (binding)

I don't think Paul's reported issue above is a blocker - but I do think
we
should address this soon.

I've run some tests on mac using tar.gz and deployed some apps to aws and
gce.  All worked as expected so I'm happy for the release to go ahead.

Duncan

On Thu, 20 Feb 2020 at 16:03, Paul Campbell 
wrote:


Hi,

I think I've pinned down my errors to my machine having openjdk-11-jre
installed.

Given Debian Buster, the current version doesn't provide a Java 8 JRE,

the

Debian package of Apache Brooklyn won't work on that version.

Getting this to work would require the user to install a system-wide

JRE 8

on their machine. (i.e. download, unpack and update system environment
variables)

Paul

On Thu, 20 Feb 2020 at 14:41, Paul Campbell <

paul.campb...@cloudsoft.io>

wrote:


Hi,

Trying to test the RC3 deb package on Debian 10 (Buster).

I'm not able to start the 'brooklyn' service properly when installed

from

the Debian package.

sudo dpkg -i apache-brooklyn-1.0.0-rc3.deb
sudo service brooklyn start

Apache Brooklyn never completes startup, but doesn't die either. Port

8081

never 

Re: [VOTE] Release Apache Brooklyn 1.0.0 [rc3]

[Aled Sage]

+1 from me.

I did some exploratory testing on OS X with the tar.gz, and deployed a 
few blueprints.


--

For the dockerfile not working out-of-the-box with base image 
`maven:3.6.3-jdk-8`, that's something we should include in the release 
notes (saying what the simple workaround is). However, I don't think 
this has to be a blocker - we don't use this to release an image to 
docker hub (or equivalent) as part of the brooklyn release. Also, that 
dockerfile is a convenience for building (if I understand correctly) 
rather than impacting end-users who are running Brooklyn; the build 
outside of docker still works.


Aled


On 18/02/2020 23:37, Richard Downer wrote:

This is to call for a vote for the release of Apache Brooklyn 1.0.0.

This release comprises of a source code distribution, and a corresponding
binary distribution, and Maven artifacts.

The source and binary distributions, including signatures, digests, etc.
can be found at:

   https://dist.apache.org/repos/dist/dev/brooklyn/apache-brooklyn-1.0.0-rc3

The artifact SHA-256 checksums are as follows:

   f6b199faadee7391a1a0509a853619aef37b0d3c1a6ecf01ecff2b07d729c42a
*apache-brooklyn-1.0.0-rc3-1.noarch.rpm
   7481b2c24949de9c29bfd1c5c9a32e1199c9fa0504af01d9a9bca764721def61
*apache-brooklyn-1.0.0-rc3-bin.tar.gz
   9e2d594021056fa3ecc76c87648609be6e0884222f4f921b8bad1f529896cd9c
*apache-brooklyn-1.0.0-rc3-bin.zip
   e0148e4bd5aa6be2979edda29609432f060cee60b36a79608de150f3263aa480
*apache-brooklyn-1.0.0-rc3-classic.tar.gz
   e0b26edd4d2e340513f06d8e089d2be12e1b882cb1dc4ffd27061fe53c72bc84
*apache-brooklyn-1.0.0-rc3-classic.zip
   3742424ce5a58813591c21ab556072915c20eb390eb0484e2650b8f8ed605494
*apache-brooklyn-1.0.0-rc3-client-cli-linux.tar.gz
   eafded3bfc5f8a63ddd543c98e142f928eae9f423ba8da68f7e062be2796a06d
*apache-brooklyn-1.0.0-rc3-client-cli-linux.zip
   13f025116cd9d3ffa40d328c522e50c9d18a6d3a1fc77238e3310a2098a92c82
*apache-brooklyn-1.0.0-rc3-client-cli-macosx.tar.gz
   f554fa5d95ce11f5f08d7b9f66e7be4a0219acdc3064067a44e4b1524d437d24
*apache-brooklyn-1.0.0-rc3-client-cli-macosx.zip
   8c41c4902004ad283f6e08a71256be82d5e747cc553d54e16f22b40214210cf5
*apache-brooklyn-1.0.0-rc3-client-cli-windows.tar.gz
   ffb1a7dd912d74968b2282b9a224dbc019f6fc81bec1c81d03fd17bf34b58277
*apache-brooklyn-1.0.0-rc3-client-cli-windows.zip
   9d6a7a04fe10f95a34b6167f2896a219f7863e16e66e107e4243ce32695d5b49
*apache-brooklyn-1.0.0-rc3-src.tar.gz
   a9e622854b2774790e0f7421b21b680a04983c18e09a086628c4360248670fa3
*apache-brooklyn-1.0.0-rc3-src.zip
   164898a211dea73397fb22faa7b8198ff4d63d0941e1c690a20c39515cfadbcc
*apache-brooklyn-1.0.0-rc3-vagrant.tar.gz
   ac81a3c07b4b59213362abc6ecbac3a57de500cee63a9793f06a42e8821ffce1
*apache-brooklyn-1.0.0-rc3-vagrant.zip
   9e45418d12edb0332dec8a3e93b991dcb0f9afa55d9f2fef4d5cd6c6c65b9a26
*apache-brooklyn-1.0.0-rc3.deb

The Nexus staging repository for the Maven artifacts is located at:


https://repository.apache.org/content/repositories/orgapachebrooklyn-1055

All release artifacts are signed with the following key:

 https://people.apache.org/keys/committer/richard.asc

KEYS file available here:

 https://dist.apache.org/repos/dist/release/brooklyn/KEYS


The artifacts were built from git commit IDs:

brooklyn: d3ef75f26240bee38d288d507364616380e4d853
brooklyn-client: 52b5546a5434eb6ecae55233d134f6dc11ce617b
brooklyn-dist: d55424d0c5994d9bceb804bcf0ecc7c80560370e
brooklyn-docs: d142ba2e2a65801e5bebea9e48bdc476d0265b7a
brooklyn-library: cb57d7ff725e6b59ac8ec2a533696664c298e917
brooklyn-server: 029fa6d1723550b500483aac5144749c28b6d6b7
brooklyn-ui: c209d0b9e9612f39ea462c912bf58b121932e3d0
All of the above have been tagged as "apache-brooklyn-1.0.0-rc3"

Please vote on releasing this package as Apache Brooklyn 1.0.0.

The vote will be open for at least 72 hours.
[ ] +1 Release this package as Apache Brooklyn 1.0.0
[ ] +0 no opinion
[ ] -1 Do not release this package because ...


Thanks!

Brooklyn 1.0.0 release candidate?

[Aled Sage]

Hi all,

I believe we are (finally!) ready to produce Apache Brooklyn 1.0.0 RC1.

Assuming folk agree, then we can kick off the RC1 release process and 
build it over the weekend or Monday.


Please give an information +1 or -1 (we'll do a proper vote on the 
actual RCs).


---

We proposed a "code freeze" for 13th Dec [1] and by lazy consensus we've 
focused on bug fixes / test fixes since then.


Aled

[1] See Martin Harris' email: "Open pull requests".

Re: [PROPOSAL] Revamp of website/docs

[Aled Sage]

Hi Thomas,

Sounds really interesting.

I'd view the docs issues slightly differently: what are our biggest pain 
points and what should be the first incremental steps to solve those?


I think the biggest pain points (and thus biggest bang for buck) are:

_*1. How to build and release the existing docs / website is not well 
documented*_


There is not a good web-page describing how one would build the docs 
locally (e.g. to test it), or a good web-page saying how to release this 
to make it go live. Writing a page that describes the current process 
would greatly improve things. [if there is already, apologies and please 
point me at it!]


Thomas, I believe you previously created a docker container for building 
the docs (with the right jekyll version etc) - is that right? Can you 
point us at that please?


_*2. Simplify releasing the docs (manually)*_

Switching to .asf.yaml sounds good. Could we do this now, before we 
switch to vuepress?


It feels like this would not be too much work to document / set up, and 
a lot of this would be reusable when/if we switch to vuepress.


I think setting this up so it can be done manually would be a 
pre-requisite before trying to automate it in CI/jenkins, so is worth 
doing first.


---

_*Affect on Downstream Projects*_

I know of at least one downstream project that embeds the brooklyn docs 
within its own docs - I know that switching to vuepress would impact 
that. We should be slightly conservative about changes to the docs 
system. But saying that, this should not force us to stick with old tech 
stacks - it's not as strict as an API contract!


Perhaps the first step for vuepress would be a spike, or sharing some 
more examples - giving us a better idea about the total effort required, 
and thus letting downstream projects know when it would be likely to 
land and what the new system would look like.


_*Alternatives to vuepress?*_

Can anyone else comment on pros/cons and alternatives? I'm not familiar 
enough with the docs domain to say whether vuepress is the best choice. 
It sounds like a much better choice than the current tech, but I don't 
know what the plausible alternatives are.


vuepress github repo [1] looks promising: 15.2k stars, 281 contributors, 
1820 commits since April 2018, and 8 commits in the last 30 days from 8 
different people.


[1] https://github.com/vuejs/vuepress


On 07/01/2020 09:58, Thomas Bouron wrote:

Hello Brooklyners, and welcome into this new decade! (i.e. Happy New Year!)

To start off nicely, I would like to make a proposal regarding the Brooklyn
website/doc.

*Background*
Currently, we have 2 separate projects in `brooklyn-docs`:
- the docs are using `gitbooks` and sources live in the `master` branch.
- the rest of the website is using an ancient version `Jekyll` and sources
live in the `website` branch.

We did this because:
1. the ancient version of Jekyll requires a very specific version of ruby
which is a pain to setup locally and on the CI. Meaning it was virtually
impossible to contribute the docs alone.
2. if there is a change in the docs, it didn't make sense to update the
website part and vice-versa.
3. we wanted to automatically publish the changes live as soon as PRs were
merged.

*Issue*
The problem with this approach is that we need to have 2 CI jobs to build 2
different parts and to somehow push these live. We never managed to
automatically publish updates to website/docs through Jenkins.
We also picked a tool that is unfortunately not maintained anymore
(gitbooks) as their team is pushing for a commercial offering. Our Jekyll
setup is very old and impossible to upgrade currently, as it's using custom
plugins that would need to be rewritten.
And finally, our setup is, in my view, way too complicated and clunky to
use.

*Proposal*
I took a look at different solutions and found one that seemed to tick all
the boxes, it's called vuepress [1] and let us to:
- reconciled the website and docs under the same branch as it supports
multiple layouts, hence build both in one go.
- have a custom navigation, based on whatever configuration we write.
- defined custom blocks that can use hardcoded HTML or vue templates (we do
have custom blocks for the blueprint tours, tips, alerts, etc...)
- have custom search like gitbooks + be responsive and SEO friendly.
- have a toolchain that is maintained, open source and easy to use.

On top of that, having the full website build in one go will enable us to
push, from Jenkins, the compiled files to another branch (like `asf-pages`)
and leverage a new service called `.asf.yaml` [2] which takes care of
publishing it.

WDYT?

Best.

[1] https://vuepress.vuejs.org/

[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405038#id-.asf.yamlfeaturesforgitrepositories-WebSiteDeploymentServiceforGitRepositories

Re: [VOTE] Release Apache Brooklyn 1.0.0 [rc3]

[Aled Sage]

Hi Justin,

Thanks for spotting this and reaching out.

Looking at the license/notice generation, I think there are two things 
that went wrong for 1.0 release:


1. The maven license plugin [1] picked the wrong license for 
dependencies when there were multiple to choose from (i.e. LGPL vs 
Apache 2.0 in [2]).


2. We're trying to include far too much stuff in NOTICE. Quoting the 
really useful link you shared [3]:


        "Do not add anything to NOTICE which is not legally required."

---

We should review point 1 above to confirm there really are no licenses 
that are forbidden in apache projects. And we should review point 2 to 
change the way we generate NOTICE files so it doesn't include everything.


Aled

[1] https://github.com/ahgittin/license-audit-maven-plugin

[2] https://github.com/java-native-access/jna/blob/master/pom-jna.xml

[3] http://www.apache.org/dev/licensing-howto.html

[4] https://www.apache.org/legal/resolved.html#category-x


On 17/05/2020 10:20, Justin Mclean wrote:

Hi,

I was looking reviewing your board report and mailing list and took a look at 
your release. The current LICENSE and NOTICE are not in line with ASF policy. 
For instance, your license contains licenses that can't be used in a source 
release. I think what you have misunderstood is that you're listing the 
licenses of all dependencies rather than just what is bundled in the release. 
Your notice file also doesn't need to list dependencies but just required 
notices, content from other ALv2 notice files and relocated copyright notices. 
This is a good guide [1] if you need help on fixing this, please reach out.

Thanks,
Justin

1. http://www.apache.org/dev/licensing-howto.html

[jira] [Created] (BROOKLYN-199) Web-console activity view shows wrong entity after using browser's back-button

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-199:
--

 Summary: Web-console activity view shows wrong entity after using 
browser's back-button
 Key: BROOKLYN-199
 URL: https://issues.apache.org/jira/browse/BROOKLYN-199
 Project: Brooklyn
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Aled Sage


I had several apps deployed. I drilled into the activity view of the first 
(tomcat's install task):
  
http://brooklyn.acme.com/#v1/applications/kNkZIqsr/entities/MprRw9di/activities/subtask/jUD94noO/subtask/anx6gZVt/subtask/DOQAUXbo/subtask/bmWGsLZD

And clicked on the the stdout link, taking me to:
  http://brooklyn.acme.com/v1/activities/bmWGsLZD/stream/stdout

I then clicked the browser's back button.

When I clicked onto another entity, the "Activity" view still showed the old 
tomcat entity.
When I clicked away from the activity view and then back again, it still showed 
the tomcat entity's activity view. 
When I clicked the activity view's left arrow to go back through the call 
chain, it showed me the callers of that tomcat task. When I then clicked on a 
different entity, it went back to showing me the tomcat's install task.

When I refreshed the browser page, things started working normally again.

This was with Brooklyn 0.9.0-SNAPSHOT, using chrome 47.0.2526.73 (64-bit) on OS 
X 10.11.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BROOKLYN-184) Entities to be auto-managed, rather than calling Entities.manage()

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-184.

   Resolution: Fixed
 Assignee: Aled Sage
Fix Version/s: 0.9.0

> Entities to be auto-managed, rather than calling Entities.manage()
> --
>
> Key: BROOKLYN-184
> URL: https://issues.apache.org/jira/browse/BROOKLYN-184
> Project: Brooklyn
>  Issue Type: Improvement
>Affects Versions: 0.8.0
>Reporter: Aled Sage
>    Assignee: Aled Sage
> Fix For: 0.9.0
>
>
> As per the dev@brooklyn e-mail thread entitled "PROPOSAL] entities always 
> managed when instantiated", entities should be automatically managed, rather 
> than us having an explicit {{Entities.manage()}} method.
> http://mail-archives.us.apache.org/mod_mbox/incubator-brooklyn-dev/201509.mbox/%3C55FFE916.7030103%40gmail.com%3E
> h2. Proposal
> When an entity is instantiated (which always involves setting its parent, 
> unless it is a top-level app) then the entity will immediately be managed.
> The call to {{Entities.manage()}} will be deprecated. Its implementation will 
> check that the entity is managed - if it is not, there will be a log.warn 
> about the deprecated behaviour and the entity will be explicitly managed (for 
> backwards compatibility).
> We will deprecate the {{NonDeploymentManagementContext}} (which is only used 
> internally), and will delete it when {{Entities.manage()}} is deleted.
> h2. Historic reason
> The reason we didn't just automatically manage an entity when it is 
> constructed is mostly historic.
> There was a perceived use-case that one might instantiate an entity, 
> explicitly set config on it, and only then manage it. However, everywhere (I 
> believe) we set the entity's configuration on the {{EntitySpec}} before 
> instantiating it - that is certainly the case for YAML-based entities! The 
> only time we couldn't do that is if we have two entities (X and Y) that 
> require config values that depend on each other - e.g. X wants a config value 
> of an attributeWhenReady on Y, and vice versa. However, that (contrived?) 
> use-case does not justify the confusion of the explicit manage. We could 
> handle such use-cases by writing Java code in the entity, if we come across 
> them.
> h2. Current code
> Currently, an entity only becomes "managed" (i.e. visible to other entities) 
> when it or its parent/grandparent/etc is explicitly managed.
> This leads to two very different ways of dealing with entities:
>  * For child entities instantiated in an entity's {{init()}}, one does not
>have to call an explicit {{manage()}}.  
>This is because those children will piggie-back on the parent entity
>becoming managed at some point after it has been initialised.
>  * For top-level apps (e.g. referenced via YAML), the code that reads
>the blueprint will automatically do the {{manage()}} so folk don't need
>to worry about it.
>  * For child entities instantiated after the parent is managed, then
>the parent must explicitly call {{manage().}}  
>For example, when a cluster grows (i.e. adds additional child
>entities), then that code explicitly calls {{manage().}}
> There is overly complicated code for handling entities that have been 
> instantiated but that are not yet managed. The entity has a 
> {{NonDeploymentManagementContext}}, which allows some operations and forbids 
> others; that is swapped for the real {{ManagementContext}} when the entity 
> becomes managed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BROOKLYN-212) Auto-scaling in finite BYON cluster keeps adding on-fire entities

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-212:
--

 Summary: Auto-scaling in finite BYON cluster keeps adding on-fire 
entities
 Key: BROOKLYN-212
 URL: https://issues.apache.org/jira/browse/BROOKLYN-212
 Project: Brooklyn
  Issue Type: Bug
Reporter: Aled Sage


When using a BYON location of a limited size (e.g. four vagrant VMs), if an 
auto-scaler policy tries to resize above this number of VMs then it goes badly 
wrong!

Auto-scaling causes a new entity to be added to the cluster, but it fails to 
start because the {{location.obtain()}} call fails. That entity is left in 
quarantine, on-fire.

The auto-scaler will then try again: another new entity is added, but this also 
fails. This repeats, adding more and more entities.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BROOKLYN-209) Web-console application tab rendered badly: tree not shown

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-209:
--

 Summary: Web-console application tab rendered badly: tree not shown
 Key: BROOKLYN-209
 URL: https://issues.apache.org/jira/browse/BROOKLYN-209
 Project: Brooklyn
  Issue Type: Bug
Reporter: Aled Sage


The Applications tab in Brooklyn web-console sometimes does not show the tree. 
The section below “Applications” is just the background colour - not even 
showing an empty box for no apps. Refreshing the page fixes it.

This is with 0.9.0-SNAPSHOT.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BROOKLYN-209) Web-console application tab rendered badly: tree not shown

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage updated BROOKLYN-209:
---
Attachment: Screen Shot 2015-12-19 at 20.19.47.png

> Web-console application tab rendered badly: tree not shown
> --
>
> Key: BROOKLYN-209
> URL: https://issues.apache.org/jira/browse/BROOKLYN-209
> Project: Brooklyn
>  Issue Type: Bug
>    Reporter: Aled Sage
> Attachments: Screen Shot 2015-12-19 at 20.19.47.png
>
>
> The Applications tab in Brooklyn web-console sometimes does not show the 
> tree. The section below “Applications” is just the background colour - not 
> even showing an empty box for no apps. Refreshing the page fixes it.
> This is with 0.9.0-SNAPSHOT.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BROOKLYN-214) OutOfMemoryError (too many threads): repeated calls to AttributeWhenReady

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-214:
--

 Summary: OutOfMemoryError (too many threads): repeated calls to 
AttributeWhenReady
 Key: BROOKLYN-214
 URL: https://issues.apache.org/jira/browse/BROOKLYN-214
 Project: Brooklyn
  Issue Type: Bug
Reporter: Aled Sage


When launching Clocker, an {{OutOfMemoryError}} was encountered due to too many 
threads. The underlying cause is repeated task execution to 
{{AttributeWhenReady}}, where each task blocks a thread.

The exception encountered was:

{noformat}
2016-01-11 16:36:32,460 DEBUG o.a.b.u.c.t.BasicExecutionManager 
[brooklyn-execmanager-vzwdtuv4-5490]: Exception running task 
Task[machine.loadAverage @ h2jAHTjo <- 
ssh[uptime->machine.loadAverage]:LBUslVfG] (rethrowing): unable to
 create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
{noformat}

Shortly before the OOME, this was the resource usage:

{noformat}
2016-01-11 16:36:26,884 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector 
[brooklyn-gc]: brooklyn gc (after) - using 202 MB / 310 MB memory (122 kB 
soft); 1987 threads; storage: {datagrid={size=7, createCount=7}, refsMapSize=0, 
listsMapS
ize=0}; tasks: 1835 active, 1040 unfinished; 1425 remembered, 169790 total 
submitted)
{noformat}

Looking at a thread dump, there are 977 threads waiting for a lock on 
{{org.apache.brooklyn.camp.brooklyn.spi.dsl.methods.DslComponent$AttributeWhenReady}}},
 e.g.

{noformat}
"brooklyn-execmanager-vzwdtuv4-1859" #57280 daemon prio=5 os_prio=31 
tid=0x7fa0baef nid=0xf307 waiting for monitor entry [0x7978]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get(BrooklynDslDeferredSupplier.java:93)
- waiting to lock <0x000784bc2828> (a 
org.apache.brooklyn.camp.brooklyn.spi.dsl.methods.DslComponent$AttributeWhenReady)
at 
org.apache.brooklyn.util.core.task.ValueResolver$2.call(ValueResolver.java:322)
at 
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:342)
at 
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:493)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

The one thread holding that lock is doing:

{noformat}
"brooklyn-execmanager-vzwdtuv4-1864" #57290 daemon prio=5 os_prio=31 
tid=0x7fa0bbc19800 nid=0x76e7 waiting on condition [0x761e1000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007851a4cb8> (a 
java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at 
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:342)
at 
org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get(BrooklynDslDeferredSupplier.java:105)
- locked <0x000784bc2828> (a 
org.apache.brooklyn.camp.brooklyn.spi.dsl.methods.DslComponent$AttributeWhenReady)
at 
org.apache.brooklyn.util.core.task.ValueResolver$2.call(ValueResolver.java:322)
at 
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:342)
at 
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:493)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

Looking at the caller of 
{{org.apache.brooklyn.util.core.task.ValueResolver$2.call(ValueResolver.java:322)}},
 it's interesting to see that there are only two instances of that. This tells 
us that the other calls (to {{ValueResolver.getMaybeInternal()}}) must all have 
had a short timeout. Inside getMaybeInternal(), it waits for the given timeout 
for the resolved value, and then calls {{task.cancel(true)}} before returning.

Given that the tasks' threads are waiting for a {{synchronized}} lock, they 
cannot be interrupted. One part of the fix is to change the implementation of 
{{BrooklynDslDeferredSupplier.get(BrooklynDslDeferredSupplier.java:93)}} 

[jira] [Created] (BROOKLYN-297) Support yaml DSL for "name"

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-297:
--

 Summary: Support yaml DSL for "name"
 Key: BROOKLYN-297
 URL: https://issues.apache.org/jira/browse/BROOKLYN-297
 Project: Brooklyn
  Issue Type: Improvement
Affects Versions: 0.9.0
Reporter: Aled Sage
Priority: Minor


When providing the name of an entity in yaml, it would be good to be able to 
use DSL as we can for config.

For example, we have a catalog item for an entity that creates a docker 
container. A natural name for it would be the prefix "Container: " followed by 
the image name (as illustrated in the syntax below).

{noformat}
brooklyn.catalog:
  version: 1.0.0
  items:
  - id: docker-container
itemType: entity
item:
  type: ...

  brooklyn.parameters:
- name: imageName
  type: string

  name:
$brooklyn:formatString:
- "Container: %s"
- $brooklyn:config("imageName")

  ...
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BROOKLYN-292) jetty "broken pipe" exceptions logged at warning if browser stops loading web-console welcome page

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-292:
--

 Summary: jetty "broken pipe" exceptions logged at warning if 
browser stops loading web-console welcome page
 Key: BROOKLYN-292
 URL: https://issues.apache.org/jira/browse/BROOKLYN-292
 Project: Brooklyn
  Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Aled Sage
Priority: Minor


I started a clean (empty) Brooklyn 0.9.0, and went to the web-console at 
http://localhost:8081. While the page was still loading (i.e. bringing up the 
"Create Application" dialog), I immediately went to another web-page in my 
browser tab.

This is with chrome 50.0.2661.102 (64-bit) on OS X.

Outwardly, everything seemed to behave itself, but my Brooklyn log file had the 
exceptions below. This happens each time I try it.

{noformat}
2016-06-08 09:42:22,311 ERROR o.a.cxf.jaxrs.utils.JAXRSUtils 
[brooklyn-jetty-server-8082-qtp2099229108-33]: Problem with writing the data, 
class java.util.ArrayList, ContentType: application/json
2016-06-08 09:42:22,318 WARN  o.a.c.p.PhaseInterceptorChain 
[brooklyn-jetty-server-8082-qtp2099229108-33]: Interceptor for 
{http://resources.rest.brooklyn.apache.org/}CatalogResource has thrown 
exception, unwinding now
org.apache.cxf.interceptor.Fault: null
at 
org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleWriteException(JAXRSOutInterceptor.java:382)
 ~[org.apache.cxf-cxf-rt-frontend-jaxrs-3.1.4.jar:3.1.4]
at 
org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:257)
 ~[org.apache.cxf-cxf-rt-frontend-jaxrs-3.1.4.jar:3.1.4]
at 
org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:118)
 ~[org.apache.cxf-cxf-rt-frontend-jaxrs-3.1.4.jar:3.1.4]
at 
org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:81)
 ~[org.apache.cxf-cxf-rt-frontend-jaxrs-3.1.4.jar:3.1.4]
at 
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
 [org.apache.cxf-cxf-core-3.1.4.jar:3.1.4]
at 
org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
 [org.apache.cxf-cxf-core-3.1.4.jar:3.1.4]
at 
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
 [org.apache.cxf-cxf-core-3.1.4.jar:3.1.4]
at 
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
 [org.apache.cxf-cxf-core-3.1.4.jar:3.1.4]
at 
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
 [org.apache.cxf-cxf-rt-transports-http-3.1.4.jar:3.1.4]
at 
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
 [org.apache.cxf-cxf-rt-transports-http-3.1.4.jar:3.1.4]
at 
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
 [org.apache.cxf-cxf-rt-transports-http-3.1.4.jar:3.1.4]
at 
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
 [org.apache.cxf-cxf-rt-transports-http-3.1.4.jar:3.1.4]
at 
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
 [org.apache.cxf-cxf-rt-transports-http-3.1.4.jar:3.1.4]
at 
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:293)
 [org.apache.cxf-cxf-rt-transports-http-3.1.4.jar:3.1.4]
at 
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:217)
 [org.apache.cxf-cxf-rt-transports-http-3.1.4.jar:3.1.4]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) 
[javax.servlet-javax.servlet-api-3.1.0.jar:3.1.0]
at 
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268)
 [org.apache.cxf-cxf-rt-transports-http-3.1.4.jar:3.1.4]
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808) 
[org.eclipse.jetty-jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
 [org.eclipse.jetty-jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
at 
org.apache.brooklyn.rest.filter.BrooklynPropertiesSecurityFilter.doFilter(BrooklynPropertiesSecurityFilter.java:125)
 [org.apache.brooklyn-brooklyn-rest-server-0.9.0.jar:0.9.0]
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
 [org.eclipse.jetty-jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
at 
org.apache.brooklyn.rest.filter.HaMasterCheckFilter.doFilter(HaMasterCheckFilter.java:91)
 [org.apache.brooklyn-brooklyn-rest-server-0.9.0.jar:0.9.0]
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
 [org.eclipse

[jira] [Created] (BROOKLYN-291) Manually setting service.isUp is overwritten by enricher (causing non-deterministic test failures)

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-291:
--

 Summary: Manually setting service.isUp is overwritten by enricher 
(causing non-deterministic test failures)
 Key: BROOKLYN-291
 URL: https://issues.apache.org/jira/browse/BROOKLYN-291
 Project: Brooklyn
  Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Aled Sage


If one creates an entity and explicitly sets its service.isUp to true, then the 
default enrichers can override this value to remove it. This is because it is 
executing concurrently, and if service-up-indicators is null then it removes 
the service.isUp sensor value!

A similar thing happens for the service.state sensor.

This is non-deterministic, but can be demonstrate with this simple test:

{noformat}
for (int i = 0; i < 100; i++) {
TestEntity entity = app.addChild(EntitySpec.create(TestEntity.class));
entity.sensors().set(TestEntity.SERVICE_UP, true);
entity.sensors().set(TestEntity.SERVICE_STATE_ACTUAL, Lifecycle.RUNNING);
Thread.sleep(10);
assertEquals(entity.sensors().get(TestEntity.SERVICE_UP), Boolean.TRUE);
assertEquals(entity.sensors().get(TestEntity.SERVICE_STATE_ACTUAL), 
Lifecycle.RUNNING);
}
{noformat}

Most entities don't explicitly set the service.isUp, so it has not been a 
problem in production systems. However, it does impact tests - it is the likely 
cause of many non-deterministic test failures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BROOKLYN-295) jclouds-openstack destroyNode failed after VM failed to provision: id must be in format regionId/id

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-295:
--

 Summary: jclouds-openstack destroyNode failed after VM failed to 
provision: id must be in format regionId/id
 Key: BROOKLYN-295
 URL: https://issues.apache.org/jira/browse/BROOKLYN-295
 Project: Brooklyn
  Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Aled Sage


I attempted to deploy an app to OpenStack. It failed to provision due to 
insufficient resources. But then jclouds tried and failed to destroy the VM 
with the exception:

{noformat}
java.lang.IllegalArgumentException: id must be in format regionId/id
{noformat}

The provisioning error (shown in the OpenStack web-console) was:

{noformat}
Message  No valid host was found. Exceeded max scheduling attempts 3 for 
instance e74952b7-f495-4148-9a09-c4dee0d75e7a. Last exception: [u'Traceback 
(most recent call last):\n', u' File 
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/c
Code 500
Details  File 
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/conductor/manager.py",
 line 669, in build_instances instances[0].uuid) File 
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/scheduler/utils.py",
 line 172, in populate_retry raise exception.NoValidHost(reason=msg)
Created  June 8, 2016, 10:33 a.m.
{noformat}

The Brooklyn debug log shows:

{noformat}
2016-06-08 11:33:45,215 DEBUG o.a.b.l.j.JcloudsLocation 
[brooklyn-execmanager-yA1Lh5GF-504]: jclouds using template 
{image={id=RegionOne/e16a7bca-7363-45b1-bd6a-02479d4cea77, 
providerId=e16a7bca-7363-45b1-bd6a-02479d4cea77, name=CentOS 7, 
location={scope=REGION, id=RegionOne, description=RegionOne, 
parent=openstack-nova}, os={family=centos, name=CentOS 7, version=, 
description=CentOS 7, is64Bit=true}, description=CentOS 7, status=AVAILABLE, 
loginUser=root}, hardware={id=RegionOne/2, providerId=2, name=m1.small, 
location={scope=REGION, id=RegionOne, description=RegionOne, 
parent=openstack-nova}, processors=[{cores=1.0, speed=1.0}], ram=2048, 
volumes=[{type=LOCAL, size=20.0, bootDevice=true, durable=true}], 
supportsImage=ALWAYS_TRUE}, location={scope=REGION, id=RegionOne, 
description=RegionOne, parent=openstack-nova}, options={loginUser=centos, 
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true, 
securityGroups=[VPN_local], 
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s, 
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm, 
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node, 
brooklyn-server-creation-date=2016-06-08-1133}, 
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false, 
keyPairName=openstack, configDrive=false}} / options {loginUser=centos, 
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true, 
securityGroups=[VPN_local], 
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s, 
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm, 
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node, 
brooklyn-server-creation-date=2016-06-08-1133}, 
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false, 
keyPairName=openstack, configDrive=false} to provision machine in 
openstack-nova:https:/acme.com:5000/v2.0@VanillaSoftwareProcessImpl{id=rj29jfrfvy}
2016-06-08 11:33:45,215 DEBUG o.a.b.l.j.JcloudsLocation 
[brooklyn-execmanager-yA1Lh5GF-504]: NOTE: unused flags passed to obtain VM in 
openstack-nova:https://acme.com:5000/v2.0@VanillaSoftwareProcessImpl{id=rj29jfrfvy}:
 {jclouds.keystone.credential-type=, 
machineCreationSemaphore=java.util.concurrent.Semaphore@78d9e87b[Permits = 
2147483643]}
2016-06-08 11:33:45,215 DEBUG jclouds.compute 
[brooklyn-execmanager-yA1Lh5GF-504]: >> running 1 node 
group(brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29) 
location(RegionOne) image(RegionOne/e16a7bca-7363-45b1-bd6a-02479d4cea77) 
hardwareProfile(RegionOne/2) options({loginUser=centos, 
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true, 
securityGroups=[VPN_local], 
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s, 
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm, 
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node, 
brooklyn-server-creation-date=2016-06-08-1133}, 
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false, 
keyPairName=openstack, configDrive=false})
2016-06-08 11:33:45,218 DEBUG o.j.l.s.f.RegionIdsFromConfiguration 
[brooklyn-execmanager-yA1Lh5GF-504]: no jclouds.regions configured for provider 
openstack-nova

2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> GET 
https://acme.com:8774/v2/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495-

[jira] [Created] (BROOKLYN-298) sshj hangs (waiting for shell to finish) after script completed - maybe VPN went down+up during exec

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-298:
--

 Summary: sshj hangs (waiting for shell to finish) after script 
completed - maybe VPN went down+up during exec
 Key: BROOKLYN-298
 URL: https://issues.apache.org/jira/browse/BROOKLYN-298
 Project: Brooklyn
  Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Aled Sage


I was deploying an app whose launch command started docker and pulled an image. 
The task hung, showing in the web-console:

{noformat}
In progress - SSH executing, launching VanillaSoftwareProcessImpl{id=nisq2gz4yi}
{noformat}

I believe this is because my VPN disconnected and then reconnected, and our 
sshj command keeps waiting for the result - even though the command has 
finished executing.

Looking at the target VM, the command has completed (and the script uploaded by 
SshjTool has been deleted). There is no evidence of any Brooklyn-initiated 
commands executing, according to {{ps aux}}.

Drilling into the activity view in the Brooklyn web-console, the currently 
executing thread shows:

{noformat}
SSH executing, launching VanillaSoftwareProcessImpl{id=nisq2gz4yi}

Task[ssh: launching VanillaSoftwareProcessImpl{id=nisq2gz4yi}]@TPnVc8Qs
Submitted by SoftlyPresent[value=Task[launch (main)]@mvL4OvdH]

In progress, thread waiting (timed) on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@408df99d
At: net.schmizz.concurrent.Promise.tryRetrieve(Promise.java:168)
net.schmizz.concurrent.Promise.retrieve(Promise.java:137)
net.schmizz.concurrent.Event.await(Event.java:103)

net.schmizz.sshj.connection.channel.AbstractChannel.join(AbstractChannel.java:282)

org.apache.brooklyn.util.core.internal.ssh.sshj.SshjTool$ShellAction.create(SshjTool.java:1012)

org.apache.brooklyn.util.core.internal.ssh.sshj.SshjTool$ShellAction.create(SshjTool.java:925)

org.apache.brooklyn.util.core.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:630)

org.apache.brooklyn.util.core.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:616)

org.apache.brooklyn.util.core.internal.ssh.sshj.SshjTool$1.run(SshjTool.java:331)

org.apache.brooklyn.util.core.internal.ssh.sshj.SshjTool.execScript(SshjTool.java:326)

org.apache.brooklyn.util.core.task.system.internal.ExecWithLoggingHelpers$1.exec(ExecWithLoggingHelpers.java:82)

org.apache.brooklyn.util.core.task.system.internal.ExecWithLoggingHelpers$3.apply(ExecWithLoggingHelpers.java:166)

org.apache.brooklyn.util.core.task.system.internal.ExecWithLoggingHelpers$3.apply(ExecWithLoggingHelpers.java:164)
org.apache.brooklyn.util.pool.BasicPool.exec(BasicPool.java:146)

org.apache.brooklyn.location.ssh.SshMachineLocation.execSsh(SshMachineLocation.java:611)

org.apache.brooklyn.location.ssh.SshMachineLocation$13.execWithTool(SshMachineLocation.java:790)

org.apache.brooklyn.util.core.task.system.internal.ExecWithLoggingHelpers.execWithLogging(ExecWithLoggingHelpers.java:164)

org.apache.brooklyn.util.core.task.system.internal.ExecWithLoggingHelpers.execScript(ExecWithLoggingHelpers.java:80)

org.apache.brooklyn.location.ssh.SshMachineLocation.execScript(SshMachineLocation.java:774)

org.apache.brooklyn.entity.software.base.AbstractSoftwareProcessSshDriver.execute(AbstractSoftwareProcessSshDriver.java:272)

org.apache.brooklyn.entity.software.base.lifecycle.ScriptHelper.executeInternal(ScriptHelper.java:366)

org.apache.brooklyn.entity.software.base.lifecycle.ScriptHelper$8.call(ScriptHelper.java:287)

org.apache.brooklyn.entity.software.base.lifecycle.ScriptHelper$8.call(ScriptHelper.java:285)

org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)

org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
{noformat}

Running {{netstat -antp TCP}} on my local machine, I still see an established 
ssh connection:

{noformat}
tcp4   0  0  10.104.3.10.54535  10.104.1.193.22ESTABLISHED
{noformat}

I do *not* see a corresponding entry when I run {{sudo netsat -anp}} on the 
target VM.

---
Looking in the Brooklyn code at {{SshjTool$ShellAction.create}}, I wonder what 
else we could call on sshj to check if our connection is ok and/or the command 
has actually completed. We are already calling {{shell.isOpen()}} and 
{{session.getExitStatus()!=null}}. We could add calls to {{session.isOpen()}}, 
{{session.getExitSignal()}} and/or {{session.getExitWasCoreDumped()}}.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BROOKLYN-289) Broken link in brooklyn-client README.md

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-289.

Resolution: Duplicate

> Broken link in brooklyn-client README.md
> 
>
> Key: BROOKLYN-289
> URL: https://issues.apache.org/jira/browse/BROOKLYN-289
> Project: Brooklyn
>  Issue Type: Bug
>Reporter: John McCabe
>Priority: Trivial
>  Labels: documentation
>
> The README.md links to a 404'ing *Runtime README* file:
> - https://github.com/apache/brooklyn-client/blob/master/README
> Not clear what this should be.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BROOKLYN-293) Web-console should not allow directory listing

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-293.

   Resolution: Fixed
Fix Version/s: 0.10.0

> Web-console should not allow directory listing
> --
>
> Key: BROOKLYN-293
> URL: https://issues.apache.org/jira/browse/BROOKLYN-293
> Project: Brooklyn
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Aled Sage
>Priority: Minor
> Fix For: 0.10.0
>
>
> A customer's security audit spotted that directory browsing is enabled in the 
> Brooklyn web-console. For example, http://localhost:8081/assets.
> In general, it is more secure to forbid directory listing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BROOKLYN-282) Avoid showing secret effector parameters

[Aled Sage (JIRA)]

[ 
https://issues.apache.org/jira/browse/BROOKLYN-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325856#comment-15325856
 ] 

Aled Sage commented on BROOKLYN-282:


See PRs:
* https://github.com/apache/brooklyn-server/pull/185
* https://github.com/apache/brooklyn-ui/pull/29
* https://github.com/apache/brooklyn-server/pull/195

> Avoid showing secret effector parameters
> 
>
> Key: BROOKLYN-282
> URL: https://issues.apache.org/jira/browse/BROOKLYN-282
> Project: Brooklyn
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Aled Sage
>
> A customer has an effector that takes sensitive data as a parameter (e.g. a 
> password).
> There are three places where the value would currently be visible:
> 1. When entering the value (if invoking the effector from the web-console).
> 2. When showing the activity info (in the activity view of the web-console).
> 3. In the logs.
> Elsewhere (e.g. in config and attributes), the default is to infer if it is a 
> secret from its config name. This is based on whether it contains any of: 
> password, passwd, credential, secret, private, access.cert or access.key. The 
> {{org.apache.brooklyn.core.config.Sanitizer}} is used to replace these values 
> with "". We are careful to call {{Sanitizer.sanitize()}} before 
> logging.
> The web-console (i.e. in JavaScript) does something similar for blurring the 
> config key values for secrets.
> As a first pass, we could apply the same logic to effector parameters. When 
> entering the value, we would mask it out as it was entered. And we would 
> subsequently use the {{Sanitizer}} before generating the activity info or 
> logging.
> ---
> Longer term, we could also add to ConfigKey (and Sensor) a method 
> {{Optional isSecret()}}. If absent, it means no explicit info has 
> been given so we can fall back to our inference logic.
> The problem with the {{isSecret()}} method is that the Sanitizer sometimes 
> works with a map of string keys. In that context, we don't have the strongly 
> typed config key any more so we can't query it to check if a given key is 
> secret. We'd need to carefully review the effector parameter usage to see if 
> we are ever trying to log (or generate activity info) when only working with 
> a map of string keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BROOKLYN-273) VanillaSoftwareProcess.restart() should call pre- and post-launch

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-273.

   Resolution: Fixed
Fix Version/s: 0.10.0

> VanillaSoftwareProcess.restart() should call pre- and post-launch
> -
>
> Key: BROOKLYN-273
> URL: https://issues.apache.org/jira/browse/BROOKLYN-273
> Project: Brooklyn
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Aled Sage
> Fix For: 0.10.0
>
>
> If calling restart() on a VanillaSoftwareProcess entity, to just restart the 
> process, it will call "launch" (skipping the install + customize phases).
> I'd expect the pre-launch and post-launch commands to also be executed, but 
> they are not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BROOKLYN-299) Stopping app takes long time: blocked on LocalUsageManager.recordLocationEvent, waiting for ssh connection timeout

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-299:
--

 Summary: Stopping app takes long time: blocked on 
LocalUsageManager.recordLocationEvent, waiting for ssh connection timeout
 Key: BROOKLYN-299
 URL: https://issues.apache.org/jira/browse/BROOKLYN-299
 Project: Brooklyn
  Issue Type: Bug
Reporter: Aled Sage


With brooklyn 0.10.0-snapshot master...

It is sometimes taking a long time to stop locations (and this is also 
affecting starting them) - a few minutes extra.

The reason is that it's blocked, waiting on the mutex in 
{{LocalUsageManager.recordLocationEvent}}, or 
{{LocalUsageManager.recordApplicationEvent}}!

While holding the lock, it calls {{location.toMetadataRecord}}. For some 
location types, this causes an ssh call to infer the machine details. That in 
itself is a bad idea. However, this is made even worse because we call it when 
unmanaging a {{JcloudsSshMachineLocation}} - i.e. when the VM has already been 
terminated. This means the ssh call times out, potentially after a couple of 
minutes.

The stacktrace for two such threads are shown below (collected using jstack):

{noformat}
"brooklyn-execmanager-hgWZmzF2-7587" daemon prio=5 tid=0x7f94ee04b800 
nid=0x14b53 waiting for monitor entry [0x79d91000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.brooklyn.core.mgmt.internal.LocalUsageManager.recordApplicationEvent(LocalUsageManager.java:247)
- waiting to lock <0x0007c06ca2f0> (a java.lang.Object)
at 
org.apache.brooklyn.core.entity.AbstractApplication.recordApplicationEvent(AbstractApplication.java:263)
at 
org.apache.brooklyn.core.entity.AbstractApplication.setExpectedStateAndRecordLifecycleEvent(AbstractApplication.java:258)
at 
org.apache.brooklyn.core.entity.AbstractApplication.stop(AbstractApplication.java:221)
at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
at 
groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
at 
org.apache.brooklyn.util.groovy.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.java:191)
at 
org.apache.brooklyn.core.mgmt.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:309)
at 
org.apache.brooklyn.core.mgmt.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:333)
at 
org.apache.brooklyn.core.mgmt.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:256)
at 
org.apache.brooklyn.core.effector.MethodEffector.call(MethodEffector.java:149)
at 
org.apache.brooklyn.core.entity.trait.Startable$StopEffectorBody.call(Startable.java:68)
at 
org.apache.brooklyn.core.entity.trait.Startable$StopEffectorBody.call(Startable.java:60)
at 
org.apache.brooklyn.core.effector.EffectorTasks$EffectorBodyTaskFactory$1.call(EffectorTasks.java:82)
at 
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)
at 
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"brooklyn-execmanager-hgWZmzF2-7583" daemon prio=5 tid=0x7f94ef137000 
nid=0xf67f waiting on condition [0x79476000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007efa1> (a 
java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
at java.util.concurrent.FutureTask.get(FutureTask.java:187)
at 
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
at org.apache.brooklyn.util.core.task.

[jira] [Resolved] (BROOKLYN-271) ConfigKey card shows object hashCode of default values

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-271.

   Resolution: Fixed
Fix Version/s: 0.10.0

> ConfigKey card shows object hashCode of default values
> --
>
> Key: BROOKLYN-271
> URL: https://issues.apache.org/jira/browse/BROOKLYN-271
> Project: Brooklyn
>  Issue Type: Documentation
>Reporter: Jose Carrasco
>Priority: Minor
> Fix For: 0.10.0
>
>
> ConfigKey cards of Entities show the object references in Brooklyn Catalog 
> when a default value is a object. Take a look to the following screenshot:
> !https://www.dropbox.com/s/rav70zzleyhnd4q/objectId.png?dl=1!
> As you can see the default value is composed by: 
> {code}
> getClass().getName() + '@' + Integer.toHexString(hashCode())
> {code}
> Probably just class name should be shown here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BROOKLYN-284) Deadlock in entity creation, when toString() calls getConfig

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-284.

   Resolution: Fixed
Fix Version/s: 0.10.0

> Deadlock in entity creation, when toString() calls getConfig
> 
>
> Key: BROOKLYN-284
> URL: https://issues.apache.org/jira/browse/BROOKLYN-284
> Project: Brooklyn
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Aled Sage
> Fix For: 0.10.0
>
>
> If an entity's toString calls getConfig() to retrieve a config value that is 
> set on the entity, it deadlocks during the initial creation of the entity.
> {noformat}
> Java stack information for the threads listed above:
> ===
> "brooklyn-execmanager-o0dJOc0a-1":
> at 
> org.apache.brooklyn.core.mgmt.internal.EntityManagementSupport.getExecutionContext(EntityManagementSupport.java:354)
> - waiting to lock <0x0007fa6ff568> (a 
> org.apache.brooklyn.core.mgmt.internal.EntityManagementSupport)
> at 
> org.apache.brooklyn.core.entity.AbstractEntity.getExecutionContext(AbstractEntity.java:1566)
> - locked <0x0007fa6fef90> (a 
> org.apache.brooklyn.core.entity.EntityTypeTest$EntityWithToStringAccessingConfig)
> at 
> org.apache.brooklyn.core.objs.AdjunctConfigMap.getConfig(AdjunctConfigMap.java:84)
> at 
> org.apache.brooklyn.core.config.internal.AbstractConfigMapImpl.getConfig(AbstractConfigMapImpl.java:50)
> at 
> org.apache.brooklyn.core.objs.AbstractEntityAdjunct$BasicConfigurationSupport.get(AbstractEntityAdjunct.java:291)
> at 
> org.apache.brooklyn.enricher.stock.Transformer.getTransformation(Transformer.java:65)
> at 
> org.apache.brooklyn.enricher.stock.AbstractTransformer.compute(AbstractTransformer.java:139)
> at 
> org.apache.brooklyn.enricher.stock.AbstractTransformer.onEvent(AbstractTransformer.java:133)
> at 
> org.apache.brooklyn.core.mgmt.internal.LocalSubscriptionManager$1.run(LocalSubscriptionManager.java:149)
> at 
> org.apache.brooklyn.util.concurrent.CallableFromRunnable.call(CallableFromRunnable.java:43)
> at 
> org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
> at 
> org.apache.brooklyn.util.core.task.SingleThreadedScheduler$1.call(SingleThreadedScheduler.java:116)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "main":
> at 
> org.apache.brooklyn.core.entity.AbstractEntity.getExecutionContext(AbstractEntity.java:1566)
> - waiting to lock <0x0007fa6fef90> (a 
> org.apache.brooklyn.core.entity.EntityTypeTest$EntityWithToStringAccessingConfig)
> at 
> org.apache.brooklyn.core.entity.internal.EntityConfigMap.getConfig(EntityConfigMap.java:117)
> at 
> org.apache.brooklyn.core.config.internal.AbstractConfigMapImpl.getConfig(AbstractConfigMapImpl.java:50)
> at 
> org.apache.brooklyn.core.entity.AbstractEntity$BasicConfigurationSupport.get(AbstractEntity.java:1180)
> at 
> org.apache.brooklyn.core.entity.AbstractEntity.getConfig(AbstractEntity.java:1266)
> at 
> org.apache.brooklyn.core.entity.EntityTypeTest$EntityWithToStringAccessingConfig.toString(EntityTypeTest.java:300)
> at 
> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:304)
> at 
> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:276)
> at 
> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:230)
> at 
> ch.qos.logback.classic.spi.LoggingEvent.(LoggingEvent.java:114)
> at 
> ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:447)
> at 
> ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:403)
> at ch.qos.logback.classic.Logger.debug(Logger.java:522)
> at 
> org.apache.brooklyn.core.mgmt.internal.LocalSubscriptionManager.subscribe(LocalSubscriptionManager.java:114)
> - locked <0x0007f955e438> (a 
> org.apache.brooklyn.core.mgmt.internal.LocalSubscriptionManager)
> at 
> org.apache.brooklyn.core.mgmt.internal.QueueingSubscriptionManager.startDelegatingForSubscribing(Q

[jira] [Commented] (BROOKLYN-287) MySQL fails on CentOS 7

[Aled Sage (JIRA)]

[ 
https://issues.apache.org/jira/browse/BROOKLYN-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325854#comment-15325854
 ] 

Aled Sage commented on BROOKLYN-287:


As commented in the duplicate (BROOKLYN-300)...

To work It needs {{yum install -y perl perl-Data-Dumper}} (without the latter, 
we get the error {{FATAL ERROR: please install the following Perl modules 
before executing 
/tmp/installs/MySqlNode_5.6.26/mysql-5.6.26-linux-glibc2.5-x86_64/scripts/mysql_install_db:
 Data::Dumper}}).

> MySQL fails on CentOS 7
> ---
>
> Key: BROOKLYN-287
> URL: https://issues.apache.org/jira/browse/BROOKLYN-287
> Project: Brooklyn
>  Issue Type: Bug
> Environment: CentOS 7.2
>Reporter: Duncan Godwin
>
> The MySQL entity fails to install on CentOS7 on all tested clouds
> The customizing stage fails with the stderr:
> /tmp/brooklyn-20160602-141451821-svI6-customizing_MySqlNodeImpl_id_S.sh: 
> /home/users/username/brooklyn-managed-processes/installs/MySqlNode_5.6.26/mysql-5.6.26-linux-glibc2.5-x86_64/scripts/mysql_install_db:
>  /usr/bin/perl: bad interpreter: No such file or directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BROOKLYN-300) MySQL provisioning fails on centos when perl not on base OS

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-300.

Resolution: Duplicate

> MySQL provisioning fails on centos when perl not on base OS
> ---
>
> Key: BROOKLYN-300
> URL: https://issues.apache.org/jira/browse/BROOKLYN-300
> Project: Brooklyn
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Aled Sage
>
> When deploying to a (very) stripped down CentOS 7 image, MySQL failed because 
> perl was not pre-installed.
> To work It needs {{yum install -y perl perl-Data-Dumper}} (without the 
> latter, we get the error {{FATAL ERROR: please install the following Perl 
> modules before executing 
> /tmp/installs/MySqlNode_5.6.26/mysql-5.6.26-linux-glibc2.5-x86_64/scripts/mysql_install_db:
>  Data::Dumper}}).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BROOKLYN-297) Support yaml DSL for "name"

[Aled Sage (JIRA)]

[ 
https://issues.apache.org/jira/browse/BROOKLYN-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331936#comment-15331936
 ] 

Aled Sage commented on BROOKLYN-297:


There is a workaround for this, using the config {{defaultDisplayName}}. See:
* https://github.com/apache/brooklyn-server/pull/200
* https://github.com/apache/brooklyn-server/pull/201

For example:

{noformat}
brooklyn.catalog:
  version: 1.0.0
  items:
  - id: docker-container
itemType: entity
item:
  type: ...
  brooklyn.parameters:
- name: imageName
  type: string
  brooklyn.config:
defaultDisplayName:
  $brooklyn:formatString:
  - "Container: %s"
  - $brooklyn:config("imageName")
  ...
{noformat}

Note this only works for some things in the DSL, because the 
{{defaultDisplayName}} is resolved in the entity's {{init()}} method. 
Referencing config on your own entity will work, but potentially not 
referencing other entities (because they might not be managed/instantiated 
yet), and not using "attributeWhenReady" (because that would block the entity's 
initialisation).

> Support yaml DSL for "name"
> ---
>
> Key: BROOKLYN-297
> URL: https://issues.apache.org/jira/browse/BROOKLYN-297
> Project: Brooklyn
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Aled Sage
>Priority: Minor
>
> When providing the name of an entity in yaml, it would be good to be able to 
> use DSL as we can for config.
> For example, we have a catalog item for an entity that creates a docker 
> container. A natural name for it would be the prefix "Container: " followed 
> by the image name (as illustrated in the syntax below).
> {noformat}
> brooklyn.catalog:
>   version: 1.0.0
>   items:
>   - id: docker-container
> itemType: entity
> item:
>   type: ...
>   brooklyn.parameters:
> - name: imageName
>   type: string
>   name:
> $brooklyn:formatString:
> - "Container: %s"
> - $brooklyn:config("imageName")
>   ...
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BROOKLYN-282) Avoid showing secret effector parameters

[Aled Sage (JIRA)]

[ 
https://issues.apache.org/jira/browse/BROOKLYN-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331376#comment-15331376
 ] 

Aled Sage commented on BROOKLYN-282:


Note that with 0.10.0 and later, it will be important to always run with 
{{org.apache.cxf}} logging above debug. Otherwise, you will likely get log 
messages like that below:

{noformat}
brooklyn.debug.log:2016-06-14 21:43:28,862 DEBUG o.a.c.s.i.AbstractInvoker 
[brooklyn-jetty-server-8082-qtp1970185138-38]: Invoking method public 
javax.ws.rs.core.Response 
org.apache.brooklyn.rest.resources.EffectorResource.invoke(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.util.Map)
 on object org.apache.brooklyn.rest.resources.EffectorResource@3c64f630 with 
params [l4p6u3phln, iletlkpn4a, resetPassword, 0, {newPassword=mypassword, 
secretPin=1234567}].
{noformat}

The default logging level should be right since 
https://github.com/apache/brooklyn-server/pull/198 was merged, which changed 
{{logging/logback-includes/src/main/resources/brooklyn/logback-logger-excludes.xml}}
 to add:

{noformat}

{noformat}

> Avoid showing secret effector parameters
> 
>
> Key: BROOKLYN-282
> URL: https://issues.apache.org/jira/browse/BROOKLYN-282
> Project: Brooklyn
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Aled Sage
>
> A customer has an effector that takes sensitive data as a parameter (e.g. a 
> password).
> There are three places where the value would currently be visible:
> 1. When entering the value (if invoking the effector from the web-console).
> 2. When showing the activity info (in the activity view of the web-console).
> 3. In the logs.
> Elsewhere (e.g. in config and attributes), the default is to infer if it is a 
> secret from its config name. This is based on whether it contains any of: 
> password, passwd, credential, secret, private, access.cert or access.key. The 
> {{org.apache.brooklyn.core.config.Sanitizer}} is used to replace these values 
> with "". We are careful to call {{Sanitizer.sanitize()}} before 
> logging.
> The web-console (i.e. in JavaScript) does something similar for blurring the 
> config key values for secrets.
> As a first pass, we could apply the same logic to effector parameters. When 
> entering the value, we would mask it out as it was entered. And we would 
> subsequently use the {{Sanitizer}} before generating the activity info or 
> logging.
> ---
> Longer term, we could also add to ConfigKey (and Sensor) a method 
> {{Optional isSecret()}}. If absent, it means no explicit info has 
> been given so we can fall back to our inference logic.
> The problem with the {{isSecret()}} method is that the Sanitizer sometimes 
> works with a map of string keys. In that context, we don't have the strongly 
> typed config key any more so we can't query it to check if a given key is 
> secret. We'd need to carefully review the effector parameter usage to see if 
> we are ever trying to log (or generate activity info) when only working with 
> a map of string keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BROOKLYN-286) merge config keys map values, where appropriate

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-286.

   Resolution: Fixed
Fix Version/s: 0.10.0

> merge config keys map values, where appropriate
> ---
>
> Key: BROOKLYN-286
> URL: https://issues.apache.org/jira/browse/BROOKLYN-286
> Project: Brooklyn
>  Issue Type: New Feature
>Affects Versions: 0.9.0
>Reporter: Aled Sage
> Fix For: 0.10.0
>
>
> See the email discussion on dev@brooklyn.apache.org, subject "[PROPOSAL] 
> merging config keys", initially kicked off 25/05/2016, 12:12.
> Below is a copy of that initial proposal, which has then been further 
> discussed on the mailing list.
> TL;DR: we should merge config when overriding entities/locations, where it's 
> obvious that such behaviour is desired. For example, where an entity type 
> defines shell.env, then a new entity extending this type should inherit and 
> add to those values.
> _*REQUIREMENTS*_
> _*shell.env in entities*_
> When extending an existing entity type in YAML, it is not possible to extend 
> the set of environment variables. Instead, if the sub-type declares shell.env 
> it will override the inherited values.
> For example, consider the catalog items below:
># Catalog
>brooklyn.catalog:
>   items:
>   - id: machine-with-env
> item:
>   type:
>org.apache.brooklyn.entity.software.base.VanillaSoftwareProcess
>   brooklyn.config:
> shell.env:
>   ENV1: myEnv1
># Blueprint
>location: ...
>services:
>- type: machine-with-env
>   brooklyn.config:
> shell.env:
>   ENV2: myEnv2
> launch.command: echo "ENV1=$ENV1, ENV2=$ENV2"
> A user might well expect the launch.command to have myEnv1 and myEnv2. 
> However, it does not get the ENV1 environment variable. This is a real pain 
> when trying to customize stock blueprints.
> We propose that the shell.env map should be *merged*.
> _*provisioning.properties*_
> An entity can be configured with provisioning.properties. These are passed to 
> the location when obtaining a new machine. They supplement and override the 
> values configured on the location. However, for templateOptions the 
> expected/desired behaviour would be to merge the options.
> Consider the blueprint below:_*
> *_
>location:
>   minCores: 1
>   templateOptions:
> networks: myNetwork
>services:
>- type: org.apache.brooklyn.entity.machine.MachineEntity
>   brooklyn.config:
> provisioning.properties:
>   minRam: 2G
>   templateOptions:
> tags: myTag
> A user might well expect the VM to be created with the given networks and 
> tags. However, currently the templateOptions in provisoining.properties will 
> override the existing value, rather than being merged with it.
> We propose that the templateOptions map should be *merged*.
> Valentin made a start to fix this in 
> https://github.com/apache/brooklyn-server/pull/151.
> _*_*provisioning.properties in sub-entities*_
> *_
> A similar argument holds for when extending an entity-type in YAML.
> If the super-type declares template options, then any additional 
> provisioning.properties declared on the entity sub-type should be *merged* 
> (including merging the templateOptions map contained within it).
> _*files.preinstall, templates.preinstall, etc*_
> The same applies for the map config for: files.preinstall, 
> templates.preinstall, files.install, templates.install, files.runtime and 
> templates.runtime.
> We propose that these maps get *merged* with the value defined in the 
> super-type.
> _*Overriding default values*_
> For default values in the super-type, we propose that this value *does* get 
> overridden, rather than merged.
> For example, in the blueprint below we suggest that the launch-command in the 
> sub-type should have ENV2 but not ENV_IN_DEFAULT.
>brooklyn.catalog:
>   items:
>   - id: machine-with-env
> version: 1.0.0
> item:
>   type:
>org.apache.brooklyn.entity.software.base.VanillaSoftwareProcess
>   brooklyn.parameters:
>   - name: shell.env
> default:
>   ENV_IN_DEFAULT: myEnvInDefault
>   - id: machine-with-env-2
> version: 1.0.0
> item:
>   type: machine-with-env
>   brooklyn.config:
> shell.env:
>   ENV2: myEnv2
> la

[jira] [Commented] (BROOKLYN-272) Non-deterministic test failures in jenkins

[Aled Sage (JIRA)]

[ 
https://issues.apache.org/jira/browse/BROOKLYN-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305409#comment-15305409
 ] 

Aled Sage commented on BROOKLYN-272:


Also encountered a non-deterministic failure of 
`LoopOverGroupMembersTestCaseTest.testMultipleChildrenOneOfWhichFails` - see 
https://github.com/apache/brooklyn-server/pull/166 for details. I have not yet 
disabled that test.

> Non-deterministic test failures in jenkins
> --
>
> Key: BROOKLYN-272
> URL: https://issues.apache.org/jira/browse/BROOKLYN-272
> Project: Brooklyn
>  Issue Type: Bug
>    Reporter: Aled Sage
>
> There are several non-deterministic tests that fail occasionally in the 
> apache jenkins build. We should fix those tests: either the underlying bug if 
> there is one, or the way to the test has been written to make it more 
> deterministic.
> Having build failures unrelated to the changes being made in a PR (and on 
> master) is extremely disruptive for development. We must avoid that wherever 
> possible.
> I therefore suggest we disable these non-deterministic failing tests 
> immediately, and use this jira issue to track fixing them. We can annotate 
> the tests with
> {noformat}
> @Test(groups={"WIP", "Non-deterministic-failure"})
> {noformat}
>  to make them easy to find. (note the "Non-deterministic-failure" group is 
> not used by any profiles, it's just there for searching purposes).
> The tests I have identified from recent build failures are:
> * {{BasicStartableTest.testTransitionsThroughLifecycles}} - see 
> https://issues.apache.org/jira/browse/BROOKLYN-256
> * {{AbstractGeoDnsServiceTest.testFiltersForRunningEntities}}, failing with:
> {noformat}
> org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed 
> succeeds-eventually, 69 attempts, 30003ms elapsed: AssertionError: 
> val={R48yHLqg=, eBHFc4Qd=}
> at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
> at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:703)
> at 
> org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:92)
> at 
> org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
> at 
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
> at 
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
> at 
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:89)
> at 
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:84)
> at 
> org.apache.brooklyn.entity.dns.AbstractGeoDnsServiceTest.testFiltersForRunningEntities(AbstractGeoDnsServiceTest.java:263)
> {noformat}
> * {{LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers}}, 
> failing with:
> {noformat}
> org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed 
> succeeds-eventually, 29 attempts, 10002ms elapsed: AssertionError: 
> actual=[18.0, 21.0, 19.0, 0.0, 20.0, 0.0, 20.0, 36.0, 19.0, 0.0, 19.0, 21.0, 
> 21.0, 19.0, 21.0, 36.0, 21.0, 19.0, 18.0, 36.0]; expected=[20.0, 20.0, 20.0, 
> 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 
> 20.0, 20.0, 20.0, 20.0] expected [20.0] but found [0.0]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:494)
> at org.testng.Assert.assertEquals(Assert.java:207)
> at 
> org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkrates(AbstractLoadBalancingPolicyTest.java:122)
> at 
> org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest$2.run(AbstractLoadBalancingPolicyTest.java:138)
> at 
> org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
> at 
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
> at 
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
> at 
> org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkratesEventually(AbstractLoadBalancingPolicyTest.java:136)
> at 
> org.apache.brooklyn.policy.loadbalancing.LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers(LoadBalancingPolicyConcurrencyTest.java:101)
> {noformat}
> * {{PollerTest.testFeedContinuesWhenPollerThrows}}, failing with:
> {noformat}
> 
> testFeedContinuesWhenPollerThrows(org.apache.brooklyn.core.fee

[jira] [Resolved] (BROOKLYN-285) Dangling reference in Propagator.producer: can cause subscription to self, and infinite subscription loop

[Aled Sage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BROOKLYN-285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aled Sage resolved BROOKLYN-285.

   Resolution: Fixed
Fix Version/s: 0.10.0

> Dangling reference in Propagator.producer: can cause subscription to self, 
> and infinite subscription loop
> -
>
> Key: BROOKLYN-285
> URL: https://issues.apache.org/jira/browse/BROOKLYN-285
> Project: Brooklyn
>  Issue Type: Bug
>Affects Versions: 0.9.0
>    Reporter: Aled Sage
> Fix For: 0.10.0
>
>
> When the "producer" entity of a Propagator enricher is unmanaged, it can 
> cause rebind to resolve the dangling reference value as {{null}}. In 0.9.0, 
> that is treated as the producer being "self". This can cause an infinite 
> loop, where the Propagator subscribes to its own sensor, and every time its 
> sensor changes it resets the sensor value.
> To reproduce, deploy this app:
> {noformat}
> location: localhost
> services:
> - type: org.apache.brooklyn.entity.stock.BasicApplication
>   brooklyn.children:
>   - type: org.apache.brooklyn.entity.machine.MachineEntity
> {noformat}
> And programmatically add a Propagator enricher to the top-level app, such as:
> {noformat}
> appId = "u4VDoIOa";
> childId = "LPYT4g27";
> sensor = org.apache.brooklyn.core.sensor.Sensors.newSensor(String.class, 
> "host.name");
> app = mgmt.getEntityManager().getEntity(appId);
> child = mgmt.getEntityManager().getEntity(childId);
> app.enrichers().add(org.apache.brooklyn.api.sensor.EnricherSpec.create(org.apache.brooklyn.enricher.stock.Propagator.class)
> .configure("enricher.producer", child)
> .configure("enricher.propagating.inclusions", 
> com.google.common.collect.ImmutableList.of(sensor)));
> {noformat}
> Then unmanange the child entity (e.g. via the advanced tab in the 
> web-console), and restart Brooklyn.
> The CPU goes through the roof, and the debug log shows log messages like:
> {noformat}
> 2016-05-31 09:02:22,599 DEBUG o.a.b.c.m.i.LocalSubscriptionManager 
> [brooklyn-execmanager-gfOvVoKq-3]: 388000 events for subscriber 
> Subscription[qF9YFEBA;BasicApplicationImpl{id=u4VDoIOa}@u4VDoIOa:host.name]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BROOKLYN-285) Dangling reference in Propagator.producer: can cause subscription to self, and infinite subscription loop

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-285:
--

 Summary: Dangling reference in Propagator.producer: can cause 
subscription to self, and infinite subscription loop
 Key: BROOKLYN-285
 URL: https://issues.apache.org/jira/browse/BROOKLYN-285
 Project: Brooklyn
  Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Aled Sage


When the "producer" entity of a Propagator enricher is unmanaged, it can cause 
rebind to resolve the dangling reference value as {{null}}. In 0.9.0, that is 
treated as the producer being "self". This can cause an infinite loop, where 
the Propagator subscribes to its own sensor, and every time its sensor changes 
it resets the sensor value.

To reproduce, deploy this app:

{noformat}
location: localhost
services:
- type: org.apache.brooklyn.entity.stock.BasicApplication
  brooklyn.children:
  - type: org.apache.brooklyn.entity.machine.MachineEntity
{noformat}

And programmatically add a Propagator enricher to the top-level app, such as:

{noformat}
appId = "u4VDoIOa";
childId = "LPYT4g27";
sensor = org.apache.brooklyn.core.sensor.Sensors.newSensor(String.class, 
"host.name");
app = mgmt.getEntityManager().getEntity(appId);
child = mgmt.getEntityManager().getEntity(childId);
app.enrichers().add(org.apache.brooklyn.api.sensor.EnricherSpec.create(org.apache.brooklyn.enricher.stock.Propagator.class)
.configure("enricher.producer", child)
.configure("enricher.propagating.inclusions", 
com.google.common.collect.ImmutableList.of(sensor)));
{noformat}

Then unmanange the child entity (e.g. via the advanced tab in the web-console), 
and restart Brooklyn.

The CPU goes through the roof, and the debug log shows log messages like:

{noformat}
2016-05-31 09:02:22,599 DEBUG o.a.b.c.m.i.LocalSubscriptionManager 
[brooklyn-execmanager-gfOvVoKq-3]: 388000 events for subscriber 
Subscription[qF9YFEBA;BasicApplicationImpl{id=u4VDoIOa}@u4VDoIOa:host.name]
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BROOKLYN-284) Deadlock in entity creation, when toString() calls getConfig

[Aled Sage (JIRA)]

Aled Sage created BROOKLYN-284:
--

 Summary: Deadlock in entity creation, when toString() calls 
getConfig
 Key: BROOKLYN-284
 URL: https://issues.apache.org/jira/browse/BROOKLYN-284
 Project: Brooklyn
  Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Aled Sage


If an entity's toString calls getConfig() to retrieve a config value that is 
set on the entity, it deadlocks during the initial creation of the entity.

{noformat}
Java stack information for the threads listed above:
===
"brooklyn-execmanager-o0dJOc0a-1":
at 
org.apache.brooklyn.core.mgmt.internal.EntityManagementSupport.getExecutionContext(EntityManagementSupport.java:354)
- waiting to lock <0x0007fa6ff568> (a 
org.apache.brooklyn.core.mgmt.internal.EntityManagementSupport)
at 
org.apache.brooklyn.core.entity.AbstractEntity.getExecutionContext(AbstractEntity.java:1566)
- locked <0x0007fa6fef90> (a 
org.apache.brooklyn.core.entity.EntityTypeTest$EntityWithToStringAccessingConfig)
at 
org.apache.brooklyn.core.objs.AdjunctConfigMap.getConfig(AdjunctConfigMap.java:84)
at 
org.apache.brooklyn.core.config.internal.AbstractConfigMapImpl.getConfig(AbstractConfigMapImpl.java:50)
at 
org.apache.brooklyn.core.objs.AbstractEntityAdjunct$BasicConfigurationSupport.get(AbstractEntityAdjunct.java:291)
at 
org.apache.brooklyn.enricher.stock.Transformer.getTransformation(Transformer.java:65)
at 
org.apache.brooklyn.enricher.stock.AbstractTransformer.compute(AbstractTransformer.java:139)
at 
org.apache.brooklyn.enricher.stock.AbstractTransformer.onEvent(AbstractTransformer.java:133)
at 
org.apache.brooklyn.core.mgmt.internal.LocalSubscriptionManager$1.run(LocalSubscriptionManager.java:149)
at 
org.apache.brooklyn.util.concurrent.CallableFromRunnable.call(CallableFromRunnable.java:43)
at 
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
at 
org.apache.brooklyn.util.core.task.SingleThreadedScheduler$1.call(SingleThreadedScheduler.java:116)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"main":
at 
org.apache.brooklyn.core.entity.AbstractEntity.getExecutionContext(AbstractEntity.java:1566)
- waiting to lock <0x0007fa6fef90> (a 
org.apache.brooklyn.core.entity.EntityTypeTest$EntityWithToStringAccessingConfig)
at 
org.apache.brooklyn.core.entity.internal.EntityConfigMap.getConfig(EntityConfigMap.java:117)
at 
org.apache.brooklyn.core.config.internal.AbstractConfigMapImpl.getConfig(AbstractConfigMapImpl.java:50)
at 
org.apache.brooklyn.core.entity.AbstractEntity$BasicConfigurationSupport.get(AbstractEntity.java:1180)
at 
org.apache.brooklyn.core.entity.AbstractEntity.getConfig(AbstractEntity.java:1266)
at 
org.apache.brooklyn.core.entity.EntityTypeTest$EntityWithToStringAccessingConfig.toString(EntityTypeTest.java:300)
at 
org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:304)
at 
org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:276)
at 
org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:230)
at ch.qos.logback.classic.spi.LoggingEvent.(LoggingEvent.java:114)
at 
ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:447)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:403)
at ch.qos.logback.classic.Logger.debug(Logger.java:522)
at 
org.apache.brooklyn.core.mgmt.internal.LocalSubscriptionManager.subscribe(LocalSubscriptionManager.java:114)
- locked <0x0007f955e438> (a 
org.apache.brooklyn.core.mgmt.internal.LocalSubscriptionManager)
at 
org.apache.brooklyn.core.mgmt.internal.QueueingSubscriptionManager.startDelegatingForSubscribing(QueueingSubscriptionManager.java:81)
- locked <0x0007fa6ff618> (a 
org.apache.brooklyn.core.mgmt.internal.QueueingSubscriptionManager)
at 
org.apache.brooklyn.core.mgmt.internal.EntityManagementSupport.onManagementStarting(EntityManagementSupport.java:186)
- locked <0x0007fa6ff568> (a 
org.apache.brooklyn.core.mgmt.internal.EntityManagementSupport)
at 
org.apache.brooklyn.core.mgmt.internal.LocalEntityManager$2.apply(LocalEntityManager.java:360)
at 
org.apache.brooklyn.core.mgmt.internal.LocalEntityManager$2.apply(LocalEntityManager.java:1)
at 
org.apache.brooklyn.core.mgmt.internal.Lo