Moving conversations to discourse and closing this list.

[Tim Penhey]

Hi all,

Juju is moving with the times and moving to have a central place for
conversations, questions, and soon, documentation.

We are using Discourse [1] and it can be found here:
   https://discourse.jujucharms.com/

Discourse offers a very nice interface for conversations, and has a bot
walk you through the interface and capabilities when you create an account.

Both the juju and juju-dev mailing lists will be closed for posting
shortly. The archives will stay around, and can be found on the
lists.ubuntu.com archive section [2], [3].

We welcome everyone on these lists to join us on discourse and ask
questions, get involved in discussions, or even get involved with
development.

Cheers,
Tim

[1] https://www.discourse.org/
[2] https://lists.ubuntu.com/archives/juju/
[3] https://lists.ubuntu.com/archives/juju-dev/

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

We are pulling he 2.3.6 agents

[Tim Penhey]

Hey people,

We have field reports where a 2.3.6 upgrade interacted badly with some
charm settings causing Juju to get itself into a stuck, somewhat corrupt
state. We are still evaluating how to fix this for stuck systems.

The symptom is the 2.3.6 upgrade fails and gets stuck.

The agents are being removed from streams to stop people accidentally
upgrading to a broken version, even though this breakage doesn't impact
everyone.

We are also looking into the continuous integration tests around our
upgrades for extra testing to catch other issues like this one.

We will be releasing 2.3.7 shortly to address this isue.

Thanks for your patience.
Tim


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju's dependencies, juju/utils and a road forward

[Tim Penhey]

Which then begs the question...

Why do we have stable gopkg.in branches depending on unstable branches?

i.e. gopkg.in/juju/charmstore.v5 depending on
gopkg.in/macaroon-bakery.v2-unstable?

This seems to smell bad.


Tim

On 04/04/18 14:17, Tim Penhey wrote:
> As a follow on...
> 
> I'd like to move juju to only use "stable" dependency branches. No more
> '-unstable'.
> 
> Tim
> 
> On 04/04/18 12:52, Tim Penhey wrote:
>> Hi folks,
>>
>> Juju has a problem with its dependencies. We have been trying to upgrade
>> a number of our dependencies recently have have been hitting road
>> blocks. Mostly because of changes to common libraries that are
>> incompatible with other libraries that we use.
>>
>> I'm pretty sure that the root of all these problems is the juju/utils
>> package. It has become a dumping ground for any common functions that
>> packages may want to share. However these common functions are changed
>> arbitrarily with only thoughts around how they impact a subset of the
>> downstreams of juju/utils.
>>
>> I think that the only sane way forward is to move all the functionality
>> we care about out of juju/utils into more cohesive, smaller, packages
>> that have meaningful names and you should be able to determine what they do.
>>
>> Where the current clashes came from were a few places.
>>  1) we wanted to change our mgo dependency
>>  2) a new file path utility function was wanted
>>  3) the debugstatus package inside utils was updated in a backward
>> incompatible way
>>
>> The root problem here is that the utils package tries to do too many
>> things. Any package called "utils" should be a warning, and we should
>> have listened to Dave Cheney when he brought this up originally.
>>
>> I think the way forward is to break up the utils package. I'd like to
>> propose that we don't add anything new to the utils package, and only
>> deletions are allowed. We create new packages that contain the
>> functionality we require, and have them be much more controlled on their
>> own dependencies.
>>
>> Thoughts?
>>
>> Tim
>>
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju's dependencies, juju/utils and a road forward

[Tim Penhey]

As a follow on...

I'd like to move juju to only use "stable" dependency branches. No more
'-unstable'.

Tim

On 04/04/18 12:52, Tim Penhey wrote:
> Hi folks,
> 
> Juju has a problem with its dependencies. We have been trying to upgrade
> a number of our dependencies recently have have been hitting road
> blocks. Mostly because of changes to common libraries that are
> incompatible with other libraries that we use.
> 
> I'm pretty sure that the root of all these problems is the juju/utils
> package. It has become a dumping ground for any common functions that
> packages may want to share. However these common functions are changed
> arbitrarily with only thoughts around how they impact a subset of the
> downstreams of juju/utils.
> 
> I think that the only sane way forward is to move all the functionality
> we care about out of juju/utils into more cohesive, smaller, packages
> that have meaningful names and you should be able to determine what they do.
> 
> Where the current clashes came from were a few places.
>  1) we wanted to change our mgo dependency
>  2) a new file path utility function was wanted
>  3) the debugstatus package inside utils was updated in a backward
> incompatible way
> 
> The root problem here is that the utils package tries to do too many
> things. Any package called "utils" should be a warning, and we should
> have listened to Dave Cheney when he brought this up originally.
> 
> I think the way forward is to break up the utils package. I'd like to
> propose that we don't add anything new to the utils package, and only
> deletions are allowed. We create new packages that contain the
> functionality we require, and have them be much more controlled on their
> own dependencies.
> 
> Thoughts?
> 
> Tim
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Juju's dependencies, juju/utils and a road forward

[Tim Penhey]

Hi folks,

Juju has a problem with its dependencies. We have been trying to upgrade
a number of our dependencies recently have have been hitting road
blocks. Mostly because of changes to common libraries that are
incompatible with other libraries that we use.

I'm pretty sure that the root of all these problems is the juju/utils
package. It has become a dumping ground for any common functions that
packages may want to share. However these common functions are changed
arbitrarily with only thoughts around how they impact a subset of the
downstreams of juju/utils.

I think that the only sane way forward is to move all the functionality
we care about out of juju/utils into more cohesive, smaller, packages
that have meaningful names and you should be able to determine what they do.

Where the current clashes came from were a few places.
 1) we wanted to change our mgo dependency
 2) a new file path utility function was wanted
 3) the debugstatus package inside utils was updated in a backward
incompatible way

The root problem here is that the utils package tries to do too many
things. Any package called "utils" should be a warning, and we should
have listened to Dave Cheney when he brought this up originally.

I think the way forward is to break up the utils package. I'd like to
propose that we don't add anything new to the utils package, and only
deletions are allowed. We create new packages that contain the
functionality we require, and have them be much more controlled on their
own dependencies.

Thoughts?

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: More juju upgrade-juju failings

[Tim Penhey]

Hi Sandor,

Streamlining upgrades is definitely on the team's road-map. We are aware
of the juju-upgrader plugin, and will be looking to incorporate some of
that functionality into the core codebase.

The core team has worked on better upgrade testing as part of our CI,
which enables more test scenarios to help discover issues between more
versions.

Cheers,
Tim

On 22/03/18 05:32, Sandor Zeestraten wrote:
> Is this shared google doc open for external contributors?
> 
> After spending a while upgrading our 2.1.x environments to 2.3.x and
> hitting some bugs along the way in staging (see below), I'd agree that
> the process needs a bit of love and wouldn't mind sharing some experiences.
> 
> Rick mentioned https://launchpad.net/juju-upgrader on the Juju show a
> couple of episodes back, but I haven't seen it mentioned anywhere else
> yet. Are those tools supposed to deal with some of these issues and
> eventually be rolled into juju-core?
> 
> https://bugs.launchpad.net/juju/+bug/1746265
> https://bugs.launchpad.net/juju/+bug/1748294
> https://bugs.launchpad.net/juju/+bug/1697936
> 
> Regards
> --
> Sandor Zeestraten
> 
> On Wed, Feb 28, 2018 at 8:30 AM, Mark Shuttleworth  <mailto:m...@ubuntu.com>> wrote:
> 
> 
> I think this UX on the upgrade process has obvious problems. Nobody
> should have to guess at what to do, in which sequence.
> 
> Can I suggest that we have a shared Google doc to mock up a nice
> experience starting with the simple command 'juju upgrade' which then
> walks people through the process, including the distinction between
> upgrading charms, agents and controllers, as well as the ability to do
>     aerospace-grade upgrades through live migration to newer controllers?
> 
> Mark
> 
> On 02/27/2018 11:26 PM, Tim Penhey wrote:
> > Hi Daniel,
> >
> > The issue here is that you are upgrading the default model, not the
> > controller model itself.
> >
> > I think we could make the error messaging more clear, and also
> have the
> > command also check the controller version before showing a lot of
> > baffling output.
> >
> > What you need to do is upgrade the controller model first, so either
> > switch to it or run:
> >
> >   juju upgrade-juju -m controller --agent-version 2.3.3
> >
> > Cheers,
> > Tim
> >
> > On 28/02/18 11:19, Daniel Bidwell wrote:
> >> I am running on juju 2.3.3-xenial-amd64 and my controllers are
> running
> >> in VMware Vsphere with version 2.3.2.  I thought that it would be a
> >> good thing for me to upgrade the controllers.
> >>
> >> I have a controller, "juju switch" generates:
> >> bannercontroller:admin/default
> >>
> >> and juju status generates:
> >> ModelControllerCloud/Region  Version  SLA
> >> default  bannercontroller  myvscloud/New
> Datacenter  2.3.2unsupported
> >>
> >> App  Version  Status  Scale  Charm  Store  Rev  OS  Notes
> >>
> >> Unit  Workload  Agent  Machine  Public address  Ports  Message
> >>
> >> Machine  State  DNS  Inst id  Series  AZ  Message
> >>
> >> doing "juju upgrade-juju --agent-version 2.3.3 --debug" generates:
> >>
> >> 17:16:19 INFO  juju.cmd supercommand.go:56 running juju [2.3.3 gc
> go1.9.2]
> >> 17:16:19 DEBUG juju.cmd supercommand.go:57   args:
> []string{"/snap/juju/3452/bin/juju", "upgrade-juju",
> "--agent-version", "2.3.3", "--debug"}
> >> 17:16:19 INFO  juju.juju api.go:67 connecting to API addresses:
> [10.1.61.239:17070 <http://10.1.61.239:17070>]
> >> 17:16:19 DEBUG juju.api apiclient.go:843 successfully dialed
> "wss://10.1.61.239:17070/model/5a057215-e835-42fb-8c5a-f9d57eded74c/api
> <http://10.1.61.239:17070/model/5a057215-e835-42fb-8c5a-f9d57eded74c/api>"
> >> 17:16:19 INFO  juju.api apiclient.go:597 connection established
> to
> "wss://10.1.61.239:17070/model/5a057215-e835-42fb-8c5a-f9d57eded74c/api
> <http://10.1.61.239:17070/model/5a057215-e835-42fb-8c5a-f9d57eded74c/api>"
> >> 17:16:19 INFO  juju.juju api.go:67 connecting to API addresses:
> [10.1.61.239:17070 <http://10.1.61.239:17070>]
> >> 17:16:19 DEBUG juju.api apiclient.go:843 successfully dialed
> "wss://10.1.61.239:17070/model/5a05

Re: More juju upgrade-juju failings

[Tim Penhey]

Hi Daniel,

The issue here is that you are upgrading the default model, not the
controller model itself.

I think we could make the error messaging more clear, and also have the
command also check the controller version before showing a lot of
baffling output.

What you need to do is upgrade the controller model first, so either
switch to it or run:

  juju upgrade-juju -m controller --agent-version 2.3.3

Cheers,
Tim

On 28/02/18 11:19, Daniel Bidwell wrote:
> I am running on juju 2.3.3-xenial-amd64 and my controllers are running
> in VMware Vsphere with version 2.3.2.  I thought that it would be a
> good thing for me to upgrade the controllers.
> 
> I have a controller, "juju switch" generates:
> bannercontroller:admin/default
> 
> and juju status generates:
> ModelControllerCloud/Region  Version  SLA
> default  bannercontroller  myvscloud/New Datacenter  2.3.2unsupported
> 
> App  Version  Status  Scale  Charm  Store  Rev  OS  Notes
> 
> Unit  Workload  Agent  Machine  Public address  Ports  Message
> 
> Machine  State  DNS  Inst id  Series  AZ  Message
> 
> doing "juju upgrade-juju --agent-version 2.3.3 --debug" generates:
> 
> 17:16:19 INFO  juju.cmd supercommand.go:56 running juju [2.3.3 gc go1.9.2]
> 17:16:19 DEBUG juju.cmd supercommand.go:57   args: 
> []string{"/snap/juju/3452/bin/juju", "upgrade-juju", "--agent-version", 
> "2.3.3", "--debug"}
> 17:16:19 INFO  juju.juju api.go:67 connecting to API addresses: 
> [10.1.61.239:17070]
> 17:16:19 DEBUG juju.api apiclient.go:843 successfully dialed 
> "wss://10.1.61.239:17070/model/5a057215-e835-42fb-8c5a-f9d57eded74c/api"
> 17:16:19 INFO  juju.api apiclient.go:597 connection established to 
> "wss://10.1.61.239:17070/model/5a057215-e835-42fb-8c5a-f9d57eded74c/api"
> 17:16:19 INFO  juju.juju api.go:67 connecting to API addresses: 
> [10.1.61.239:17070]
> 17:16:19 DEBUG juju.api apiclient.go:843 successfully dialed 
> "wss://10.1.61.239:17070/model/5a057215-e835-42fb-8c5a-f9d57eded74c/api"
> 17:16:19 INFO  juju.api apiclient.go:597 connection established to 
> "wss://10.1.61.239:17070/model/5a057215-e835-42fb-8c5a-f9d57eded74c/api"
> 17:16:19 INFO  juju.juju api.go:67 connecting to API addresses: 
> [10.1.61.239:17070]
> 17:16:19 DEBUG juju.api apiclient.go:843 successfully dialed 
> "wss://10.1.61.239:17070/api"
> 17:16:19 INFO  juju.api apiclient.go:597 connection established to 
> "wss://10.1.61.239:17070/api"
> 17:16:19 DEBUG juju.cmd.juju.commands upgradejuju.go:466 searching for agent 
> binaries with major: 2
> 17:16:22 INFO  cmd upgradejuju.go:363 available agent binaries:
> 2.3.3-artful-amd64
> 2.3.3-artful-arm64
> 2.3.3-artful-ppc64el
> 2.3.3-artful-s390x
> 2.3.3-bionic-amd64
> 2.3.3-bionic-arm64
> 2.3.3-bionic-ppc64el
> 2.3.3-bionic-s390x
> 2.3.3-centos7-amd64
> 2.3.3-trusty-amd64
> 2.3.3-trusty-arm64
> 2.3.3-trusty-ppc64el
> 2.3.3-trusty-s390x
> 2.3.3-win10-amd64
> 2.3.3-win2012-amd64
> 2.3.3-win2012hv-amd64
> 2.3.3-win2012hvr2-amd64
> 2.3.3-win2012r2-amd64
> 2.3.3-win2016-amd64
> 2.3.3-win2016nano-amd64
> 2.3.3-win7-amd64
> 2.3.3-win8-amd64
> 2.3.3-win81-amd64
> 2.3.3-xenial-amd64
> 2.3.3-xenial-arm64
> 2.3.3-xenial-ppc64el
> 2.3.3-xenial-s390x
> best version:
> 2.3.3
> 17:16:22 DEBUG juju.api monitor.go:35 RPC connection died
> 17:16:22 DEBUG juju.api monitor.go:35 RPC connection died
> 17:16:22 DEBUG juju.api monitor.go:35 RPC connection died
> ERROR a hosted model cannot have a higher version than the server model: 
> 2.3.3 > 2.3.2
> 17:16:22 DEBUG cmd supercommand.go:459 error stack: 
> a hosted model cannot have a higher version than the server model: 2.3.3 > 
> 2.3.2
> github.com/juju/juju/rpc/client.go:149: 
> github.com/juju/juju/api/apiclient.go:924: 
> 
> 
> I am a little baffled at what the problem might be.  This controller has no 
> vm associated with it.
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

2.3.3 delayed slightly

[Tim Penhey]

Hi folks,

Just to let you know that the 2.3.3 release that was planned for today
is going to be delayed a day or two while we fix some upgrade issues
that were caught late.

At least they were caught before the release, so that is a good thing.

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Beware 2.2.7 release

[Tim Penhey]

Hi folks,

Last night we were alerted to a fundamental issue with the 2.2.7 agent
only release.

There was a weird race condition that was only observable on larger
models which would cause a model to be non-responsive to controller
generated events, like config updates, actions, juju run etc.

We have a fix available, and it has been tested on one of the broken
controllers and shown to fix the problem.

Expect a 2.2.8 agent only release shortly to address this issue.

Thanks
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju 2.3-rc2 is here!

[Tim Penhey]

Hi Merlijn,

You should be able to go:
  juju upgrade-juju -m controller

That should do the trick. If the client version is different you may
want to specify the agent version:

  juju upgrade-juju -m controller --agent-version 2.3-rc2

Tim

On 07/12/17 06:26, Merlijn Sebrechts wrote:
> Is there a way to upgrade a controller from rc1.1 to rc2?
> 
> 2017-11-29 1:03 GMT+01:00 Nicholas Skaggs  >:
> 
> A new release of Juju is here, 2.3-rc2. This is primarily a bug fix
> release which addresses issues carried over from rc1.
> 
> 
> ## New and Improved
> 
> 
> * juju list-models is significantly faster, especially on
> controllers with large numbers of models
> 
> 
> * juju no longer records the execution of the update-status hook.
> This means that you will no longer see 'idle' units move to
> 'executing' showing 'running update-status hook'. This also means
> that it isn't recorded in the show-status-log for the unit. As a
> related item, the squashing of the status-log, which was buggy, has
> been removed.
> 
> 
> ## Fixes
> 
> 
> For a list of all bugs fixed in this release, see
> 
> 
> https://launchpad.net/juju/+milestone/2.3-rc2
> 
> 
> 
> 
> ## How can I get it?
> 
> 
> The best way to get your hands on this release of Juju is to install
> it as a snap package (see https://snapcraft.io/for more info on snaps).
> 
> 
> snap install juju --classic --candidate
> 
> 
> Other packages are available for a variety of platforms. Please see
> the online documentation at
> https://jujucharms.com/docs/stable/reference-install
> . Those
> subscribed to a snap channel should be automatically upgraded. If
> you’re using the ppa/homebrew, you should see an upgrade available.
> 
> 
> ## Feedback Appreciated!
> 
> 
> We encourage everyone to let us know how you're using Juju. Send us a
> 
> message on Twitter using #jujucharms, join us at #juju on freenode, and
> 
> subscribe to the mailing list at j...@lists.ubuntu.com
> .
> 
> 
> ## More information
> 
> To learn more about juju please visit https://jujucharms.com.
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
> 
> 
> 
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju 2.3 beta2 is here!

[Tim Penhey]

On 10/11/17 12:12, Dmitrii Shcherbakov wrote:
> It's situations like the following that I am trying to avoid:
> 
>   rabbitmq-server:
>     charm: cs:xenial/rabbitmq-server
>     bindings:
>       "": *oam-space
>       amqp: *internal-space
>       cluster: *internal-space
>     options:
>       source: *openstack-origin
>       min-cluster-size: 3
>       cluster-partition-handling: pause_minority
>     num_units: 3
>     to:
>     - lxd:0
>     - lxd:1
>     - lxd:2
> 
> serialized by hand to: juju deploy rabbitmq-server --config
> ../rabbitmq.yaml -n 3 --to lxd:0,lxd:1,lxd:12 --bind "space-oam
> amqp=space-oam cluster=space-oam"
> 
> cat ../rabbitmq.yaml 
> rabbitmq-server:
>   source: cloud:xenial-ocata
>   min-cluster-size: 3
>   cluster-partition-handling: pause_minority
> 
> Which includes brain-parsing the definition, serializing it to
> .yaml + -n  --to  --bind 

I don't understand what it is you are trying to avoid here?

What is it you are trying to do?

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju 2.3 beta2 is here!

[Tim Penhey]

No we aren't going to reuse --to.

The --to directive is just for placement directives. Trying to use that
to overload machine mappings for bundles adds complexity for no real value.

We will use --map-machines. I'm a big proponent of explicit is better
than implicit.

While I'm not 100% fixed on using the term "existing", I don't think
either '...' nor ':' make a lot of sense. They are not used elsewhere in
juju anywhere, and aren't really intuitive.

We also aren't looking to map machine placement directives to other
bundle placement directive types.

Tim

On 10/11/17 10:39, Dmitrii Shcherbakov wrote:
> I think it's nice to reuse --to because we don't use it with bundles on
> juju deploy. A unified --map[-machines] would also be fine to me.
> 
>  := ( |
> )=( | )
> --to ([,]+[,:] ) | :
> 
> Remap two, otherwise use existing:
> 
> --to 1=2,3=5,:
> 
> The same with app names, but have to error out on lxd:1 or kvm:5 in a
> bundle if exapp/2 or exapp/5 are themselves in lxd or kvm "containers"
> (surely nested containers or VMs are possible to do by hand but insane
> in this case):
> 
> --to 1=exapp/2,3=exapp/5,:
> 
> This is kind of complex as there *may* be a conflicting placement
> directive in a bundle for exappplugin/2 or it may be a subordinate (in
> which case we ignore the mapping altogether)
> 
> --to exappplugin/2=exapp/2,3=exappother/5,:
> 
> FWIW it may be a bundle without machine definitions and placement
> directives, so, still a valid case.
> 
> Remap two, otherwise allocate new:
> 
> --to 1=2,3=5
> 
> Use existing:
> 
> --to :
> 
> Allocate new:
> 
> 
> Another idea, albeit more complex, is to introduce ranges to that:
> 
> 3:5=41:43
> 
> 3:5=41:43,:
> 
> 3:5=neutron-gateway/0:neutron-gateway/2
> 
> Also have to be careful about the following (I haven't seen the feature
> code yet):
> 
> 1. odd charm or app names during machine spec parsing:
> https://launchpad.net/charm-6wind-virtual-accelerator
> <https://launchpad.net/charm-6wind-virtual-accelerator>
> 
> 2. endpoint -> network space bindings and storage bindings for existing
> apps in a given model that may be present in a newly deployed bundle.
> 
> Really appreciate the major changes coming to Juju 2.3 - thanks a lot!
> 
> 
> Best Regards,
> Dmitrii Shcherbakov
> 
> Field Software Engineer
> IRC (freenode): Dmitrii-Sh
> 
> On Thu, Nov 9, 2017 at 1:20 PM, roger peppe  <mailto:roger.pe...@canonical.com>> wrote:
> 
> I still like the idea of overloading the --to flag rather than having
> a new --map-machines flag. It's concise and fits well, I think - we
> want the machines in this bundle to mapped *to* the machines we're
> specifying here.
> 
> I like the thrust of Tim's suggestion for "existing" but I'm not
> entirely sure about that word - it's quite long and it doesn't quite
>     express the identity relationship that I see there. How about "same"?
> 
> For example:
> 
>      juju deploy --to 1=2,2=3,same some-bundle
> 
> Another possibility: use ellipses to imply the rest of the mapping:
> 
>     juju deploy --to 1=2,2=3,... some-bundle
>     juju deploy --to ... some-bundle
> 
> 
> 
> On 9 November 2017 at 02:43, Tim Penhey  <mailto:tim.pen...@canonical.com>> wrote:
> >
> >
> > On 09/11/17 13:06, Mark Shuttleworth wrote:
> >> On 11/07/2017 03:11 PM, John Meinel wrote:
> >>> ...
> >>>
> >>>
> >>>     > Perhaps just:
> >>>     >
> >>>     >   juju deploy --map-machines A=B,C=D
> >>>     >
> >>>     > ... or some variant of that?
> >>>     >
> >>>     > Let's use the betas to refine and condense and clarify.
> >>>
> >>>     +1 to that. I'm wondering if use-existing-machines is ever
> appropriate
> >>>     on its own, as the machine numbers in a model are ephemeral but
> >>>     machine numbers in a bundle are static.
> >>>
> >>>
> >>> Feedback from Admins that one of their big use case really is for
> >>> bundle-a to lay down a definition/base charm across everything, and
> >>> bundle-b to be meant as an exact overlay, and all of the machine-ids
> >>> are exact matches. And having to specify 0=0,...50=50 is a lot
> of ugly
> >>> boilerplate.
> 

Re: Juju 2.3 beta2 is here!

[Tim Penhey]

On 09/11/17 13:06, Mark Shuttleworth wrote:
> On 11/07/2017 03:11 PM, John Meinel wrote:
>> ...
>>  
>>
>> > Perhaps just:
>> >
>> >   juju deploy --map-machines A=B,C=D
>> >
>> > ... or some variant of that?
>> >
>> > Let's use the betas to refine and condense and clarify.
>>
>> +1 to that. I'm wondering if use-existing-machines is ever appropriate
>> on its own, as the machine numbers in a model are ephemeral but
>> machine numbers in a bundle are static.
>>
>>
>> Feedback from Admins that one of their big use case really is for
>> bundle-a to lay down a definition/base charm across everything, and
>> bundle-b to be meant as an exact overlay, and all of the machine-ids
>> are exact matches. And having to specify 0=0,...50=50 is a lot of ugly
>> boilerplate.
> 
> I would expect that --map-machines means that machine numbers correspond
> UNLESS remapped. So your ugly boilerplate is not needed.

Been thinking more... how about this as a proposal:

I think we can combine both the --use-existing-machines and the
--bundle-machine into the single --map-machines:

So...

To use the existing machines as is:
  --map-machines existing

To only map two machines,
  --map-machines 1=2,2=3

To use existing, and map two machines
  --map-machines existing,1=2,2=3

Thoughts?

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju 2.3 beta2 is here!

[Tim Penhey]

On 09/11/17 13:06, Mark Shuttleworth wrote:
> On 11/07/2017 03:11 PM, John Meinel wrote:
>> ...
>>  
>>
>> > Perhaps just:
>> >
>> >   juju deploy --map-machines A=B,C=D
>> >
>> > ... or some variant of that?
>> >
>> > Let's use the betas to refine and condense and clarify.
>>
>> +1 to that. I'm wondering if use-existing-machines is ever appropriate
>> on its own, as the machine numbers in a model are ephemeral but
>> machine numbers in a bundle are static.
>>
>>
>> Feedback from Admins that one of their big use case really is for
>> bundle-a to lay down a definition/base charm across everything, and
>> bundle-b to be meant as an exact overlay, and all of the machine-ids
>> are exact matches. And having to specify 0=0,...50=50 is a lot of ugly
>> boilerplate.
> 
> I would expect that --map-machines means that machine numbers correspond
> UNLESS remapped. So your ugly boilerplate is not needed.

Part of the problem with this is that this isn't how any of the flags
work just now.

All flags are either bool flags, which take no args, or other types that
*always* take args.

Now this isn't to say that we couldn't force something, but it isn't as
intuitive.

We could make
  --map-machines
work by itself, and we could make
  --map-machines=1=2,3=4
work, but the underlying flag parsing library wouldn't like:
  --map-machines 1=2,3=4

Because if you are a "bool" type flag, you can work without any args, or
work with an explicit arg, the second case, but it doesn't work with the
third.

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Juju 2.3 release train starting

[Tim Penhey]

Hi folks,

We are gearing up to release Juju 2.3. Juju 2.3 brings two headline
features:
 * Cross model relations - the ability to relate applications in
different models
 * Persistent storage - storage can outlive the unit or model it was
created for.

The team will be releasing 2.3-beta1 tomorrow, and we will be releasing
a beta every week until we hit the release candidate.

We do know that there will be at least a beta2 as we are still finishing
off a few features.

Good things are coming.

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: We are excited to announce the release of Juju 2.2.3!

[Tim Penhey]

We are still getting the docs updated.

https://jujucharms.com/docs/2.2/charms-bundles#setting-charm-configurations-options-in-a-bundle
has a note about the include-file:// and include-base64:// items.

We still seem to be missing details about --bundle-config.

Tim

On 08/09/17 22:41, Sandor Zeestraten wrote:
> Hey Burton, 
> 
> Where can I find the docs on these new bundle options (include-file and
> include-base64) and the --bundle-config option?
> 
> Have a nice weekend
> --
> Sandor Zeestraten
> 
> On Fri, Sep 8, 2017 at 9:43 AM, Burton Swan  > wrote:
> 
> ## New and Improved
> 
> * The remove-machine command has a --keep-instance flag which allows
> the cloud instance to be left running when the machine is removed
> from the Juju model
> 
> * Bundles can now reference local resources by specifying a relative
> path (as can already be done for local charms).
> 
> * Values in local bundles for options and annotations can now
> specify a file to be read for the specified value. This is to
> support charm options where the value is some structured content,
> such as a configuration file. For binary external files, such as
> binary certificates, there is an option to base64 encode the
> contents of the file so it can be used as a string value. The
> referenced file can include the path to the file. The file location
> is relative to the bundle file location. e.g.:
> 
> applications:
> my-app:
> charm: some-charm
> options:
> config: include-file://my-config.yaml
> cert: include-base64://my-cert.crt
> 
> * There is a new option for deploying bundles: --bundle-config. This
> configuration file needs to be a YAML file, and currently only
> supports applications as a top level key. The format of the
> applications is the same as applications section in the bundle. Any
> values specified for an application in the bundle-config file
> override those values defined in the bundle, with the exception of
> the map type values, where the maps are merged with preference given
> to the bundle-config. The purpose of this to allow the use of a
> common bundle definition, and have model specific configuration kept
> in a separate file. Option and annotation values specified in the
> bundle-config file can also use the include-file:// and
> include-base64:// directives mentioned above for local bundles.
> Paths specified are relative to the bundle-config file.
> 
> 
> For a list of all bugs fixed in this release, see
> https://launchpad.net/juju/+milestone/2.2.3
> 
> 
> ## How can I get it?
> 
> The best way to get your hands on this release of Juju is to install
> it as a snap package (see https://snapcraft.io for more info on snaps).
> 
>  snap install juju --classic 
> 
> Other packages are available for a variety of platforms. Please see
> the online documentation at
> https://jujucharms.com/docs/stable/reference-install
> 
> 
> Those subscribed to a snap channel should be automatically upgraded.
> If you’re using the ppa/homebrew, you should see an upgrade available.
> 
> For highlights of this release, please see the documentation at
> https://jujucharms.com/docs/2.2/whats-new
> 
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
> 
> 
> 
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: developer commands

[Tim Penhey]

I don't think the YAML format for status should be the serialized model
description as it holds more info, and not all relevant for status.

dump-model was originally used to test export code.

dump-db was used to get access to the contents of the db without going
through mongo shell.

Both very dev focused, and both only usable by admins.

Tim

On 29/08/17 13:11, Andrew Wilkins wrote:
> On Mon, Aug 28, 2017 at 5:17 PM Tim Penhey  <mailto:tim.pen...@canonical.com>> wrote:
> 
> FYI, the developer commands were originally designed like the controller
> commands.
> 
> You don't say "juju destroy-model -m foo".
> 
> 
> I think that holds for "dump-model", but I'm not so sure about
> "dump-db". It feels natural enough to me to say "juju dump-db -m foo",
> or "juju dump-db".
> 
> I'm wondering about the future of "dump-model". Since you've been
> looking at using the model description to feed "juju status", I wonder
> if it wouldn't make sense to update the "juju status" YAML format to be
> exactly the model description. Then "dump-model" becomes redundant. We'd
> have to make it an option, though, because backwards compatibility.
> 
> Cheers,
> Andrew
>  
> 
> Tim
> 
> On 28/08/17 19:48, Anastasia Macmood wrote:
> > Hi
> >
> > Just a quick note for developers that use developer commands.
> >
> > 'juju dump-model' and 'juju dump-db' are changing on develop tip [1],
> > from 2.3.x.
> > These commands are now fully-fledged model commands as they were
> > originally designed.
> > This means that they will now accept model name as an option
> instead of
> > as a positional argument.
> >
> > i.e.
> > 'juju dump-model -m modelName' NOT 'juju dump-model modelName'
> > 'juju dump-db -m modelName' NOT 'juju dump-db modelName'
> >
> > Happy juju-ing!
> >
> > Sincerely Yours,
> > Anastasia
> >
> > [1]
> > https://github.com/juju/juju/pull/7797
> >
> >
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com <mailto:Juju-dev@lists.ubuntu.com>
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: developer commands

[Tim Penhey]

FYI, the developer commands were originally designed like the controller
commands.

You don't say "juju destroy-model -m foo".

Tim

On 28/08/17 19:48, Anastasia Macmood wrote:
> Hi
> 
> Just a quick note for developers that use developer commands.
> 
> 'juju dump-model' and 'juju dump-db' are changing on develop tip [1],
> from 2.3.x.
> These commands are now fully-fledged model commands as they were
> originally designed.
> This means that they will now accept model name as an option instead of
> as a positional argument.
> 
> i.e.
> 'juju dump-model -m modelName' NOT 'juju dump-model modelName'
> 'juju dump-db -m modelName' NOT 'juju dump-db modelName'
> 
> Happy juju-ing!
> 
> Sincerely Yours,
> Anastasia
> 
> [1]
> https://github.com/juju/juju/pull/7797
> 
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: developer commands

[Tim Penhey]

binMOC2fYDKR9.bin
Description: PGP/MIME version identification


encrypted.asc
Description: OpenPGP encrypted message
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Organising apiserver facades

[Tim Penhey]

I agree with John. Having the agent facades separate from the client
facades would also be good.

Tim

On 06/07/17 23:09, John Meinel wrote:
> I'd really like to see us split apart the facades-by-purpose. So we'd
> collect the facades for Agents separately from facades for Users (and
> possibly also facades for Controller).
> I'm not sure if moving things just into 'facades' just moves the problem
> around and leaves us with just a *different* directory that is a bit
> cluttered.  But I'm +1 on things that would help organize the layout.
> 
> John
> =:->
> 
> On Thu, Jul 6, 2017 at 1:55 PM, Andrew Wilkins
> mailto:andrew.wilk...@canonical.com>> wrote:
> 
> The juju/apiserver package currently has a whole lot of facade
> packages within it, as well as some other packages related to
> authentication, logging, and other bits and bobs. I find it
> difficult to navigate and tell what's what a lot of the time.
> 
> I'd like to move the apiserver facade packages into a common
> "facades" sub-directory:
>   apiserver/facades/application
>   apiserver/facades/client
>   apiserver/facades/controller
>   etc.
> 
> Any objections? Or alternative suggestions?
> 
> Cheers,
> Andrew
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
> 
> 
> 
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: We are excited to announce the release of Juju 2.2.1!

[Tim Penhey]

Yes indeed.

Congratulations to the whole team.

Tim

On 23/06/17 23:35, Ante Karamatić wrote:
> Congrats!
> 
> Excellent release and some of our high profile users are already on
> 2.2.1 as of  now. The fact that one can just upgrade big
> environments like that, while drinking coffee, speaks volumes!

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Release schedule

[Tim Penhey]

Hi folks,

2.2.1 should be released hopefully by this time tomorrow. It includes a
number of key fixes that didn't quite make the 2.2.0 release but were
sufficiently important that we wanted to get another release out quick
smart.

>From now we will enter what we hope to be a four week cadence for point
releases on 2.2 until 2.3.0 is released.

We are planning on a 2.3-rc1 on or about the first of September.

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Maximum AllWatcher frame size?

[Tim Penhey]

It is configured in juju


// Use a 64k frame size for the websockets while we need to deal
// with x/net/websocket connections that don't deal with recieving
// fragmented messages.
const websocketFrameSize = 65536


On 14/06/17 12:00, Cory Johns wrote:
> Do we know what size the gorilla/websocket library uses for automatic
> chunking?
> 
> On Wed, Jun 14, 2017 at 11:35 AM, John Meinel  > wrote:
> 
> In 2.1 we did not chunk, in 2.2 we switch to gorilla/websocket which
> does support chunking into frames. I don't think we do any internal
> "well that would be too much information so we wont send it all".
> 
> John
> =:->
> 
> 
> On Wed, Jun 14, 2017 at 7:11 PM, Cory Johns
> mailto:cory.jo...@canonical.com>> wrote:
> 
> https://github.com/juju/python-libjuju/issues/136
>  raises the
> issue that, for large models, the initial AllWatcher response
> frame can be large enough that it overruns the default maximum
> frame size of the websocket library (1MB).  We can increase this
> limit fairly easily, but I wanted to find out whether there is
> any maximum size from the Juju size and if large enough frames
> could get chunked into multiple frames?
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
> 
> 
> 
> 
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

develop has been branched in preparation of 2.2-rc1 release

[Tim Penhey]

Hi all,

develop is now the future 2.3 branch. A patch will be landing soon to
bump the version.

There is now a 2.2 branch. This branch will be used for the 2.2 release
candidate. Landings onto this branch are currently restricted. The
restriction will be lifted when 2.2.0 is released.

Thanks,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Weekly Development Summary - and Juju 2.2-rc1 date

[Tim Penhey]

On 02/06/17 18:07, Anastasia Macmood wrote:
> On 02/06/17 15:11, Tim Penhey wrote:
...
>> Once we have confirmation from Solutions QA and JAAS that they are happy
>> with the release candidate, we will release is as 2.0.0
> 2.2.0 maybe?

Yes. That is what I meant.

Thanks,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Weekly Development Summary - and Juju 2.2-rc1 date

[Tim Penhey]

Hi all,

Sorry for the weird encrypted one.

The date for the release candidate is Tuesday the 6th of June. We really
wanted to get it out this week, but we hit two issues that we just had
to get done before the release candidate. We decided that a few extra
days for a better quality release was worth it.

On Tuesday we will be branching develop into a 2.2 branch. Also, we are
enacting a slightly different policy around landing changes into the 2.2
branch during the RC period. No extra bug fixes will be landed in 2.2
once the release candidate has been cut unless it is a critical
regression, in which case another candidate will be released. The
intention here is that the release candidate will be a true candidate,
and not just be another week to get fixes in before the final release.
Once we have confirmation from Solutions QA and JAAS that they are happy
with the release candidate, we will release is as 2.0.0 final. Once this
happens the landing restrictions will be lifted on the 2.2 branch.

We already have a number of bugs that we want to get fixed for 2.2.1.
Bugs that we have decided need to be fixed and put into the user's hands
before the 2.3 release, which won't be for a number of months.

The notable fixes of the last week:
 * a number of race test fixes, Go 1.8 found more than 1.6
 * long awaited log compression during log rotation
 * additional configuration around max txn log size so large controllers
can better handle large workloads happening very quickly
 * issues with migrating models that use local charms

There are just a couple more changes that we hope to land over the next
day, that will improve performance on larger models.

Also, the 2.2-rc1 milestone on Launchpad got significantly cleaned up.
There were bugs that were marked as high priority that had been bounced
from milestone to milestone with no one addressing them. These have been
removed from milestones as they just weren't getting addressed that way.

A lot of work has gone into the CI infrastructure this week to make it
more robust and to produce less noise on the test runs. We are looking
much better across all the CI tests. There are a few intermittent tests
still bugging us, and a few CI tests that needed updating due to changes
that landed over the week. On the whole though, we are very happy with
the stability and robustness of this upcoming release.

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Weekly Development Summary - and Juju 2.2-rc1 date

[Tim Penhey]

bintUcW_UQOrQ.bin
Description: PGP/MIME version identification


encrypted.asc
Description: OpenPGP encrypted message
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: /etc/juju/certs.d status?

[Tim Penhey]

Pretty sure it can be removed and cleaned up now.

Thanks Roger

On 25/05/17 02:22, roger peppe wrote:

I recently came across the code introduced by
https://github.com/juju/juju/pull/2294 which provides support for
reading extra certificates from /etc/juju/certs.d when connecting to
an API server. The PR description says:

"This feature is likely to be used when there is an agreed central JES
system that users can log in to without having to supply their own CA
certificate"

However we have gone in a different direction for that - controllers
can now use public certificates so no extra certificates are needed.

I can't see any references in the code to CreateCertPool or
paths.CertDir, so it seems like it's redundant. Can anyone think of a
reason this code can't be removed now?

   cheers,
 rog.



--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

2.2-rc1 progress

[Tim Penhey]

Hi folks,

The development team is fully focused on fixing the remaining issues for 
the upcoming release candidate.


The key things that still need to be done before the release candidate are:
  * fixing migration of models using local charms and local resources
  * adding robustness around dealing with massive change in a short 
period, like deleting 140 models at once on a controller

  * fixing the extra race conditions that the Go 1.8 compiler found
  * fixing a race in resource-get which is breaking solutions-qa

Along with that, there have been other stakeholder fixes going in that 
are really needed.


We were hoping to get the RC out at the end of this week, but the 
pending critical issues are likely to push this out until next week.


Just letting you all know,

Cheers,
Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Scale testing analysis

[Tim Penhey]

Hi folks,

We had another scale test today to analyse why the controller CPU usage 
didn't fall away as expected when the models were removed.


I'll be filing a bunch of bugs from the analysis process, but there is 
one bug that is, I believe, the culprit for the high CPU usage.


Interestingly enough, Juju developers were not able to reproduce the 
problem with smaller deployments. The scale that we were testing was 140 
models each with 10 machines and about 20 total units.


During the teardown process of the testing, all models were destroyed at 
once.


We have most of the responsive nature of Juju is driven off the 
watchers. These watchers watch the mongo oplog for document changes. 
What happened was that there were so many mongo operations, the capped 
collection of the oplog was completely replaced between our polled 
watcher delays. The watchers then errored out in a new unexpected way.


Effectively the watcher infrastructure needs an internal reset button 
that it can hit when this happens that invalidates all the watchers. 
This should cause all the workers to be torn down and restarted from a 
known good state.


There was a model that got stuck being destroyed, this is tracked back 
to a worker that should be doing the destructions not noticing.


All the CPU usage can be tracked back to the 139 models in the apiserver 
state pools each still running leadership and base watcher workers. The 
state pool should have removed all these instances, but it didn't notice 
they were gone.


There are some other bugs around logging things as errors that really 
aren't errors that contributed to log noise, but the fundamental error 
here is not being robust in the face of too much change at once.


This needs to be fixed for the 2.2 release candidate, so it may well 
push that out past the end of this week.


Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Tim Penhey]

On 20/05/17 19:48, Merlijn Sebrechts wrote:
On May 20, 2017 09:05, "John Meinel" > wrote:


I would actually prefer if it shows up in 'juju status' but that we
suppress it from 'juju status-log' by default.


This is still very strange behavior. Why should this be default? Just 
pipe the output of juju status through grep and exclude update-status if 
that is really what you want.


However, I would even argue that this isn't what you want in most 
use-cases.  "update-status" isn't seen as a special hook in 
charms.reactive. Anything can happen in that hook if the conditions are 
right. Ignoring update-status will have unforeseen consequences...


Hmm... there are (at least) two problems here.

Firstly, update-status *should* be a special case hook, and it shouldn't 
take long.


The purpose of the update-status hook was to provide a regular beat for 
the charm to report on the workload status. Really it shouldn't be doing 
other things.


The fact that it is a periodic execution rather than being executed in 
response to model changes is the reason it isn't fitting so well into 
the regular status and status history updates.


The changes to the workload status would still be shown in the history 
of the workload status, and the workload status is shown in the status 
output.


One way to limit the execution of the update-status hook call would be 
to put a hard timeout on it enforced by the agent.


Thoughts?

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Tim Penhey]

On 19/05/17 19:21, roger peppe wrote:

On 19 May 2017 at 03:13, Tim Penhey  wrote:

Hi folks,

Currently juju will update the status of any hook execution for any unit to
show that it is busy doing things. This was all well and good until we do
things based on time.

Every five minutes (or so) each unit will have the update-status hook
executed to allow the unit to set or update the workload status based on
what is currently going on with that unit.

Since all hook executions are stored, this means that the show-status-log
will show the unit jumping from executing update-status to ready and back
every five minutes.

The proposal is to special case the update-status hook and show in status
(or the status-log) that the hook is being executed. debug-log will continue
to show the hook executing if you are looking.


Presumably you mean *not* show in status here?


Yes. Sorry.

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

PROPOSAL: stop recording 'executing update-status hook'

[Tim Penhey]

Hi folks,

Currently juju will update the status of any hook execution for any unit 
to show that it is busy doing things. This was all well and good until 
we do things based on time.


Every five minutes (or so) each unit will have the update-status hook 
executed to allow the unit to set or update the workload status based on 
what is currently going on with that unit.


Since all hook executions are stored, this means that the 
show-status-log will show the unit jumping from executing update-status 
to ready and back every five minutes.


The proposal is to special case the update-status hook and show in 
status (or the status-log) that the hook is being executed. debug-log 
will continue to show the hook executing if you are looking.


This will reduce noise in the status-log, simplify some of our code 
around dealing with status-log, and reduce load on controllers looking 
after hundreds or thousands of units.


Is anyone opposed to this change?

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Weekly Development Summary

[Tim Penhey]

Hi folks,

We are trying a new initiative to provide a simple way for casual 
observers of the mailing list to be able to get a quick view into the 
work being done in Juju.


There are several key areas of development that are ongoing within the 
Juju team.


Polish for the upcoming 2.2 release. Working out kinks in the openstack 
and oracle providers to make them easier to use, continuing improvements 
around memory and disk usage on the controller machines. 2.2-beta4 
should be coming out very shortly to get the latest fixes into peoples 
hands through the snap beta channel. Instructions will be in the release 
notes.


Continued work on the Cross Model Relations (CMR) commands and 
functionality. This is to allow applications to relate to other 
applications running in different models. At this stage work is focusing 
on making this work within a single controller, but done in such a way 
that expanding this to work with JAAS will be a progressive step. The 
CMR work is currently behind the feature flag "cross-model".


A key new feature is support for OpenSUSE 
(https://github.com/juju/juju/pull/7257) using series "opensuseleap".


Quick links:
  Work Pending: https://github.com/juju/juju/pulls
  Recent commits: https://github.com/juju/juju/commits/develop

Until next time...

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Constraints for Additional Units

[Tim Penhey]

On 17/03/17 14:23, James Beedy wrote:

Currently, if I scale the units of my application, the new units do not
have the same constraints as the previously deployed units.


Well, I certainly think they should be carrying over the constraints as 
the constraints are set on the application, not the unit.


If this isn't working, I believe it is a bug.

Can you give an example of what is not working?

Cheers,
Tim


Moreover, it
doesn't seem there is anyway to specify constraints for additional
units. I'm thinking the 'add-unit' command needs to be able to accept
constraints, or constraints need to be carried forward for additional
units of the same application.  This is a request for the 'add-unit'
command to accept constraints, or for constraints to grow some sort of
associativity/sensitivity to applications. Also, this is a huge blocker
for me right now. Any insight as to how we might solve this issue would
be greatly appreciated.

Thanks

https://bugs.launchpad.net/juju/+bug/1673639




--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: regression: restore-backup broken by recent commit

[Tim Penhey]

OK, I think I got it now...

This is all crazy, and it was a change due to the  gorilla/websocket change.

So... what happens when there is a successful restore on the server side 
is that it calls os.Exit(...) which then has the pid 1 restart the 
agent. However from the API client, this is an abnormal closure.


In the rpc layer, I capture a number of websocket close errors as 
"normal", but I missed the Abnormal closure case, which is 1006.


I'll update, and repropose to devel.

Hazaah.

Tim



On 24/02/17 16:17, Tim Penhey wrote:

Hi Curtis (also expanding to juju-dev),

I have been looking into this issue. And the good news is that it
doesn't appear to be a real problem with gorilla/websocket at all, but
instead a change in timing showed an existing issue that hadn't surfaced
before.

I'll be looking into that issue - where the restore command after
bootstrapping, doesn't appear to retry if it gets an error like "denied:
upgrade in progress".

Secondly I tried to reproduce on lxd to find that there is an issue with
the rebootstrap and lxd - it just doesn't work.

Then I tried with AWS, to mirror the CI test as close as possible. I
didn't hit the same timing issue as before, but instead got a different
failure with the mongo restore:

  http://pastebin.ubuntu.com/24056766/

I have no idea why juju.txns.stash failed but juju.txns and
juju.txns.logs succeeded.

Also, a CI run of a develop revision just before the gorilla/websocket
reversion hit this:

http://reports.vapour.ws/releases/4922/job/functional-ha-backup-restore/attempt/5045#highlight


cannot create collection "txns": unauthorized mongo access: not
authorized on juju to execute command { create: "txns" }
(unauthorized access)

Not sure why that is happening either. Seems that the restore of mongo
is incredibly fragile.

Again, this shows errors in the restore code, but luckily it has nothing
to do with gorilla/websockets.

Tim

On 23/02/17 04:02, Curtis Hovey-Canonical wrote:

Hi Tim, et al.

All the restore-backup tests in all the substrates failed with your
recent gorilla socket commit. The restore-backup command is often
fails when bootstrap or connection behaviours change. This new bug is
definitely a connection failure while the client is driving a
restore.

We need the develop branch fixed. As the previous commit was blessed,
as are certain 2.2-alpha1 was in very good shape before the gorilla
change.

Restore backup failed websocket: close 1006
https://bugs.launchpad.net/juju/+bug/1666898

As seen at
http://reports.vapour.ws/releases/issue/5550dda7749a561097cf3d44

All the restore-backup tests failed when testing commit
https://github.com/juju/juju/commit/f06c3e96f4e438dc24a28d8ebf7d22c76fff47e2


We see
Initial model "default" added.
04:54:39 INFO juju.juju api.go:72 connecting to API addresses:
[52.201.105.25:17070 172.31.15.167:17070]
04:54:39 INFO juju.api apiclient.go:569 connection established to
"wss://52.201.105.25:17070/model/89bcc17c-9af9-4113-8417-71847838f61a/api"

...
04:55:20 ERROR juju.api.backups restore.go:136 could not clean up
after failed restore attempt: 
04:55:20 ERROR cmd supercommand.go:458 cannot perform restore: :
codec.ReadHeader error: error receiving message: websocket: close 1006
(abnormal closure): unexpected EOF

This is seen in aws, prodstack, gce







--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: regression: restore-backup broken by recent commit

[Tim Penhey]

On 24/02/17 16:17, Tim Penhey wrote:

Also, a CI run of a develop revision just before the gorilla/websocket
reversion hit this:

http://reports.vapour.ws/releases/4922/job/functional-ha-backup-restore/attempt/5045#highlight


cannot create collection "txns": unauthorized mongo access: not
authorized on juju to execute command { create: "txns" }
(unauthorized access)

Not sure why that is happening either. Seems that the restore of mongo
is incredibly fragile.


Actually 4922 was the test run after the reversion.

Clearly there is some issue here, and particularly errors that showed up 
from the CI tests. What we need is more data...


Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: regression: restore-backup broken by recent commit

[Tim Penhey]

Hi Curtis (also expanding to juju-dev),

I have been looking into this issue. And the good news is that it 
doesn't appear to be a real problem with gorilla/websocket at all, but 
instead a change in timing showed an existing issue that hadn't surfaced 
before.


I'll be looking into that issue - where the restore command after 
bootstrapping, doesn't appear to retry if it gets an error like "denied: 
upgrade in progress".


Secondly I tried to reproduce on lxd to find that there is an issue with 
the rebootstrap and lxd - it just doesn't work.


Then I tried with AWS, to mirror the CI test as close as possible. I 
didn't hit the same timing issue as before, but instead got a different 
failure with the mongo restore:


  http://pastebin.ubuntu.com/24056766/

I have no idea why juju.txns.stash failed but juju.txns and 
juju.txns.logs succeeded.


Also, a CI run of a develop revision just before the gorilla/websocket 
reversion hit this:


http://reports.vapour.ws/releases/4922/job/functional-ha-backup-restore/attempt/5045#highlight

cannot create collection "txns": unauthorized mongo access: not
authorized on juju to execute command { create: "txns" }
(unauthorized access)

Not sure why that is happening either. Seems that the restore of mongo 
is incredibly fragile.


Again, this shows errors in the restore code, but luckily it has nothing 
to do with gorilla/websockets.


Tim

On 23/02/17 04:02, Curtis Hovey-Canonical wrote:

Hi Tim, et al.

All the restore-backup tests in all the substrates failed with your
recent gorilla socket commit. The restore-backup command is often
fails when bootstrap or connection behaviours change. This new bug is
definitely a connection failure while the client is driving a
restore.

We need the develop branch fixed. As the previous commit was blessed,
as are certain 2.2-alpha1 was in very good shape before the gorilla
change.

Restore backup failed websocket: close 1006
https://bugs.launchpad.net/juju/+bug/1666898

As seen at
http://reports.vapour.ws/releases/issue/5550dda7749a561097cf3d44

All the restore-backup tests failed when testing commit
https://github.com/juju/juju/commit/f06c3e96f4e438dc24a28d8ebf7d22c76fff47e2

We see
Initial model "default" added.
04:54:39 INFO juju.juju api.go:72 connecting to API addresses:
[52.201.105.25:17070 172.31.15.167:17070]
04:54:39 INFO juju.api apiclient.go:569 connection established to
"wss://52.201.105.25:17070/model/89bcc17c-9af9-4113-8417-71847838f61a/api"
...
04:55:20 ERROR juju.api.backups restore.go:136 could not clean up
after failed restore attempt: 
04:55:20 ERROR cmd supercommand.go:458 cannot perform restore: :
codec.ReadHeader error: error receiving message: websocket: close 1006
(abnormal closure): unexpected EOF

This is seen in aws, prodstack, gce





--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: removing colours from loggo package

[Tim Penhey]

Tech board says "yes let's do it", I'll respond to PR.

On 22/02/17 14:15, Andrew Wilkins wrote:

On Tue, Feb 21, 2017 at 10:02 PM roger peppe mailto:roger.pe...@canonical.com>> wrote:

In August, the loggo package was changed to make it log ANSI-terminal
colour escape sequences by default.

I'm not sure that was the right decision, for a couple of reasons:

- daemons are often given a pseudo tty to run in, so the log files
produced by our running services are now obscured by escape sequences,
making them hard to process automatically (and larger).

- it means that the loggo dependencies are much larger.  Where
previously the loggo package had no non-stdlib dependencies, it now
depends on 5 significantly sized repositories holding >100k lines of
source, including at least one system-dependent (Windows-only) repo.

I'd like to propose that we remove the colour-by-default logic from
loggo and move it to a separate package where it can be used if
required by a given client. Perhaps github.com/utils/loggocolor
 might
be a reasonable place.

Then we can change the juju command line client to use that writer by
default meaning there should be no change in externally visible
behaviour to juju client users. The jujud server executable should
probably not be using coloured logging output IMO.

Thoughts?


Seems reasonable to me. I've added it to today's tech board agenda.


  cheers,
rog.

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com 
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev





--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Change in underlying websocket library

[Tim Penhey]

Hi folks,

Just merged into the "devel" branch (will be 2.2 branch) is a change in 
the library Juju is using for websockets.


Hopefully you won't notice a thing, but if you do see strange 
interactions I'd love to know.


I did test with new CLI / old controller, and it works, and tested with 
new controller / old agents and CLI and it works. And obviously new / 
new works too.


The decision to change the websocket library was made by the tech board 
over a year ago but was just put on the back burner because other things 
were more urgent.  I was following through a thread on memory leaks and 
thought that the websocket library may be contributing to missing when 
some servers went away.


There has been a lot of work in the 2.1 branch dealing with memory 
leaks. Many have been fixed, but there is one particular case that we 
know is still there.  When agents disappear, sometimes the controller 
part that is handling the logs for that agent doesn't notice. This will 
mean extra goroutines, and if the model has been deleted, an extra leak 
of a database object. This type of leak especially hits continuous 
testing situations where models are being created, deployed, then 
destroyed in quick succession - as is the case with the Canonical OIL 
testing.


I have a gut feeling that the problem isn't entirely fixed just by 
changing the libraries, but it may have helped. What it has done though 
is given us some more options to fixing it fully - the proper way.


Cheers,
Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Next 2.1 release to be beta5 not rc1

[Tim Penhey]

Hi all,

Based on a number of key issues with beta4, and the nature of some of 
the networking changes that will be landing at the start of next week, 
it is clear that the next release isn't a release candidate, as we need 
to make sure we have dealt with the issues.


I'm not a fan of calling things that they are not, and calling the next 
release of 2.1 a release candidate would just be wrong.


The target release date for 2.1-beta5 is 2017-01-26 (yay ISO8601).

Cheers,
Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: A (Very) Minimal Charm

[Tim Penhey]

Make sure you also run on LXD with a decent delay to the APT archive.

That is what makes my local testing slow.

Tim

On 15/12/16 13:34, Marco Ceppi wrote:

So, I wanted to circle back around to this thread. I think a lot of good
feedback has come from this, and we're looking into making the reactive
framework leaner and better for charm authors. However, I ran a few
deploy tests and have the following results:

15 Dec 2016 00:18:53Zworkload waitingwaiting for machine
15 Dec 2016 00:18:53Zjuju-unitallocating
15 Dec 2016 00:20:13Zworkload waitinginstalling agent
15 Dec 2016 00:20:13Zworkload waitingagent initializing
15 Dec 2016 00:20:14Zworkload maintenanceinstalling charm software
15 Dec 2016 00:20:14Zjuju-unitexecuting  running install hook
15 Dec 2016 00:20:46Zworkload active ready
15 Dec 2016 00:20:46Zjuju-unitexecuting  running leader-elected hook
15 Dec 2016 00:20:47Zjuju-unitexecuting  running start hook

I did this a few more times on Amazon, and the results were almost
identical. We have 80 seconds from machine requested to booted in cloud.
Less than a second for agent to initialize and 32 seconds to go from
install hook running to the workload being ready and active. While I'm
sure we can slim that down 10-15 seconds by not installing
build-essentials the largest time suck is still the cloud bringing up
the instance.

I plan on doing this across all the clouds I have access to, and track
in a spreadsheet. I'll share that sheet out in a bit.

Thanks,
Marco Ceppi

On Thu, Dec 1, 2016 at 5:00 AM Adam Collard mailto:adam.coll...@canonical.com>> wrote:

On Thu, 1 Dec 2016 at 04:02 Nate Finch mailto:nate.fi...@canonical.com>> wrote:

On IRC, someone was lamenting the fact that the Ubuntu charm
takes longer to deploy now, because it has been updated to
exercise more of Juju's features.  My response was - just make a
minimal charm, it's easy.  And then of course, I had to figure
out how minimal you can get.  Here it is:

It's just a directory with a metadata.yaml in it with these
contents:

name: min
summary: nope
description: nope
series:
  - xenial

(obviously you can set the series to whatever you want)
No other files or directories are needed.


This is neat, but doesn't detract from the bloat in the ubuntu charm.

IMHO the bloat in the ubuntu charm isn't from support for Juju
features, but the switch to reactive plus conflicts in layer-base
wanting to a) support lots of toolchains to allow layers above it to
be slimmer and b) be a suitable base for "just deploy me" ubuntu.
--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com 
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev





--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: juju/retry take 2 - looping

[Tim Penhey]

On 26/10/16 10:30, Katherine Cox-Buday wrote:

I think this is a hint that this is the wrong approach. The edge-cases begin 
showing the cracks in the abstraction and end up making the code more complex. 
Consider your example instead of:

var finalResult Foo
for loop := retry.Loop(spec); loop.Next(); {
result, err := SomeFunc(blah)
if err != nil || resultNotGoodEnough(result) {
continue
}

finalResult = result
break
}

There are no special errors, no mixing of concerns, just a boring imperative 
loop. It works like any other loop written in Go.


I was trying to reduce the need for continue/break. Forcing 
continue/break does not make it like all boring imperative loops.


The err used inside the loop should be declared outside if you want to 
be able to see if the loop succeeded or failed.


Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: juju/retry take 2 - looping

[Tim Penhey]

Some comments first, then addressing issues:

I thought it might help if I walked you through my thought process of 
making this change.


* the purpose of the retry package is to provide a way to retry actions 
that may fail until they succeed, or some predetermined limit is hit

* the way that failure is idiomatically handled is with an error
* I wanted the interface to be very simple to use, and hard to get wrong
* it should work inline in a simple for loop
* the library should make the simple case very simple

var result Foo
var err error
for loop := retry.Loop(spec); loop.Next(err); {
 result, err = SomeFunc(blah)
}
if err != nil {
 // what ever
}
// continue using result

What about fatal errors?

var result Foo
var err error
for loop := retry.Loop(spec); loop.Next(err); {
 result, err = SomeFunc(blah)
 if isFatalErr(err) {
 break
 }
}
if err != nil {
   // what ever
}
// continue using result


On 21/10/16 06:33, roger peppe wrote:

On 20 October 2016 at 16:30, Katherine Cox-Buday
 wrote:

John Meinel  writes:


As commented on the pull request, passing 'err' into Next() initially feels 
weird, but
it allows a big improvement to whether the loop exited because we ran out of
retries, or it exited because the request succeeded. (loop.Error()
tells us either way.)


I don't understand why the retry logic has the concern of why the loop exits 
(i.e. Next(error)).


The rationale behind that is to make the the library handle the 80% case 
easily. The main reason of passing the error into Next is so you don't 
have to do the check and explicit break within the loop.



The motivation of moving towards loops was so that the concern of what's being 
retried is brought back to be inline with the surrounding code. Having the 
retry mechanism inspecting errors doesn't fall in line with that goal.

In my mind, any retry solutions only needs to cover two things:
1. Provide a primitive that will delay a for iteration in a controlled way.
2. Be preemptable.

Everything else is the logic of the thing you're retrying, including why it 
eventually succeeded/failed.


Here is were we disagree slightly. I also think that very high on the 
list are:

   * easy to get right
   * hard to get wrong


That's well put, and fits well with my thoughts too, thanks.

See also Pawel Stolowski's comment on
https://github.com/juju/retry/pull/5. He's from the Snappy
team who have also been trying to move towards using a standard retry mechanism.
There are many possible reasons for wanting to retry - error values
are just one possibility.


Sure, but this could be trivially handled by assigning something to err 
(consider if we made a public error in the retry package for this):


var result Foo
var err error
for loop := retry.Loop(spec); loop.Next(err); {
   result, err = SomeFunc(blah)
   if err == nil && resultNotGoodEnough(result) {
 err = retry.ErrTryAgain
   }
}
if err != nil {
   // what ever
}
// continue using result

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

juju/retry take 2 - looping

[Tim Penhey]

Hi folks,

https://github.com/juju/retry/pull/5/files

As often is the case, the pure solution is not always the best. What 
seemed initially like the best approach didn't end up that way.


Both Katherine and Roger had other retry proposals that got me thinking 
about changes to the juju/retry package. The stale-mate from the tech 
board made me want to try another approach that I thought about while 
walking the dog today.


I wanted the security and fall-back of validation of the various looping 
attributes, while making the call site much more obvious.

The pull request has the result of this attempt.

It is by no means perfect, but an improvement I think. I was able to 
trivially reimplement retry.Call with the retry.Loop concept with no 
test changes.


The tests are probably the best way to look at the usage.

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju 2.0 is here!

[Tim Penhey]

I concur. You only have to use 1.25 for a short while again to see how 
far Juju has come.


Be proud of your work, celebrate the release.

Go team!

Tim

On 14/10/16 17:50, Mark Shuttleworth wrote:


Congrats everyone, this is a release to be proud of. Multi-user
multi-model, great CLI, it's a joy to train people on it. Well done.

Mark

On 14/10/16 06:34, Nicholas Skaggs wrote:

Juju 2.0 is here! This release has been a year in the making. We’d
like to thank everyone for their feedback, testing, and adoption of
juju 2.0 throughout its development process! Juju brings refinements
in ease of use, while adding support for new clouds and features.

## New to juju 2?

https://jujucharms.com/docs/2.0/getting-started

## Need to install it?

If you are running Ubuntu, you can get it from the juju stable ppa:

sudo add-apt-repository ppa:juju/stable
sudo apt update; sudo apt install juju-2.0

Or install it from the snap store

snap install juju --beta --devmode

Windows, Centos, and MacOS users can get a corresponding installer at:

https://launchpad.net/juju/+milestone/2.0.0

## Want to upgrade to GA?

Those of you running an RC version of juju 2 can upgrade to this
release by running:

juju upgrade-juju

## Feedback Appreciated!

We encourage everyone to subscribe the mailing list at
j...@lists.ubuntu.com and join us on #juju on freenode. We would love
to hear
your feedback and usage of juju.







--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Github Reviews vs Reviewboard

[Tim Penhey]

-1, like Menno I was initially quite hopeful for the github reviews.

My main concerns are around easily having a list to pull from, and being 
able to see status, comments on a single dashboard.


Tim

On 14/10/16 11:44, Menno Smits wrote:

We've been trialling Github Reviews for some time now and it's time to
decide whether we stick with it or go back to Reviewboard.

We're going to have a vote. If you have an opinion on the issue please
reply to this email with a +1, 0 or -1, optionally followed by any
further thoughts.

  * +1 means you prefer Github Reviews
  * -1 means you prefer Reviewboard
  * 0 means you don't mind.

If you don't mind which review system we use there's no need to reply
unless you want to voice some opinions.

The voting period starts *now* and ends my*EOD next Friday (October 21)*.

As a refresher, here are the concerns raised for each option.

*Github Reviews*

  * Comments disrupt the flow of the code and can't be minimised,
hindering readability.
  * Comments can't be marked as done making it hard to see what's still
to be taken care of.
  * There's no way to distinguish between a problem and a comment.
  * There's no summary of issues raised. You need to scroll through the
often busy discussion page.
  * There's no indication of which PRs have been reviewed from the pull
request index page nor is it possible to see which PRs have been
approved or otherwise.
  * It's hard to see when a review has been updated.

*Reviewboard*

  * Another piece of infrastructure for us to maintain
  * Higher barrier to entry for newcomers and outside contributors
  * Occasionally misses Github pull requests (likely a problem with our
integration so is fixable)
  * Poor handling of deleted and renamed files
  * Falls over with very large diffs
  * 1990's looks :)
  * May make future integration of tools which work with Github into our
process more difficult (e.g. static analysis or automated review tools)

There has been talk of evaluating other review tools such as Gerrit and
that may still happen. For now, let's decide between the two options we
have recent experience with.

- Menno




--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Big memory usage improvements

[Tim Penhey]

I have a strong feeling that we are still "leaking".

Was talking with Christian this morning and he has some plans for 
introspection to isolate the issues.


Tim

On 13/10/16 10:36, Katherine Cox-Buday wrote:

Menno Smits  writes:


Christian (babbageclunk) has been busy fixing various memory leaks in the Juju
controllers and has made some significant improvements.


Awesome, good work, Christian!

Not to detract from this fantastic news, but it's still worrisome that an idle 
Juju seems to have a memory which is growing linearly (before picture looked 
like the beginning of an exponential curve?). Do we feel this is memory which 
will at some point be GCed?



--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Big memory usage improvements

[Tim Penhey]

This is awesome.

Tim

On 13/10/16 09:39, Menno Smits wrote:

Christian (babbageclunk) has been busy fixing various memory leaks in
the Juju controllers and has made some significant improvements. Chris
(veebers) has been tracking resource usage for a long running test which
adds and removes a bunch of models and he noticed the difference.

Take a look at the memory usage graphs here:

Before: http://people.canonical.com/~leecj2/perfscalemem/
After: http://people.canonical.com/~leecj2/perfscalemem2/

Interestingly the MongoDB memory usage profile is quite different as
well. I'm not sure if this is due to Christian's improvements or
something else.

There's possibly still some more small leaks somewhere but this is
fantastic regardless. Thanks to Christian for tackling this and Chris
for tracking the numbers.

- Menno






--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: List plugins installed?

[Tim Penhey]

Thanks Marco.

Would be good to have something solid to drive the plug-in changes with.

Tim

On 30/09/16 11:16, Marco Ceppi wrote:

Thanks have some ideas about this, I'll file a bug (blueprint?) about
it. I really care about plugins and would like to make them more robust
in Juju.

Marco


On Thu, Sep 29, 2016, 6:07 PM Tim Penhey mailto:tim.pen...@canonical.com>> wrote:

If we do that, then we can make the plug-in also install a metadata file
that explains help and usage, so you don't call the script to do that.

It makes it easy to list plug-ins, because you are searching a known
location, and not the entire path. Only show plug-ins that have the
appropriate meta-data file.

Tim

On 30/09/16 10:47, Nate Finch wrote:
> Seem alike the easiest thing to do is have a designated plugin
directory
> and have juju install  copy the binary/script there.
> Then we're only running plugins the user has specifically asked to
install.
>
> On Thu, Sep 29, 2016, 4:33 AM Stuart Bishop
mailto:stuart.bis...@canonical.com>
> <mailto:stuart.bis...@canonical.com
<mailto:stuart.bis...@canonical.com>>> wrote:
>
> On 28 September 2016 at 22:45, roger peppe
> mailto:roger.pe...@canonical.com>
<mailto:roger.pe...@canonical.com
<mailto:roger.pe...@canonical.com>>> wrote:
>
> On 28 September 2016 at 14:55, Rick Harding
> mailto:rick.hard...@canonical.com>
<mailto:rick.hard...@canonical.com <mailto:rick.hard...@canonical.com>>>
> wrote:
> > This is just a miss. The original ability to see the
plugins was a subset of
> > the help command and didn't make our CLI spreadsheet for
things to rework. I
> > agree that list-plugins is the right idea here and that
means that plugins
> > becomes a noun in our language.
> >
> > What's interesting is that add/remove fall out because that
> > installing/uninstalling. I think that show-plugin might
be interesting to
> > auto run the --description flag to bring it into CLI
alignment with the new
> > world order.
>
> I've voiced discomfort with this before - I don't think
that we
> should
> arbitrarily run all executables that happen to have a "juju-"
> prefix.
> It's potentially dangerous (for example, note that
although git
> relies heavily
> on plugins, it doesn't execute a plugin until you explicitly
> name it).
>
> Perhaps there could be a standard way for a plugin to provide
> metadata about itself as a data file.
>
>
> It also might be time to work out how a Juju snap is going to call
> or install plugins. I don't think the existing design is going to
> work, and there is still time to flag it as deprecated in the
> changelogs for 2.0 and work out the way forward for 2.1.
>
>
> --
> Stuart Bishop mailto:stuart.bis...@canonical.com>
> <mailto:stuart.bis...@canonical.com
<mailto:stuart.bis...@canonical.com>>>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com <mailto:Juju-dev@lists.ubuntu.com>
<mailto:Juju-dev@lists.ubuntu.com <mailto:Juju-dev@lists.ubuntu.com>>
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com <mailto:Juju-dev@lists.ubuntu.com>
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev



--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: List plugins installed?

[Tim Penhey]

If we do that, then we can make the plug-in also install a metadata file 
that explains help and usage, so you don't call the script to do that.


It makes it easy to list plug-ins, because you are searching a known 
location, and not the entire path. Only show plug-ins that have the 
appropriate meta-data file.


Tim

On 30/09/16 10:47, Nate Finch wrote:

Seem alike the easiest thing to do is have a designated plugin directory
and have juju install  copy the binary/script there.
Then we're only running plugins the user has specifically asked to install.

On Thu, Sep 29, 2016, 4:33 AM Stuart Bishop mailto:stuart.bis...@canonical.com>> wrote:

On 28 September 2016 at 22:45, roger peppe
mailto:roger.pe...@canonical.com>> wrote:

On 28 September 2016 at 14:55, Rick Harding
mailto:rick.hard...@canonical.com>>
wrote:
> This is just a miss. The original ability to see the plugins was a 
subset of
> the help command and didn't make our CLI spreadsheet for things to 
rework. I
> agree that list-plugins is the right idea here and that means that 
plugins
> becomes a noun in our language.
>
> What's interesting is that add/remove fall out because that
> installing/uninstalling. I think that show-plugin might be 
interesting to
> auto run the --description flag to bring it into CLI alignment with 
the new
> world order.

I've voiced discomfort with this before - I don't think that we
should
arbitrarily run all executables that happen to have a "juju-"
prefix.
It's potentially dangerous (for example, note that although git
relies heavily
on plugins, it doesn't execute a plugin until you explicitly
name it).

Perhaps there could be a standard way for a plugin to provide
metadata about itself as a data file.


It also might be time to work out how a Juju snap is going to call
or install plugins. I don't think the existing design is going to
work, and there is still time to flag it as deprecated in the
changelogs for 2.0 and work out the way forward for 2.1.


--
Stuart Bishop mailto:stuart.bis...@canonical.com>>
--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com 
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev





--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Timeout added to kill-controller

[Tim Penhey]

Hi all,

NOTE: we do very much consider it a bug if the models don't die properly.

I have just landed a fix for a kill-controller issue where it would just 
sit there for a long time with nothing apparent happening.


Now kill-controller has a default timeout of 5 minutes. If there is no 
change in the timeout period, the command switches to a direct 
destruction mode where it will contact the cloud provider on behalf of 
each remaining model and destroy it that way.


The following examples all used LXD, and were a single controller 
machine with ubuntu deployed in the default model.


$ time juju kill-controller kill-test-fine -y
Destroying controller "kill-test-fine"
Waiting for resources to be reclaimed
Waiting on 1 model, 1 machine, 1 application
Waiting on 1 model, 1 machine, 1 application
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model
All hosted models reclaimed, cleaning up controller machines

real0m27.443s


Nothing much changes here. Everything died nicely.
You can specify a timeout with --timeout (or -t). Valid formats are 
things like "2m" for two minutes, "30s" for thirty seconds.

Zero also works:

$ time juju kill-controller kill-test-no-delay -t 0 -y
Destroying controller "kill-test-no-delay"
Waiting for resources to be reclaimed
Killing admin@local/default directly
  done
All hosted models destroyed, cleaning up controller machines

real0m2.492s


I had to throw a wrench in the works to get the provisioner to not kill 
the machine (wrench is a test facility we have). This allows me to show 
you a model that doesn't die like it should. I just specify a one minute 
timeout. The polling time was moved from two seconds to five seconds. 
Now you will see a countdown starting after 30 seconds of no change.


$ juju kill-controller kill-test -t 1m -y
Destroying controller "kill-test"
Waiting for resources to be reclaimed
Waiting on 1 model, 1 machine, 1 application
Waiting on 1 model, 1 machine, 1 application
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine, will kill machines directly in 29s
Waiting on 1 model, 1 machine, will kill machines directly in 24s
Waiting on 1 model, 1 machine, will kill machines directly in 19s
Waiting on 1 model, 1 machine, will kill machines directly in 14s
Waiting on 1 model, 1 machine, will kill machines directly in 9s
Waiting on 1 model, 1 machine, will kill machines directly in 4s
Killing admin@local/default directly
  done
All hosted models destroyed, cleaning up controller machines


Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Proposal: display-name for charm metadata

[Tim Penhey]

+1 for optional 'display-name' field that doesn't have the naming 
restrictions of 'name'


-1 for case insensitivity use as part of this work.

Tim

On 25/09/16 05:02, Marco Ceppi wrote:

On Sat, Sep 24, 2016 at 11:34 AM Alex Kavanagh
mailto:alex.kavan...@canonical.com>> wrote:

Why not allow the display of the name to be case sensitive, but all
usage to be case insensitive?  So name is MySQL, but you can juju
deploy mYSqL if you really wanted to.


I expect display names may also include unicode characters in the
future, for example

Übersoftware

Which would need name to still be defined from a store/unique id
perspective.

As for case insensitive juju deploy command, I'd consider that out of
scope of this proposal.

On Sat, Sep 24, 2016 at 2:50 PM, Marco Ceppi
mailto:marco.ce...@canonical.com>> wrote:

Hey everyone,

I know we're rocking towards 2.0 but this is a problem I've seen
voiced a few times now. To date, the `name` field in charm has
always been [a-z-0-9_-] where you can't end with `-#`. This
makes sense, simple flat names that are all lowercase make it
easy to do `juju deploy wordpress` instead of following branding
guidelines of `juju deploy WordPress`.

However, a lot of applications have very specific branding
guidelines for how their display name should be represented.
Just a few for example:

- WordPress
- NS1
- MySQL
- PostgreSQL

Today, in the charmstore each is rendered as:

- Wordpress
- Ns1
- Mysql
- Postgresql

Very rarely do the display names in the charm store and the
intended branding of application align. I'd like to propose an
optional field in the charm metadata, `display-name` which would
allow slightly more control over charmstore display:

```
name: ns1
display-name: NS1
```

```
name: mysql
display-name: MySQL
```

etc. This would lead to the store and other places across the
Juju Charms properties which referenced the Application, instead
of the deployment instructions, to use the display-name field
(see attached).

Curious opinions on this, repercussions of adding metadata
fields, esp for older versions of Juju, and if this is worth
pursing.

--
Juju mailing list
j...@lists.ubuntu.com 
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju




--
Alex Kavanagh - Software Engineer
Cloud Dev Ops - Solutions & Product Engineering - Canonical Ltd





--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Reviews on Github

[Tim Penhey]

I'm +1 if we can remove the extra tools and we don't get email per comment.

On 15/09/16 08:03, Nate Finch wrote:

In case you missed it, Github rolled out a new review process.  It
basically works just like reviewboard does, where you start a review,
batch up comments, then post the review as a whole, so you don't just
write a bunch of disconnected comments (and get one email per review,
not per comment).  The only features reviewboard has is the edge case
stuff that we rarely use:  like using rbt to post a review from a random
diff that is not connected directly to a github PR. I think that is easy
enough to give up in order to get the benefit of not needing an entirely
separate system to handle reviews.

I made a little test review on one PR here, and the UX was almost
exactly like working in reviewboard: https://github.com/juju/juju/pull/6234

There may be important edge cases I'm missing, but I think it's worth
looking into.

-Nate




--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Latest new about Juju master branch - upload-tools obsoleted

[Tim Penhey]

OK, I think I've got it now.

On 16/08/16 15:19, Ian Booth wrote:



On 16/08/16 12:58, Tim Penhey wrote:



On 16/08/16 10:50, Ian Booth wrote:


On 16/08/16 03:09, Nate Finch wrote:

Ian, can you describe how Juju decides if it's running for a developer or
an end user?  I'm worried this could trip people up who are both end users
and happen to have a juju development environment.



It's not so much Juju deciding - the use cases given were from the point of view
of a developer or end user.

Juju will decide that it can automatically fallback to try to find and use a
local jujud (so long as the version of the jujud found matches that of the Juju
client being used to bootstrap or upgrade) if:

- the Juju client version is newer than the agents running
- the client or agents have a build number > 0

(the build number is 0 for released Juju agents but non zero when jujud is used
or built locally from source).


But this isn't entirely true is it? The build number is a horrible hack
involving a version override file.

When I build jujud locally from source there is no version override and it is
just the version as defined in the code I'm building.



My wording was sadly suboptimal.
The agent reports a version containing a non-zero build number if uploaded or
built from source. So I was trying to refer to the version that the client had
reported to it.



--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Latest new about Juju master branch - upload-tools obsoleted

[Tim Penhey]

On 16/08/16 10:50, Ian Booth wrote:


On 16/08/16 03:09, Nate Finch wrote:

Ian, can you describe how Juju decides if it's running for a developer or
an end user?  I'm worried this could trip people up who are both end users
and happen to have a juju development environment.



It's not so much Juju deciding - the use cases given were from the point of view
of a developer or end user.

Juju will decide that it can automatically fallback to try to find and use a
local jujud (so long as the version of the jujud found matches that of the Juju
client being used to bootstrap or upgrade) if:

- the Juju client version is newer than the agents running
- the client or agents have a build number > 0

(the build number is 0 for released Juju agents but non zero when jujud is used
or built locally from source).


But this isn't entirely true is it? The build number is a horrible hack 
involving a version override file.


When I build jujud locally from source there is no version override and 
it is just the version as defined in the code I'm building.


Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

RFC: time to remove the "smart" output formatter?

[Tim Penhey]

Hi folks,

While perusing the codebase, I was reminded of our "smart" output formatter.

It is the yaml but not quite yaml one.

I propose we remove it, and default commands that were defaulting to 
that to use yaml instead.


Thoughts?

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

RFC: proposed changes to debug-log API call

[Tim Penhey]

Hi folks,

This is regarding the streaming /api/:model-uuid/log api server endpoint.

Currently this method sends an initial possible error, then just turns 
on a tap of bytes of preformatted log messages.


What I propose is to change the streaming protocol from just text to a 
stream of structured log messages.


This initial change will have no impact at all on the CLI, but allows 
for future changes like:

  - showing logs in local time instead of UTC
  - hiding the filenames by default
  - not showing the date by default
  - colors

Is anyone currently using the streaming text aspects of the API right now?

How much warning would you like?  Ideally this change needs to happen 
before we freeze the API for backwards compatibility.


Cheers,
Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: BDX <-> Juju

[Tim Penhey]

This sounds AMAZING!!!

Great to hear Juju making a real difference.

Thanks for sharing.
Tim

On 02/08/16 16:38, James Beedy wrote:

Team,

As some of you may know, I have taken on a new position as DevOps
Engineer for a creative company -> CreativeDrive
. CreativeDrive is the parent company of 5
child creative companies which it acquired over the last year. Being as
CreativeDrive itself is just over a year old, and with the acquisition
of the 5 child companies; we have a large need, and a lot of room for
Juju in our application/dev-ops stack. In short, I/we have 100s (not
joking) of applications that could benefit from being charmed up and
deployed (to a private openstack if I can forge my will) back to aws.

I have immediately (in my first week) charmed up 3 big apps, and
demonstrated the power of Juju internally throughout the company with a
lot of success in the acceptance of its uptake. Minus a small hiccup
with an aws vpc (operator error) (also huge thanks @lazyPower for
pulling through with the assist in diagnosing the performance of the
cluster on the spot), I was also able to flawlessly save a demo of an
application whose elasticsearch cluster had been compromised, by
spinning up a new Juju deployed elasticsearch cluster immediately, and
getting it re-indexed within a few hours - this was overseen by our VP
of technology and made a huge impression with our higher ups (first day
on the job). I have since created an infrastructure plan for how we
might use Juju to facilitate our many needs throughout the company, and
shown how simple reactive patterns can be a charm template for every
next application.

One piece of the puzzle I wanted to get squared away before the rest is
our secrets lifecycle management. Following too many (high level yet
technical) discussions with upper management concerning the concept of
"secrets" in general, we have decided to procure an implementation of
vault + consul for encrypted key/value store. I think this is a great
choice for us, seeing as we have many separate apps, app envs, and app
developers; the use of vault will allow me (as an admin) to interface to
the application developers with the respective vault api for their
application language(s), and in a sense create an
tooling/platform/application/language agnostic interface for secrets
that can be entirely decoupled from other tooling and easily consumed by
any application.

Hypothetically, with the vault + consul stack in place, the app devs
will interface to it, and use a similar "get secrets in our app"
template for each language/framework/app, leaving me free to charm up
our applications. In the eyes of the company I have incurred two primary
efficiencies: a template for new applications need be charmed - few
custom modifications needed for each next/new app to become "charmed" or
deployable, and a secrets interface to the app devs through vault -
tightening down our secrets visibility, whilst opening them up for
consumption, and templating of their provisioning in the application code.

All this to say, I will be making heavy use of aws and rackspace
providers over the next amount of time, and want to say thank you in
advance for your support :-)

Hope you enjoyed the update - more to come!

2.0 - so close SO EXCITED!!! WH00T!!!




--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Schema for Juju RPC messages

[Tim Penhey]

While at the London sprint I was toying with the idea of adding the 
ability of the rpc package to do some rudimentary initial validation.


We could get a good part of the way with relatively little effort IMO.

Using the reflect package, we can interrogate the public attributes of 
the structure we are attempting to serialize into. This gives access to 
the structure serialization tags. We could then use these tags to reject 
incoming messages if they provide any keys that are unexpected.


For example, lets look at the ApplicationDeploy params:
(had to munge some of the struct tags for the email)
type ApplicationDeploy struct {
  ApplicationName  string   `json:"application"`
  Series   string   `json:"series"`
  CharmUrl string   `json:"charm-url"`
  Channel  string   `json:"channel"`
  NumUnits int  `json:"num-units"`
  Config   map[string]string `json:"config,omitempty"`
  ConfigYAML   string`json:"config-yaml"`
  Constraints  constraints.Value  `json:"constraints"`
  Placement[]*instance.Placement
 `json:"placement,omitempty"`
  Storage  map[string]storage.Constraints
 `json:"storage,omitempty"`
  EndpointBindings map[string]string
 `json:"endpoint-bindings,omitempty"`
  Resourcesmap[string]string
 `json:"resources,omitempty"`
}

This would give us a set of valid keys:
  ("application", "series", "charm-url", "channel", "num-units",
   "config", "config-yaml", "constraints", "placement", "storage",
   "endpoint-bindings", "resources")

There would be overhead in doing the check though, because instead of 
deserializing directly into the structure, we'd deserialize into 
something like map[string]interface{} first, and do a key check, then 
only deserialize into the structure if it passed.


We could use the "omitempty" tags to mark optional args that can be missing.

Just doing this would catch the typos of keys, and would also make us be 
more strict in the "I'll just add this extra field" to existing facade 
versions, as they could then fail with older versions of the server with 
"unknown field" type errors.


Thoughts?
Tim

On 28/07/16 04:29, John A Meinel wrote:

We've had some requests from people trying to drive Juju that it would
actually be really nice if we were stricter with the messages that we
accept on our API. Specifically, as we've changed the API methods, they
had several hard-to-debug problems because they were passing a parameter
that was named incorrectly, and Juju was not giving them any indication
that something was wrong.

I put together a (very hackish) test branch, to see if I could use
JSONSchema to define our request and response messages, and give nicer
error messages (rather than silent acceptance). As I was working with
JSON, I realized the extra " and { characters really got in the way of
making a document that was readable, so I leveraged our existing YAML to
JSON conversion mechanisms to write the description in YAML and then
load the object tree into JSONSchema to validate our requests.

I did end up getting basic validation of the outer structure (just the
Request/Response message, not the contents of Parameters) functioning here:
https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec.go
You can see what some of the error messages look like here:
https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec_test.go

The actual code itself isn't useful as is, because it needs to have the
schema validation stuff pulled out into its own file, etc. But it does
give a proof-of-concept of basic message validation. I'm not super
excited by some of the error messages
(gojsonschema.ResultError.Description is very nice by itself for missing
keys, extra properties, etc, but not enough information for invalid
values, while ResultError.String() is overly verbose in the opposite way.)

I'd like to get a bit of feedback on whether people would find this
interesting, especially if we brought that all the way into the Param
structs as well. I haven't done any benchmarking to determine if this is
a large overhead or not. But the golfing community seems determined to
not do a Strict Unmarshal function, and seems to be recommending using a
Schema instead.

I'm not completely sold either way, though I do find the YAML format for
the schema description to be far more human readable than the JSON format.

Thoughts?

John
=:->





--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju Secrets

[Tim Penhey]

On 25/07/16 06:32, James Beedy wrote:

  Proposed Solution: Juju Secrets

To give Juju a combative edge on the privacy pinwheel of secrets
distribution in the realm of bleeding edge devops tooling, behold my
hypothetical proposed solution: |juju secrets|.
Juju secrets could be used like so:
|juju add-secret mycloud:mymodel -f secrets.yaml|

I know you guys are pressing hard to get 2.0 out the door, so please
don’t mind my nonsense here. I just wanted to throw the idea out there
and possibly get some feedback, and have others weigh in on this topic.

Thoughts?


Interesting idea. How would this really work in practice?

For the secrets to be any use, the units need to be able to get access 
to the information right?


Who are we hiding the information from? This is always a key question 
that needs to be answered in order to choose a good solution.


Are we hiding information from the Juju users?

Are we trying to hide information from other processes on the same 
machines as the agents?


Also, who generates and holds the keys?

Just some more questions.


Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Reminder: Leaving code better than you find it.

[Tim Penhey]

Hi all,

I'd like to reiterate a key tenet [1] for the core team, and that is
  "leave the code better than you found it"

Often we get too focused on the one thing we are trying to land, and 
disregard other small fixes or changes that could be made to make the 
code better.


I'm not saying that if you touch a file you need to make it "golden", or 
fix everything in the file, but sometimes you can make small related 
changes that leave things in a better state.


Just landing more code often leaves things in the same or worse state, 
please work to leave things in a better state.


Thanks,
Tim


[1] noun, a principle or belief

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Proposed addition to code review guidelines

[Tim Penhey]

Hi folks,

With model migration entering the world as a first class citizen, we 
need to ensure that all fields and documents added to state have 
appropriate migrations done for them.


In particular when adding a field it is likely that the 
state/migrations_internal_test.go will have a failing test as it checks 
the public fields of the documents.


We should add a code review check that ensures that no fields are added 
to that test without the appropriate additions to 
state/migration_export.go and state/migration_import.go otherwise it is 
just a lie, and the migration result will not reflect the new database 
state.


Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

model-migration feature branch has been merged into master

[Tim Penhey]

Hi folks,

The model-migration branch has now been fully merged back into master.

I have deleted the model-migration branch in the github.com/juju/juju 
repo. All future model migration merges are to land in master.


That is all.

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: RFC: remove juju publish?

[Tim Penhey]

Was actually Aaron, I was just confirming with the list :-)

Tim

On 16/07/16 00:08, Rick Harding wrote:

+1, good catch thanks Tim.

On Fri, Jul 15, 2016 at 3:15 AM roger peppe mailto:rogpe...@gmail.com>> wrote:

Yes, it should be removed. It's superceded by charm push.

On 15 July 2016 at 04:32, Tim Penhey mailto:tim.pen...@canonical.com>> wrote:
 > Does 'juju publish' still serve a purpose in Juju 2.0?
 >
 > Should we just remove it?
 >
 > Tim
 >
 > --
 > Juju-dev mailing list
 > Juju-dev@lists.ubuntu.com <mailto:Juju-dev@lists.ubuntu.com>
 > Modify settings or unsubscribe at:
 > https://lists.ubuntu.com/mailman/listinfo/juju-dev

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com <mailto:Juju-dev@lists.ubuntu.com>
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev



--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

RFC: remove juju publish?

[Tim Penhey]

Does 'juju publish' still serve a purpose in Juju 2.0?

Should we just remove it?

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Juju API breaks landed in master, soon to be released as beta 11

[Tim Penhey]

Hi folks,

Continuing the rationalization of the API wire protocol there are more 
breaks to the API.


With beta 10, we thought we fixed all the structures that were sent over 
the API to be consistently lowercase with dash separated words. However 
it was brought to our attention that there were a number of places that 
were missed:
- the top level RPC fields for both requests and responses (things like 
Request, RequestId, Version, Params, Response etc.);

- all the mega-watcher types;
- Client.CharmInfo responses (api.CharmInfo fields are not annotated, 
except for Metrics, capitalised)

- Placement fields

These should now all be lowercase with the dash separated words.

Also there have been a few method moves:
- CharmInfo method has moved from the "Client" facade to the "Charms" facade
- DestroyModel has moved from the "Client" facade to the "ModelManager" 
facade.


Also, the "payloads" and "resources" facade names have been capitalized 
to now be "Payloads" and "Resources" so they are consistent with the 
other facade names.


If you find any key values that do not conform, it is a bug and please 
let us know.


Details of the github pull requests that changed the tagging and names:
 - https://github.com/juju/juju/pull/5739/files
   Payloads and Resources facade name changes, tagging of payloads
   and resource structs
 - https://github.com/juju/juju/pull/5737/files
   CharmInfo tags, multiwatcher Address tags
 - https://github.com/juju/juju/pull/5730/files
   top level RPC calls
 - https://github.com/juju/juju/pull/5724/files
   megawatcher tags, also found a server-uuid which is now
   controller-uuid.
 - https://github.com/juju/juju/pull/5703/files
   tags on charm actions and relation endpoints
 - https://github.com/juju/juju/pull/5702/files
   moves the DestroyModel call

Cheers,
Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

fslock is dead, long live mutex

[Tim Penhey]

Hi folks,

We have finally managed to exorcise the fslock from the codebase. Both 
1.25 and master no longer refer to it at all. We need to remove it from 
the juju/utils package to make sure that people don't accidentally try 
and use it again.


There is a new replacement, the juju/mutex package. To acquire a mutex, 
you create a mutex.Spec structure. You must at least provide a name, 
delay and clock. The delay is how long the code waits between tries to 
acquire the mutex. Named mutexs are shared between processes on the same 
machine. If a process holds the mutex and the process dies, the mutex is 
automatically released.  A spec can also have a timeout value, and/or an 
abort channel.


On linux this is implemented with abstract domain sockets, on windows it 
uses a named semaphore, and on other platforms a flock is used on a 
named temp file.


https://godoc.org/github.com/juju/mutex

Cheers,
Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Breaking API change landing for 2.0-beta10

[Tim Penhey]

Hi folks,

This impacts all people calling the Juju API directly. If you use the 
juju client, you *should* be fine.


Taking advantage of our time in "beta", we are adding consistency to the 
wire-protocol used by the juju client to talk to the juju apiserver.


In general this means that all the parameters are lower-case with words 
separated with dashes. *Error returns should all be "omitempty" meaning 
that if there was no error, there is no "error" entry in the return.


There is no API facade version bump with this change. This allows us to 
start 2.0 with a clean and consistent slate.


The change can be found here:
https://github.com/juju/juju/pull/5674

This change will be landing very soon, perhaps by the time you read this 
message.


Cheers,
Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju 2.0-beta9 ETA

[Tim Penhey]

Hi folks,

Due to a change I landed without fully thinking through the 
implications, the reverting of said change has pushed us out a day.


I was trying to add consistency to the wire-protocol that Juju uses by 
changing the serialisation names. Thinking that we were still in the "we 
don't need no backward compatibility phase" I made this change without 
bumping the facade versions. This would have broken all the API users. 
This is what we are backing out.


Sorry for the inconvenience.

Tim

On 15/06/16 12:19, Cheryl Jennings wrote:

Hi Everyone,

Due to a critical bug [0] found in the daily PPA, the release of
2.0-beta9 will be delayed while we test the fix.  We are aiming for a
release tomorrow.

Thanks!
-Cheryl

[0] https://bugs.launchpad.net/juju-core/+bug/1592210

On Fri, Jun 10, 2016 at 1:20 PM, Cheryl Jennings
mailto:cheryl.jenni...@canonical.com>>
wrote:

The team has been busily working on 2.0-beta9 with an aim to address
usability feedback and ensure that beta9 will be upgradeable to
subsequent releases.  Ensuring upgradeability has required a
significant amount of work to finalize internal details, and the
team needs a few extra days to make sure this work is completed.

To achieve this guarantee, we are moving the expected date for
2.0-beta9 to Tuesday, June 14.

Some of the great things coming in beta9 include:
- Renaming of 'service' to 'application' to better align terminology
- Shortened instance IDs for the lxd provider (ex: 'juju-622af3-0')
- Addition of a `juju unregister` command to remove references to
controllers
- Separation of controller config vs. model config
- Improved status output
- Numerous bug fixes

There is an early beta9 available in the juju daily ppa
(ppa:juju/daily) for those who wish early access to the above features.

If you have any questions, please let me know.
Thanks!
-Cheryl






--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Regarding Synnefo environment provider for Juju

[Tim Penhey]

Hi Thodoris,

I had a quick look through the current code in the branch, and nothing 
is obvious there.


What is the command line that you are running?

The failure is due to the database already thinking there is a model 
that exists for the same user with the same name. Although now what we 
normally create is two models on bootstrap:

 * controller - where the machines running Juju live (apiserver and db)
 * default - where the user is left to deploy workloads.

The error is that the model name is "test". This shouldn't be the case 
as the database should be empty.


Tim

On 08/06/16 23:28, Thodoris Sotiropoulos wrote:

Hi all,

You may remember previous e-mails sent by my partner Stavros Sachtouris
regarding
the case of implementing a Juju environment provider for our open source
IaaS called
Synnefo.

We have started implementation of the basics (configuration schema,
instance creation,
instance queries, preparation of environment, etc). Our goal is to make
a proof
of concept implementation of the bootstrap command and that's why we
have ignored
networking and storage configuration, a.k.a mocked.

So far, we have managed to create and communicate with a machine
instance. However,
during the last step of bootstrap process (insertion of the admin model
to database)
we are facing an unexpected problem (method `NewModel` of
`state/model.go`).

Here is the corresponding log file:
https://pithos.okeanos.grnet.gr/public/8EHM5jpEm2W7bSwly9wFG

I tried to investigate the problem but I cannot figure out why I get the
`model %s for %q already exists`. What am I missing? I would appreciate any
help or guidance.»

Thank you in advance

Thodoris Sotiropoulos
Developer @ GRNET
theo...@grnet.gr



--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju 2.0-beta9 ETA

[Tim Penhey]

On 12/06/16 21:00, Mark Shuttleworth wrote:

 From an upgrade point of view, I would focus on the migration
capability, because that provides the cleanest semantics. If we have
everything we need there and upgrades happen to work from b9 onwards,
then that's great, but please communicate that I will not hesitate to
make breaking changes until we are in RC if that's the right thing to do
for 2.0. I'll call the RC when I think the experience is crisp - we have
made huge strides in the pat weeks in that regard, let's not slow down now.


Unfortunately, migrations is a piece of work that isn't yet ready. We 
are working on it, but it won't be ready for a number of weeks yet.


Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

A cautionary tale - mgo asserts

[Tim Penhey]

Hi folks,

tl;dr: not use structs in transaction asserts

I have spent the last two days looking at this bug:
https://bugs.launchpad.net/juju-core/+bug/1537585

It was one where the instance poller got itself in a knot, and couldn't 
get out. Transaction asserts were failing, and often hard to reproduce.


Nate spent last week on it, and I pulled Menno in for his mgo expertise 
yesterday and today because I was getting stumped.


Here is what was happening:

The machine document was being read, and we were wanting to update the 
preferred public and private addresses. An assert was added to make sure 
that the value was either what we expected it to be (from reading the 
doc), or unset. Now, really, it wouldn't be unset, but that is a 
different story.


The problem is that the assert was failing. Logging was added to check 
what the db had, and what the asserts were transformed to with the multi 
model layer. All the asserts matched. The data was there, everything was 
good, except it wasn't.


This morning, I pulled the data out with a bson.D, and it became readily 
apparent that the address structure ordering was different in the times 
it failed. Now normally we are guaranteed ordering. When the struct is 
marshalled to bson, it is as a bson.D with the order defined in by the 
order of the members.  However when using the struct in the assert, it 
was matching different orders, so the assert failed.


Now, if mgo is not under load, the transaction can be performed 
immediately, so the values stored use the bson.D.


However, when under load, the whole transaction may be stashed 
temporarily in a collection, and then loaded for execution at a later 
date. Reading the operations back out of the collection end up in a 
bson.M, and it loses all ordering.  The ordering doesn't matter when 
unmarshalling back into a struct, but it sure does matter when asserting.


The solution is to not use a field struct equality, even though it is 
easy to write, but to use the dotted field notation to check the 
embedded field values.


Yet another thing to add to the list of things to check when doing reviews.

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Fixing "juju help commands"

[Tim Penhey]

On 25/05/16 00:12, Marco Ceppi wrote:

Even if you don't expect people to run them, hidding them seems awkward.
Better to simply educate with good help output about what the command
does and when/why use the command.


Was thinking, perhaps it would be better to have a feature flag to use 
the "hidden" commands instead of the ability to hide commands.


If you set the feature flag, you get the additional commands, and they 
show up in help etc.  That way a way to get users to run them could be 
something like:


  JUJU_FEATURE_FLAGS=dev-debug juju dump-model

or something like that.

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

list, show and get (was Re: Fixing "juju help commands")

[Tim Penhey]

On 25/05/16 01:49, Nate Finch wrote:

While we're on the subject... we have list-foos, show-foos, and
get-foo... and I honestly cannot remember when to use which. Can we just
standardize on one, so no one has to remember if it's list-models or
show-models or show-model-config or get-model-config?  They all mean the
same thing, let's just pick one.


'list-foos' which is to be an alias for 'foos' returns a list of foos.
'show-foo' shows information about a single 'foo'
'get-foo' gets configuration about a particular 'foo'

This is consistent across all our foos now.

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Fixing "juju help commands"

[Tim Penhey]

Hidden would be a primitive defined in the github.com/juju/cmd package, 
whereas the "blockable" consideration is an implementation detail of 
juju commands.


I don't think we should specify them the same way.

However, that doesn't mean that Juju shouldn't have a base blockable 
command which the commands that could be blocked would use.  Just a thought.


Tim

On 24/05/16 16:31, Anastasia Macmood wrote:

Tim

If you are adding the ability to specify commands as "hidden", we may
greatly benefit from specifying commands as "blockable" too

Sincerely Yours,

Anastasia

On 24/05/16 14:19, Tim Penhey wrote:

Hi folks,

More from the "let's fix the output" school of thought.

One thing that has bugged me about 'juju help commands' was the
repeated mentions of "alias for ".

I propose that we don't show aliases by default, and allow the user to
task for them.

Also, the supercommand describe commands method was written before I
knew about the tabular output, so some code could be cleaned up there.

Proposal:

juju help commands
   - shows the commands that are neither aliases, nor hidden
juju help commands --alias
   - shows either just the aliases, or everything including aliases
juju help commands --all
   - shows basic commands, aliases and hidden commands.

I'd like to add the ability to say a command is hidden so it doesn't
show up in the commands list. The purpose for these could be debugging
assisting type functions, like "dump-model" or "debug-controller" (two
commands that don't yet exist).

Thoughts?

Tim





--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Fixing "juju help commands"

[Tim Penhey]

Hi folks,

More from the "let's fix the output" school of thought.

One thing that has bugged me about 'juju help commands' was the repeated 
mentions of "alias for ".


I propose that we don't show aliases by default, and allow the user to 
task for them.


Also, the supercommand describe commands method was written before I 
knew about the tabular output, so some code could be cleaned up there.


Proposal:

juju help commands
  - shows the commands that are neither aliases, nor hidden
juju help commands --alias
  - shows either just the aliases, or everything including aliases
juju help commands --all
  - shows basic commands, aliases and hidden commands.

I'd like to add the ability to say a command is hidden so it doesn't 
show up in the commands list. The purpose for these could be debugging 
assisting type functions, like "dump-model" or "debug-controller" (two 
commands that don't yet exist).


Thoughts?

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Quick win - juju check

[Tim Penhey]

Hi folks,

We talked quite a bit in Vancouver about quick wins. Things we could get 
into Juju that are simple to write that add quick value.


There was also talk about "juju check".

The idea of this command is to surface things that should really be 
looked at.


I was thinking that we could get something in place, and iterate, rather 
than going for a gold plated command to start with.


Using the migration export command, we could easily do cross-entity checks.

The things that leap to mind are:
 * machines that only host units that don't have any units
 * applications (nee services) that have no units
 * machines that have been pending for more than X minutes
 * perhaps surfacing anything that is in an error state?

Thoughts?

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: adding unit tests that take a long time

[Tim Penhey]

My 2c just so people know where I stand.

I'm not in favour of skipping any tests by default.

If people want to add flags so they can run subsets of tests, all credit
to them, but I feel that it is just test noise.

Making it too easy to skip long "unit" tests makes the problem too easy
to ignore IMO.

That is all.
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: auto-upgrading models

[Tim Penhey]

On 26/04/16 17:44, John Meinel wrote:
> How do we get the tools on the new controller? Since we did a fresh
> bootstrap it won't have the old tools cached. Is part of the pre-flight
> migration  checks ensuring that the new controller can get the right
> version of tools? How does it work if someone used --upload-tools on the
> old host? Is there a "migrate and upgrade" mechanism so you can make
> forward progress?
> 
> John
> =:->

The tools for the model are sent to the new controller in one of the
phases of migration.

The team has disallowed the ability to upload tools for a hosted model.

We are not considering a migrate and upgrade at this stage.

Tim


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: auto-upgrading models

[Tim Penhey]

On 26/04/16 14:06, Rick Harding wrote:
> The key thing is that soon our path on upgrading models is via model
> migration. We should keep that in mind as we think of how best to lead
> the user to 'just do the right thing'

Yes and No.

A hosted model will never be allowed to upgrade beyond the controller
that it is running on.

Due to the high potential risk of upgrading the controller and
potentially breaking multiple deployed models, migration is the intended
process to ALLOW upgrades.

The migration of a model from one controller to another will not upgrade
the hosted model itself.

Consider this scenario:

Controller-1 has 20 hosted models. The controller and the models are
running Juju 2.2.0.

The new hotness of Juju 2.3 has been released and has many new bells and
whistles.

In order for the hosted environments to upgrade to Juju 2.3, they must
be on a controller running Juju 2.3. Since we don't want to upgrade
controller-1, we bootstrap a new controller, "controller-2" with Juju 2.3.

We can then migration the models from controller-1 to controller-2 on an
as needed basis. The models, once on controller-2 will hopefully then
inform the users of that model (through status) that an upgrade is
possible. The users can then upgrade the model at their leisure.

Tim

> 
> 
> On Mon, Apr 25, 2016, 2:38 PM Andrew Wilkins
> mailto:andrew.wilk...@canonical.com>> wrote:
> 
> On Sat, Apr 23, 2016 at 4:15 AM Eric Snow  > wrote:
> 
> In a recent bug I was working on the issue of auto-upgrading models
> came up.  I also ran into this personally not too long ago.
> Currently, running "juju upgrade-juju -m admin --upload-tools"[1]
> upgrades the admin model to a new version.  To set the version
> of any
> other model to the uploaded one, you must do so separately
> afterward,
> e.g. "juju upgrade-juju -m default 2.0-rc1.3". [2]
> 
> The fact that you must upgrade the non-admin model separately is
> something new with multi-model support.  Perhaps it is only
> something
> that will throw off folks that have relied on --upload-tools
> previously and perhaps it is something that we'll just adjust to.
> However, I wanted to bring the matter up here and identify potential
> courses of action (not all mutually exclusive):
> 
> 1. do nothing (rely on users to know to always upgrade juju
> per-model)
> 2. clearly point this out in the documentation
> 3. add a note in the output of "juju upgrade-juju --upload-tools"
> reminding admins to manually upgrade each model
> 4. make the "juju is out-of-date" warnings also show up for models
> relative to the controller
> 5. auto-upgrade models when the controller is upgraded
> 6. auto-upgrade but have a flag to not auto-upgrade
> 7. have a flag to auto-upgrade
> 
> 
> Whatever we do, #2 should never be optional.
> 
> I would like us to have all of #2, #3, #4, #7. For #3, we could say
> "upgrade each model ... or run `juju
> upgrade-juju --all-models `".
> 
> I don't think this should be restricted to "--upload-tools", but
> rather just upgrading the admin model in general.
> 
> Cheers,
> Andrew
> 
> FWIW, I don't consider #1 or #5 to be acceptable options.  I'm
> on the
> fence about #6; it aligns with what I expect would be a better
> default
> experience but hesitate to make unrequested changes of that sort by
> default.  So #7 might be more appropriate if the consequences of #6
> would be too risky.  Regardless, I do think the user experience of
> upgrade-juju could be improved.
> 
> Thoughts?
> 
> -eric
> 
> 
> [1] You can no longer upload tools to any other model than admin.
> [2] Thankfully, due to some recent work by axw the new version is
> *available* to all models.
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
> 
> 


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Short lived fork of master

[Tim Penhey]

Hi folks,

Since master is blocked while we get the release organised, and we don't
want to have a bunch of branches backed up, we created a branch on
github.com/juju/juju called "next".

Please target reviewed work to this and land.

When master is unblocked I expect that we'll just merge this entire
branch in and let CI do it's thing.

The reason for this and not branching for the release is time pressure
around CI runs, and the current focus on master CI runs.

I expect that as we move forwards, we'll branch for the release and
leave master open.

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: New juju in ubuntu

[Tim Penhey]

We could probably set an environment variable for the plugin called
JUJU_BIN that is the juju that invoked it.

Wouldn't be too hard.

Tim

On 07/04/16 17:25, Stuart Bishop wrote:
> On 7 April 2016 at 03:55, Marco Ceppi  wrote:
>>
>> On Wed, Apr 6, 2016 at 10:07 AM Stuart Bishop 
>> wrote:
>>>
>>> On 5 April 2016 at 23:35, Martin Packman 
>>> wrote:
>>>
 The challenge here is we want Juju 2.0 and all the new functionality
 to be the default on release, but not break our existing users who
 have working Juju 1.X environments and no deployment upgrade path yet.
 So, versions 1 and 2 have to be co-installable, and when upgrading to
 xenial users should get the new version without their existing working
 juju being removed.

 There are several ways to accomplish that, but based on feedback from
 the release team, we switched from using update-alternatives to having
 'juju' on xenial always be 2.0, and exposing the 1.X client via a
 'juju-1' binary wrapper. Existing scripts can either be changed to use
 the new name, or add the version-specific binaries directory
 '/var/lib/juju-1.25/bin' to the path.
>>>
>>> How do our plugins know what version of juju is in play? Can they
>>> assume that the 'juju' binary found on the path is the juju that
>>> invoked the plugin, or is there some other way to tell using
>>> environment variables or such? Or will all the juju plugins just fail
>>> if they are invoked from the non-default juju version?
>>
>>
>> You can invoke `juju version` from within the plugin and parse the output.
>> That's what I've been doing when I need to distinguish functionality.
> 
> That seems fine if you are invoking the plugin from the default
> unnumbered 'juju'. But running 'juju2 wait' will mean that juju-wait
> will be executing juju 1.x commands and fail. And conversely running
> 'juju1 wait' will invoke juju 2.x and probably fail.
> 
> I think the plugin API needs to be extended to support allowing
> multiple juju versions to coexist. An environment variable would do
> the trick but require every plugin to be fixed. Altering $PATH so
> 'juju' runs the correct juju would allow existing plugins to run
> unmodified (the bulk of them will work with both juju1 and juju2,
> since the cli is similar enough that many plugins will work
> unmodified.
> 


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Unable to kill-controller

[Tim Penhey]

On 06/04/16 23:13, Nick Veitch wrote:
> Sure, I am just concerned about a proliferation of commands to do the
> same (ultimately) task
> 
> destroy-controller

The most correct way to take down a controller.

> kill-controller

The OMG it is broken, please do as much as you can and I know I'm going
to have to manually check any resources left around that it couldn't
clean up.

> forget/purge-controller

Remove local references to the controller.


Not really the same things at all.

Tim


> 
> 
> 
> On 6 April 2016 at 11:59, Horacio Duran  > wrote:
> 
> The issue I see with that approach is that in that case
> kill-controller might be doing less than you expect instead of more,
> suppose the controller is having transient issues and kill
> controller cannot reach the cloud for deletion, this would forget
> the controller and leave it in the cloud, forget-controller instead
> tells us very clearly what is going to happen, the change is going
> to be local and not affect the controller.
> My 2c
> 
> 
> On Wednesday, 6 April 2016, Nick Veitch  > wrote:
> 
> just my tuppence
> 
> instead of having another command, can't we just add this as an
> option to kill-controller?
> 
> juju kill-controller --cleanup 
> 
> 
> 
> On 6 April 2016 at 11:05, Horacio Duran
>  wrote:
> 
> 
> I might be biased by years of apt-get but purge makes me
> think that you are going to do what kill is supposed to do,
> forget sound more aligned whit what you are really aiming to.
> 
> On Wednesday, 6 April 2016, Andrew Wilkins
>  wrote:
> 
> On Tue, Apr 5, 2016 at 2:29 AM Cheryl Jennings
>  wrote:
> 
> Relevant bug:
>  https://bugs.launchpad.net/juju-core/+bug/1553059
> 
> We should provide a way to clean up controllers
> without making the user manually edit juju's files.
> 
> 
> Unless anyone objects, or has a better spelling, I will
> be adding a command to do this:
> 
> juju purge-controller 
> 
> The command will require a "-y" or prompt for
> confirmation, like kill-controller. It will not attempt
> to destroy the controller, it will just remove the
> details of it from the client.
> 
> (Alternative suggestion for spelling: "juju
> forget-controller". Purge-controller may suggest that
> we're purging a controller of its contents, rather than
> purging the controller from the client?)
> 
> Cheers,
> Andrew
> 
> On Mon, Apr 4, 2016 at 7:05 AM, Nate Finch
>  wrote:
> 
> This just happened to me, too.  Kill-controller
> needs to work if at all possible.  That's the
> whole point.  And yes, users may not hit
> specific problems, but devs do, and that wastes
> our time trying to figure out how to manually
> clean up the garbage.
> 
> On Mon, Apr 4, 2016 at 8:33 AM Rick Harding
>  wrote:
> 
> On Sun, Apr 3, 2016 at 6:56 PM Andrew
> Wilkins  wrote:
> 
> In a non-beta release we would make sure
> that the config changes aren't backwards
> incompatible.
> 
> 
> I think this is the key thing. I think that
> kill-controller is an exception to this
> rule. I think we should always at least give
> the user the ability to remove their stuff
> and start over with the new alpha/beta/rc
> release. I'd like to ask us to explore
> making kill-controller an exception to this
> policy and that if tests prove we can't
> bootstrap on one beta and kill with trunk
> that it's a blocking bug for us.  
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at

Re: Move provider implementations under their related projects.

[Tim Penhey]

I'm a very strong -1 on this idea.

Providers are Juju specific, and the other libraries should focus on
exposing a useful Go API.

Tim

On 25/03/16 10:12, Eric Snow wrote:
> Perhaps we should move the implementation of some providers over to
> the repos for the projects with which the providers interact (and then
> just import them in providers/all).  Then those providers would be
> more likely to stay up-to-date in the face of changes in the project,
> particularly backward-incompatible ones.  For example:
> 
> * provider/maas -> github.com/juju/gomaasapi/maasprovider
> * provider/lxd -> github.com/lxc/lxd/lxdprovider
> * ...
> 
> or something like that.  It's not a perfect solution nor a complete
> one, but it should help.
> 
> -eric
> 


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Go 1.6 is now in trusty-proposed

[Tim Penhey]

Awesome news Michael.

Thank you for all your work on this.

Tim

On 24/03/16 19:03, Michael Hudson-Doyle wrote:
> Hi,
> 
> As of a few minutes ago, there is now a golang-1.6 package in
> trusty-proposed:
> https://launchpad.net/ubuntu/trusty/+source/golang-1.6 (thanks for the
> review and copy, Steve).
> 
> One difference between this and the package I prepared earlier is that
> it does not install /usr/bin/go but rather /usr/lib/go-1.6/bin/go so
> Makefiles and such will need to be adjusted to invoke that directly or
> put /usr/lib/go-1.6/bin on $PATH or whatever. (This also means it can
> be installed alongside the golang packages that are already in
> trusty).
> 
> Cheers,
> mwh
> (Hoping that we can now really properly ignore gccgo-4.9 ppc64el bugs!)
> 
> On 17 February 2016 at 07:58, Michael Hudson-Doyle
>  wrote:
>> I have approval for the idea but also decided to wait for 1.6 and upload
>> that instead. I'm also on leave currently so hopefully this can all happen
>> in early March.
>>
>> Cheers,
>> mwh
>>
>> On 17/02/2016 1:17 am, "John Meinel"  wrote:
>>>
>>> To start with, thanks for working on this. However, doesn't this also
>>> require changing the CI builds to use your ppa?
>>>
>>> What is the current state of this? I was just looking around and noticed
>>> golang1.5-go isn't in anything specific for Trusty that I can see. I realize
>>> if its going into an SRU it requires a fair amount of negotiation with other
>>> teams, so I'm not  surprised to see it take a while. I just wanted to check
>>> how it was going.
>>>
>>> Thanks,
>>>
>>> John
>>> =:->
>>>
>>> On Mon, Jan 18, 2016 at 7:32 AM, Michael Hudson-Doyle
>>>  wrote:

 Hi all,

 As part of the plan for getting Go 1.5 into trusty (see here
 https://wiki.ubuntu.com/MichaelHudsonDoyle/Go15InTrusty) I've built
 packages (called golang1.5-go rather than golang-go) for trusty in my
 ppa:

 https://launchpad.net/~mwhudson/+archive/ubuntu/go15-trusty/+packages

 (assuming 3:1.5.3-0ubuntu4 actually builds... I seem to be having a
 "make stupid packaging mistakes" day)

 I'll write up a SRU bug to start the process of getting this into
 trusty tomorrow but before it does end up in trusty it would seem like
 a good idea to run the CI tests using juju-core packages built with
 this version of the go compiler. Is that something that's feasible to
 arrange

 The only packaging requirement should be to change the build-depends
 to be on golang1.5-go rather than golang-go or gccgo.

 Cheers,
 mwh

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>>
>>
> 


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

model migration now behind a feature flag

[Tim Penhey]

Hi folks,

Since the 'juju migrate' command does not currently finish, we have put
it behind a feature flag.

Don't bother trying it yet, it doesn't work.

We'll let you know when it is functional, and for what models it will
work and under what conditions.

There will be many caveats initially.

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

model-migrations is now merged into master

[Tim Penhey]

Hi folks,

tl;dr: you might get some new test failures when changing the DB
structure, see below.

Those quick amongst you will notice that there is still a branch in
github.com/juju/juju for model-migrations. This exists to allow us to
continue landing bits while master aims for its bless for the next beta
release.

OK, so it merged, what does it do?

Well it almost migrates models, but not quite. It does however have many
of the moving parts.

** New package: core/description

This contains a database agnostic representation of the model. The aim
is to have it match every feature we have, like payloads, resources,
storage, network spaces etc, but it isn't quite there yet. The model
description manages machines, services, units, relations, status (and
history), annotations, and relation data. We will be working to get as
much additional information in the description before the full release.

** New package: migration

This package has exported methods like: ExportModel, ImportModel,
UploadBinaries, and Precheck. These are the higher level methods that
deal with migrating models between controllers.

The binaries we handle right now are charms and tools. Resources are to
come real soon now (hopefully before release).

** Database collection and document tests

There are a bunch of tests added to state, particularly in
migration_internal_test.go. These tests make assertions about the
collections that migrations know about, and the exported fields in the
mongo documents for the various collections.

If you add, rename, or remove fields from documents, the tests will fail
and you will need to deal with the migration implications of those changes.

Adding or removing a collection will also cause a test failure, until
you deal with it.

** New command

juju migrate

This will initiate a migration for the specified hosted model to the
specified controller. The actual progress of the migration will be
exposed in the status output and logs.

** New API façades

There are three new façades to deal with different aspects of migrations.

MigrationMaster - used by the migration master worker to manage the
migration from the source controller.

MigrationTarget - used by the migration master worker when
communicating to the target controller.

MigrationSlave - used by the migration slave worker which runs in each
agent in a hosted model to monitor progress of the migration and take
migration related actions.



Questions?

Tim


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

httpbakery .Client overly restrictive on body - io.ReaderSeeker

[Tim Penhey]

Hi folks,

I came across this recently while looking at sending tools and charms
between two controllers.

The local storage interface gives us an io.ReaderCloser. Unfortunately,
we can't give that to the http client for the POST method as it needs an
io.ReaderSeeker.

I'd like to challenge the need for the seek method.

Is there any way we could change the method to take an io.Reader?

Perhaps with a documentation caveat that if it implements
io.ReaderSeeker, that Seek(0,0) will be called.

Right now, I'm going through a local temp file, but I'd rather not have
to if I don't need to.

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: admin is dead, long live $USER

[Tim Penhey]

Ah yeah... it is mostly coming back to me now.

I actually have a branch lying around that does most of this.

Perhaps I should look at it again :-)

Tim

On 04/03/16 20:08, Ian Booth wrote:
> Hey Tim
> 
> The new bootstrap UX has not removed any --admin-user flag.
> I can see that the server jujud bootstrap command has an --admin-user argument
> but it appears this is never set anywhere in the cloud init scripts. Or not 
> that
> I can see. I've checked older version of the relevant files and can't see 
> where
> we've ever used this.
> 
> So maybe we have a capability to bootstrap the controller agent with a 
> specified
> admin-user but have not hooked it up yet?
> 
> On 04/03/16 08:11, Tim Penhey wrote:
>> Ah... it used to be there :-) At least it is on my feature branch, but I
>> don't think I have merged the most recent master updates that has the
>> work to re-work bootstrap for the new cloud credentials stuff.
>>
>> Tim
>>
>> On 04/03/16 10:09, Rick Harding wrote:
>>> If we do that we need to also make it configurable on bootstrap as an
>>> option.
>>>
>>> +1 overall
>>>
>>>
>>> On Thu, Mar 3, 2016, 4:07 PM Tim Penhey >> <mailto:tim.pen...@canonical.com>> wrote:
>>>
>>> Hi folks,
>>>
>>> I was thinking that with the upcoming big changes with 2.0, we should
>>> tackle a long held issue where we have the initial user called "admin".
>>>
>>> There was a request some time back that we should use the current user's
>>> name. The reason it wasn't implemented at that time was due to logging
>>> into the GUI issues. These have been resolved some time back with the
>>> multiple user support that was added.
>>>
>>> All the server side code handles the ability to define the initial user
>>> for the controller model, and we do this in all the tests, so the
>>> default test user is actually called "test-admin".
>>>
>>> I *think* that all we need to do is change the default value we use in
>>> the bootstrap command for the AdminUserName (--admin-user flag) from
>>> "admin" to something we derive from the current user.
>>>
>>> Probably worth doing now.
>>>
>>> Thoughts?
>>>
>>> Tim
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com <mailto:Juju-dev@lists.ubuntu.com>
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>
>>


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: admin is dead, long live $USER

[Tim Penhey]

Ah... it used to be there :-) At least it is on my feature branch, but I
don't think I have merged the most recent master updates that has the
work to re-work bootstrap for the new cloud credentials stuff.

Tim

On 04/03/16 10:09, Rick Harding wrote:
> If we do that we need to also make it configurable on bootstrap as an
> option.
> 
> +1 overall
> 
> 
> On Thu, Mar 3, 2016, 4:07 PM Tim Penhey  <mailto:tim.pen...@canonical.com>> wrote:
> 
> Hi folks,
> 
> I was thinking that with the upcoming big changes with 2.0, we should
> tackle a long held issue where we have the initial user called "admin".
> 
> There was a request some time back that we should use the current user's
> name. The reason it wasn't implemented at that time was due to logging
> into the GUI issues. These have been resolved some time back with the
> multiple user support that was added.
> 
> All the server side code handles the ability to define the initial user
> for the controller model, and we do this in all the tests, so the
> default test user is actually called "test-admin".
> 
> I *think* that all we need to do is change the default value we use in
> the bootstrap command for the AdminUserName (--admin-user flag) from
> "admin" to something we derive from the current user.
> 
> Probably worth doing now.
> 
> Thoughts?
> 
> Tim
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com <mailto:Juju-dev@lists.ubuntu.com>
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

admin is dead, long live $USER

[Tim Penhey]

Hi folks,

I was thinking that with the upcoming big changes with 2.0, we should
tackle a long held issue where we have the initial user called "admin".

There was a request some time back that we should use the current user's
name. The reason it wasn't implemented at that time was due to logging
into the GUI issues. These have been resolved some time back with the
multiple user support that was added.

All the server side code handles the ability to define the initial user
for the controller model, and we do this in all the tests, so the
default test user is actually called "test-admin".

I *think* that all we need to do is change the default value we use in
the bootstrap command for the AdminUserName (--admin-user flag) from
"admin" to something we derive from the current user.

Probably worth doing now.

Thoughts?

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Dependency engine in the machine agent

[Tim Penhey]

Thanks Menno. That was very helpful.

Apart from state connections, or api connections, what sort of resources
are shared between workers?

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

model-migration: the what, why, and progress

[Tim Penhey]

Hi folks,

I'm writing this to communicate what we are doing, why, and how its
going. Something I hope will prompt others to do the same.

I'm also keeping this on the internal juju list because I will be
mentioning JAAS.

What and why:

Model migration is a key component of JAAS and allows us to work with
the most up to date Juju Controllers without fear. With the new world
order where a controller can host multiple models, we are now in the
situation where upgrading a controller could have dramatic and drastic
problems for other models hosted on it should things go wrong. This is
not something we even want to have as an option where we are dealing
with models that are not ours.

Models hosted in a controller are not able to upgrade to versions of
Juju beyond that of the controller they are running on. So one of two
things needs to happen:
 1) we upgrade the controller
 2) we move the model to a newer controller

Once one of these things has happened, the hosted model can upgrade to a
new version (up to what the controller is running).

Since we have already decided that (1) is not a viable option in the
JAAS world, (2) is what we are working on.

Model migration also gives us the ability to load balance should one
controller get overloaded by moving some models to a different controller.

Progress:

Team Onyx has been focusing on model migrations for some time now, and
many pieces are falling into place.

In order to migrate a model, as a user, you need to be a "controller
admin" in both the source and target controllers. There is a CLI command
to initiate the migration "juju migrate". Status of the migration will
be reported as part of "juju status".

This starts a chain of events that will result in the model being moved
and all the agents looking at the new controller.

One of the first big changes was to change the machine agent to use the
new dependency engine for managing dependencies between the various
workers. This is needed to cleanly quiesce environments.  This work
landed into master some time ago, and we are changing the individual
workers one at a time to fit into the new engine.

Some complexities and steps are not shown to avoid too much detail.

The general process of migration goes something like this:
 0) initial checks to confirm no large pending changes or error states
   - no machines being provisioned
   - no missing agents
   - no dying or dead entities
   - there is a grace period, but if it doesn't settle down, the
migration is aborted
 1) the model is quiesced and agents enter a read only state
 2) the general state is checked again to make sure all things are quiet
and stable
 3) the model is serialized into a versioned, database agnostic, wire
format and imported into the target controller
 4) binaries used by the model, tools, charms, resources etc are added
to the target controller
 5) logs are streamed across
 6) all agents are told to check in with the new controller to confirm
access and that the state looks consistent as it would expect
 7) agents switch over to new controller
 8) documents in old controller are cleaned up

At any state prior to switching the agents over, the migration may be
aborted and the model goes back into an active mode.

Right now we are working like mad to get this working.

The model import/export is at a stage where by the end of this week we
should be able to deal with "simple" environments, where simple means
not dealing with the new features like: payloads, storage, networks, or
resources.  These extra parts will be added ASAP.

Several weeks ago I was able to do a proof of concept migration of
simple machines (this was before I got services, units and relations
going) between two controllers using a separate binary to do the export
and import and much manual-hacker-fu.

We are actively working on the controller managing the migration process.

We are making good progress I do feel that things are going to be tight.

The import/export work is in the feature branch "model-migrations".
The control aspect is in the feature branch "model-migration-control".
Moving all the environment workers to a nested dependency engine is the
feature branch "MADE-state-workers" (MADE was short hand for Machine
Agent Dependency Engine). The machine-dep-engine feature branch merged
into master back in January. The MADE-workers feature branch landed
earlier this month. Individual worker changes are now targeting master.

Each of the feature branches are independent of each other for now.

A side benefit I see of the migration work is the potential to speed up
'juju status' significantly.  The model export takes the entire db into
a structured object model, and it reads entire collections at a time, so
is much faster. I'd love to test this in real world large environments.

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Tags and object IDs

[Tim Penhey]

Here is my short answer:

  use names.UnitTag

That is all.

Tim

On 23/01/16 09:53, Nate Finch wrote:
> Working in the model layer on the server between the API and the DB. 
> Specifically in my instance, an API call comes in from a unit,
> requesting the bytes for a resource.  We want to record that this unit
> is now using the bytes from that specific revision of the resource.  I
> have a pointer to a state.Unit, and a function that takes a Resource
> metadata object and some reference to the unit, and does the actual
> transaction to the DB to store the unit's ID and the resource information.
> 
> On Fri, Jan 22, 2016 at 3:34 PM William Reade
> mailto:william.re...@canonical.com>> wrote:
> 
> Need a bit more context here. What layer are you working in?
> 
> In general terms, entity references in the API *must* use tags;
> entity references that leak out to users *must not* use tags;
> otherwise it's a matter of judgment and convenience. In state code,
> it's annoying to use tags because we've already got the globalKey
> convention; in worker code it's often justifiable if not exactly
> awesome.
> See https://github.com/juju/juju/wiki/Managing-complexity#workers
> 
> Cheers
> William
> 
> On Fri, Jan 22, 2016 at 6:02 PM, Nate Finch
> mailto:nate.fi...@canonical.com>> wrote:
> 
> I have a function that is recording which unit is using a
> specific resource.  I wrote the function to take a UnitTag,
> because that's the closest thing we have to an ID type. However,
> I and others seem to remember hearing that Tags are really only
> supposed to be used for the API. That leaves me with a problem -
> what can I pass to this function to indicate which unit I'm
> talking about?  I'd be fine passing a pointer to the unit object
> itself, but we're trying to avoid direct dependencies on state. 
> People have suggested just passing a string (presumably
> unit.Tag().String()), but then my API is too lenient - it
> appears to say "give me any string you want for an id", but what
> it really means is "give me a serialized UnitTag".
> 
> I think most places in the code just use a string for an ID, but
> this opens up the code to abuses and developer errors.
> 
> Can someone explain why tags should only be used in the API? It
> seems like the perfect type to pass around to indicate the ID of
> a specific object.
> 
> -Nate
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
> 
> 


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Defaulting tests internal to the package

[Tim Penhey]

I'm going to throw my cool-blue paint against the bike shed.

I disagree that we should change the default expectations of package tests.

By default, package tests should be external tests, small and fast, not
full stack, and parameterize from above rather than patch.

However, I also see much value in internal unit tests for exactly the
same reason as Nate and Roger. It gives me confidence in my
implementation. These tests have nothing to do really with the exposed
interface of the package, but with the current implementation.

This whole issue is close to my heart right now as I deal with model
migrations. Many of the tests I am writing are internal tests.

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

New feature branch MADE-workers

[Tim Penhey]

Hi all,

Since we are holding back the machine-dep-engine feature branch from
master until the first 2.0 alpha, we needed a place to merge in the
worker changes without impacting the blessedness of the
machine-dep-engine branch.

This feature branch should just hold the related manifold worker changes
over and above the machine-dep-engine changes.

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Be very careful in assumptions in tests

[Tim Penhey]

Hi folks,

I came across an interesting bug yesterday and during investigation
found that there was a very comprehensive test that covered the situation.

The problem is that the asserts were not actually running.

The asserts were inside a loop where the expectation was that the loop
would run exactly once as a previous call returned an array of items
with one value in it.

Except it didn't.

It returned an empty list.

So, if you see something like this, a simple line like:

  c.Assert(someList, gc.HasLen, 1)

Is sufficient to have caught this particular issue.


Remember a key step when writing tests is to have the test fail first,
then make it pass.

One trick I use is to assert checks against data I know is wrong. If the
test passes then I know I have other problems.

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Reviews for merges _from_ master to feature branches

[Tim Penhey]

On 01/12/15 13:56, Ian Booth wrote:
> 
> 
> On 01/12/15 10:17, David Cheney wrote:
>> Hello,
>>
>> Why are reviewers being created for merges from master to feature
>> branches ? What purpose does this serve ?
>>
> 
> They appear because you create a github PR which triggers a reviewboard review
> to be created. I just self approve and everything gets processed soon enough.
> You still want a PR to interface with the landing bot.

+1 I also just self approve merges from master into feature branches,
and I have told others that they can do the same.

No need to review merges of master into your own feature.

Makes it go through the bot, which I like. Also best practice to just
merge in a blessed revision.

Tim


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

utils/fslock needs to DIAF

[Tim Penhey]

Hi folks,

The fslock was a mistake that I added to the codebase some time back. It
provided an overly simplistic solution to a more complex problem.

Really the filesystem shouldn't be used as a locking mechanism.

Most of the code that exists for the fslock now is working around its
deficiencies. Instead we should be looking for a better replacement.

Some "features" that were added to fslock were added to work around the
issue that the lock did not die with the process that created it, so
some mechanism was needed to determine whether the lock should be broken
or not.

What we really need is a good OS agnostic abstraction that provides the
ability to create a "named" lock, acquire the lock, release the lock,
and make sure that the lock dies when the process dies, so another
process that is waiting can acquire the lock. This way no "BreakLock"
functionality is required, nor do we need to try and do think like
remember which process owns the lock.

So...

We have three current operating systems we need to support:

Linux - Ubuntu and CentOS
MacOS - client only - bit the CLI uses a lock for the local cache
Windows

For Linux, and possibly MacOS, flock is a possibility, but can we do
better? Is there something that is better suited?

For Windows, while you can create global semaphores or mutex instances,
I'm not sure of entities that die with the process. Can people recommend
solutions?

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev