Re: Schema for Juju RPC messages

[Tim Penhey]

While at the London sprint I was toying with the idea of adding the 
ability of the rpc package to do some rudimentary initial validation.


We could get a good part of the way with relatively little effort IMO.

Using the reflect package, we can interrogate the public attributes of 
the structure we are attempting to serialize into. This gives access to 
the structure serialization tags. We could then use these tags to reject 
incoming messages if they provide any keys that are unexpected.


For example, lets look at the ApplicationDeploy params:
(had to munge some of the struct tags for the email)
type ApplicationDeploy struct {
  ApplicationName  string   `json:"application"`
  Series   string   `json:"series"`
  CharmUrl string   `json:"charm-url"`
  Channel  string   `json:"channel"`
  NumUnits int  `json:"num-units"`
  Config   map[string]string `json:"config,omitempty"`
  ConfigYAML   string`json:"config-yaml"`
  Constraints  constraints.Value  `json:"constraints"`
  Placement[]*instance.Placement
 `json:"placement,omitempty"`
  Storage  map[string]storage.Constraints
 `json:"storage,omitempty"`
  EndpointBindings map[string]string
 `json:"endpoint-bindings,omitempty"`
  Resourcesmap[string]string
 `json:"resources,omitempty"`
}

This would give us a set of valid keys:
  ("application", "series", "charm-url", "channel", "num-units",
   "config", "config-yaml", "constraints", "placement", "storage",
   "endpoint-bindings", "resources")

There would be overhead in doing the check though, because instead of 
deserializing directly into the structure, we'd deserialize into 
something like map[string]interface{} first, and do a key check, then 
only deserialize into the structure if it passed.


We could use the "omitempty" tags to mark optional args that can be missing.

Just doing this would catch the typos of keys, and would also make us be 
more strict in the "I'll just add this extra field" to existing facade 
versions, as they could then fail with older versions of the server with 
"unknown field" type errors.


Thoughts?
Tim

On 28/07/16 04:29, John A Meinel wrote:

We've had some requests from people trying to drive Juju that it would
actually be really nice if we were stricter with the messages that we
accept on our API. Specifically, as we've changed the API methods, they
had several hard-to-debug problems because they were passing a parameter
that was named incorrectly, and Juju was not giving them any indication
that something was wrong.

I put together a (very hackish) test branch, to see if I could use
JSONSchema to define our request and response messages, and give nicer
error messages (rather than silent acceptance). As I was working with
JSON, I realized the extra " and { characters really got in the way of
making a document that was readable, so I leveraged our existing YAML to
JSON conversion mechanisms to write the description in YAML and then
load the object tree into JSONSchema to validate our requests.

I did end up getting basic validation of the outer structure (just the
Request/Response message, not the contents of Parameters) functioning here:
https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec.go
You can see what some of the error messages look like here:
https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec_test.go

The actual code itself isn't useful as is, because it needs to have the
schema validation stuff pulled out into its own file, etc. But it does
give a proof-of-concept of basic message validation. I'm not super
excited by some of the error messages
(gojsonschema.ResultError.Description is very nice by itself for missing
keys, extra properties, etc, but not enough information for invalid
values, while ResultError.String() is overly verbose in the opposite way.)

I'd like to get a bit of feedback on whether people would find this
interesting, especially if we brought that all the way into the Param
structs as well. I haven't done any benchmarking to determine if this is
a large overhead or not. But the golfing community seems determined to
not do a Strict Unmarshal function, and seems to be recommending using a
Schema instead.

I'm not completely sold either way, though I do find the YAML format for
the schema description to be far more human readable than the JSON format.

Thoughts?

John
=:->





--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Small script to connect to Juju's mongo in LXD

[Menno Smits]

Small correction: http://paste.ubuntu.com/21232706/

Serves me right for cleaning it up a little before sending.

On 28 July 2016 at 15:11, Menno Smits  wrote:

> Nice. I'd suggest 2 things:
>
>- Change the mongo command line to include `--authenticationDatabase
>admin` and then change the database being connected to in the URL to "juju"
>instead of "admin". That way you get dropped straight into the juju
>database which is usually what you want.
>- Quote $PASSWORD, just in case.
>
> I have a similar thing which works for controllers on any cloud type but
> it's pretty awful. FWIW here it is: http://paste.ubuntu.com/21232100/
>
> It currently only works for xenial machines and adds a PPA to get a
> MongoDB 3.2 client onto the host. The 2.6 client which ships with Xenial
> can't connect to a MongoDB 3.2 instance (I think something was being sorted
> out for this). It wouldn't be hard to have it install install
> mongodb-clients for non-xenial machines.
>
> - Menno
>
>
>
> On 28 July 2016 at 14:46, Andrew Wilkins 
> wrote:
>
>> On Thu, Jul 28, 2016 at 12:32 AM John Meinel 
>> wrote:
>>
>>> Did you intend to attach the script to the email? It does sound like
>>> something useful. I know when we were investigating at some client sites we
>>> had a small snippet of a bash function to dig the content out of agent.conf
>>> and launch mongo with the right options. It would be nice to have that in a
>>> more official place so it doesn't get forgotten.
>>>
>>
>> Kapil wrote a plugin for inspecting Mongo:
>> https://github.com/kapilt/juju-dbinspect. It's almost certainly broken
>> in Juju 2.0. I've found it handy in the past, it'd be good to have that
>> brought up to date.
>>
>> Cheers,
>> Andrew
>>
>>
>>> John
>>> =:->
>>>
>>>
>>> On Wed, Jul 27, 2016 at 6:19 PM, Katherine Cox-Buday <
>>> katherine.cox-bu...@canonical.com> wrote:
>>>
 I frequently need to connect to Juju's Mongo instance to poke around
 and see if something I've done is having the desired effect. Back when we
 were using LXC, I had a script that would pull the password from agent.conf
 and open a shell. When we switched to LXD my script broke, and I never
 updated it. I finally got frustrated enough to modify[1] it, and thought
 others might find this useful for poking around Mongo.

 Let me know if you have any suggestions.

 --
 Katherine

 [1] - http://pastebin.ubuntu.com/21155985/

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Small script to connect to Juju's mongo in LXD

[Menno Smits]

Nice. I'd suggest 2 things:

   - Change the mongo command line to include `--authenticationDatabase
   admin` and then change the database being connected to in the URL to "juju"
   instead of "admin". That way you get dropped straight into the juju
   database which is usually what you want.
   - Quote $PASSWORD, just in case.

I have a similar thing which works for controllers on any cloud type but
it's pretty awful. FWIW here it is: http://paste.ubuntu.com/21232100/

It currently only works for xenial machines and adds a PPA to get a MongoDB
3.2 client onto the host. The 2.6 client which ships with Xenial can't
connect to a MongoDB 3.2 instance (I think something was being sorted out
for this). It wouldn't be hard to have it install install mongodb-clients
for non-xenial machines.

- Menno



On 28 July 2016 at 14:46, Andrew Wilkins 
wrote:

> On Thu, Jul 28, 2016 at 12:32 AM John Meinel 
> wrote:
>
>> Did you intend to attach the script to the email? It does sound like
>> something useful. I know when we were investigating at some client sites we
>> had a small snippet of a bash function to dig the content out of agent.conf
>> and launch mongo with the right options. It would be nice to have that in a
>> more official place so it doesn't get forgotten.
>>
>
> Kapil wrote a plugin for inspecting Mongo:
> https://github.com/kapilt/juju-dbinspect. It's almost certainly broken in
> Juju 2.0. I've found it handy in the past, it'd be good to have that
> brought up to date.
>
> Cheers,
> Andrew
>
>
>> John
>> =:->
>>
>>
>> On Wed, Jul 27, 2016 at 6:19 PM, Katherine Cox-Buday <
>> katherine.cox-bu...@canonical.com> wrote:
>>
>>> I frequently need to connect to Juju's Mongo instance to poke around and
>>> see if something I've done is having the desired effect. Back when we were
>>> using LXC, I had a script that would pull the password from agent.conf and
>>> open a shell. When we switched to LXD my script broke, and I never updated
>>> it. I finally got frustrated enough to modify[1] it, and thought others
>>> might find this useful for poking around Mongo.
>>>
>>> Let me know if you have any suggestions.
>>>
>>> --
>>> Katherine
>>>
>>> [1] - http://pastebin.ubuntu.com/21155985/
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Small script to connect to Juju's mongo in LXD

[Anastasia Macmood]

We have started putting useful information and similar handy scripts on
juju/wiki [1].

It'd be nice to grow this knowledge base and have this information in
one central area - easier to maintain/keep current.

Mailing lists are a nice way of communicating new and interesting
information but can be tedious to search through when something is needed.

[1]

https://github.com/juju/juju/wiki/Debugging-Juju


On 28/07/16 12:46, Andrew Wilkins wrote:
> On Thu, Jul 28, 2016 at 12:32 AM John Meinel  > wrote:
>
> Did you intend to attach the script to the email? It does sound
> like something useful. I know when we were investigating at some
> client sites we had a small snippet of a bash function to dig the
> content out of agent.conf and launch mongo with the right options.
> It would be nice to have that in a more official place so it
> doesn't get forgotten.
>
>
> Kapil wrote a plugin for inspecting
> Mongo: https://github.com/kapilt/juju-dbinspect. It's almost certainly
> broken in Juju 2.0. I've found it handy in the past, it'd be good to
> have that brought up to date.
>
> Cheers,
> Andrew
>  
>
> John
> =:->
>
>
> On Wed, Jul 27, 2016 at 6:19 PM, Katherine Cox-Buday
>  > wrote:
>
> I frequently need to connect to Juju's Mongo instance to poke
> around and see if something I've done is having the desired
> effect. Back when we were using LXC, I had a script that would
> pull the password from agent.conf and open a shell. When we
> switched to LXD my script broke, and I never updated it. I
> finally got frustrated enough to modify[1] it, and thought
> others might find this useful for poking around Mongo.
>
> Let me know if you have any suggestions.
>
> --
> Katherine
>
> [1] - http://pastebin.ubuntu.com/21155985/
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Small script to connect to Juju's mongo in LXD

[Andrew Wilkins]

On Thu, Jul 28, 2016 at 12:32 AM John Meinel  wrote:

> Did you intend to attach the script to the email? It does sound like
> something useful. I know when we were investigating at some client sites we
> had a small snippet of a bash function to dig the content out of agent.conf
> and launch mongo with the right options. It would be nice to have that in a
> more official place so it doesn't get forgotten.
>

Kapil wrote a plugin for inspecting Mongo:
https://github.com/kapilt/juju-dbinspect. It's almost certainly broken in
Juju 2.0. I've found it handy in the past, it'd be good to have that
brought up to date.

Cheers,
Andrew


> John
> =:->
>
>
> On Wed, Jul 27, 2016 at 6:19 PM, Katherine Cox-Buday <
> katherine.cox-bu...@canonical.com> wrote:
>
>> I frequently need to connect to Juju's Mongo instance to poke around and
>> see if something I've done is having the desired effect. Back when we were
>> using LXC, I had a script that would pull the password from agent.conf and
>> open a shell. When we switched to LXD my script broke, and I never updated
>> it. I finally got frustrated enough to modify[1] it, and thought others
>> might find this useful for poking around Mongo.
>>
>> Let me know if you have any suggestions.
>>
>> --
>> Katherine
>>
>> [1] - http://pastebin.ubuntu.com/21155985/
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Schema for Juju RPC messages

[Rick Harding]

>
> That being said, any API client that requires three API calls to login
> should be beat over the head with a fail stick, but there are many cases
> where performing some workflow might require multiple api calls. The API
> clients should work out a single function of "doSomethingAwesome(param1,
> pararm1)" when it's in fact performing multiple API steps because it makes
> sense for end users, or I guess developers in this case.
>

Just to flesh this out with an obvious example that came to mind. There are
three "addCharm"-ish API calls.

addCharm [1]
addCharmWithAuthorization [2]
addLocalCharm [3]

Writing an API client in Python or other languages I'd want to provide a
single addCharm method that just made the right API call to Juju based on
the arguments provided. If you provided a macaroon  from a current logged
in session, I'd make a addCharmWithAuthorization. If you passed a local
'./' path as the charm url then I'd call out with addLocalCharm.
There's no need to push the complexity of those three different calls to
the end user of the API client.

My understanding is that we need three distinct calls in the API because of
Go and the way that these things need to be declared/defined. However,
pushing that down into API clients is pushing implementation details (Go in
this case) down into clients in other languages where this would not be so
required.

1: https://godoc.org/github.com/juju/juju/api#Client.AddCharm
2:
https://godoc.org/github.com/juju/juju/api#Client.AddCharmWithAuthorization
3: https://godoc.org/github.com/juju/juju/api#Client.AddLocalCharm
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Schema for Juju RPC messages

[Rick Harding]

On Wed, Jul 27, 2016 at 3:18 PM Katherine Cox-Buday <
katherine.cox-bu...@canonical.com> wrote:

> Good point, Mark. I agree that it's difficult to make an auto-generated
> client idiomatic/nice to use. What I like to do is use the schema to
> auto-generate the boilerplate, and then wrap that with a small shim that is
> more crisp.
>
> Do you have an opinion on this approach?


I think that using a schema to help generate useful documentation and
improved error messaging/API input validation when the API isn't used
correctly (as in the case of what drove John to chase this down originally
[1]) is great and awesome. It keeps docs in sync with reality and forces us
to focus on the naming and thoughtfulness of the API since future work is
generated from the actual code. We don't get to fake it.

However, an API client in any language should never be auto generated.
Languages are unque and value different things. An API client is all about
taking a given API, and mapping it to the ideas and constructs a language
values. For instance, a python client for an API would not expect to expose
to users of that client HTTP error codes. They'd expect those to get mapped
to proper exceptions that mean something in Python to the user. A
JavaScript API would expect to have the API mapped to data types and other
constraints that JS present that won't make sense for other languages.

On top of that, clients are meant to be used by users farther down the
pipe. The API calls themselves are meant to be used by a few client
authors. Those client authors are intended to provide abstractions,
shortcuts, and workflows that may span multiple API calls to the end user.
Imagine an API that takes three steps to login. API clients should work
hard to map those to a single API client function call, internalizing the
complexity of the API so that the broad user base of the client does't have
to deal with that complexity.

There's no way to auto generate a client that does these helpful and
necessary simplifications across a given API automatically.

That being said, any API client that requires three API calls to login
should be beat over the head with a fail stick, but there are many cases
where performing some workflow might require multiple api calls. The API
clients should work out a single function of "doSomethingAwesome(param1,
pararm1)" when it's in fact performing multiple API steps because it makes
sense for end users, or I guess developers in this case.

1: https://bugs.launchpad.net/juju-core/+bug/1585289
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Schema for Juju RPC messages

[Katherine Cox-Buday]

Mark Shuttleworth  writes:

> On 27/07/16 19:43, Reed O'Brien wrote:
>
> +1
> 
> 
> On Wed, Jul 27, 2016 at 9:52 AM, Katherine Cox-Buday
>  wrote:
> 
> 
> I am a huge fan of publishing schema's and then generating
> SDKs and documentation websites off of them. It brings your
> API closer to language agnostic, removes the burden on us
> having to hand-craft both our API client and our documentation
> site, and makes validation implicit. I'm not up on what the
> best tools for doing this are, but Swagger[1] seems to be
> popular.
>
> Schemas are excellent for test suites and tool validation, but
> auto-generated bindings are *always* ugly. Please don't make that
> plan A for any language you care about. From a house perspective, Go,
> JS and Python need designed and human-curated bindings that are
> idiomatic for coders who love those languages.
>
> Mark
>

Good point, Mark. I agree that it's difficult to make an auto-generated client 
idiomatic/nice to use. What I like to do is use the schema to auto-generate the 
boilerplate, and then wrap that with a small shim that is more crisp.

Do you have an opinion on this approach?

-- 
Katherine

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Schema for Juju RPC messages

[Mark Shuttleworth]

On 27/07/16 19:43, Reed O'Brien wrote:
> +1
>
> On Wed, Jul 27, 2016 at 9:52 AM, Katherine Cox-Buday
>  > wrote:
>
>
> I am a huge fan of publishing schema's and then generating SDKs
> and documentation websites off of them. It brings your API closer
> to language agnostic, removes the burden on us having to
> hand-craft both our API client and our documentation site, and
> makes validation implicit. I'm not up on what the best tools for
> doing this are, but Swagger[1] seems to be popular.
>

Schemas are excellent for test suites and tool validation, but
auto-generated bindings are *always* ugly. Please don't make that
plan A for any language you care about. From a house perspective, Go, JS
and Python need designed and human-curated bindings that are idiomatic
for coders who love those languages.

Mark
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Schema for Juju RPC messages

[Reed O'Brien]

+1

On Wed, Jul 27, 2016 at 9:52 AM, Katherine Cox-Buday <
katherine.cox-bu...@canonical.com> wrote:

>
> I am a huge fan of publishing schema's and then generating SDKs and
> documentation websites off of them. It brings your API closer to language
> agnostic, removes the burden on us having to hand-craft both our API client
> and our documentation site, and makes validation implicit. I'm not up on
> what the best tools for doing this are, but Swagger[1] seems to be popular.
>
> [1] - http://swagger.io/
>
> John A Meinel  writes:
>
> > We've had some requests from people trying to drive Juju that it would
> > actually be really nice if we were stricter with the messages that we
> > accept on our API. Specifically, as we've changed the API methods,
> > they had several hard-to-debug problems because they were passing a
> > parameter that was named incorrectly, and Juju was not giving them any
> > indication that something was wrong.
> >
> > I put together a (very hackish) test branch, to see if I could use
> > JSONSchema to define our request and response messages, and give nicer
> > error messages (rather than silent acceptance). As I was working with
> > JSON, I realized the extra " and { characters really got in the way of
> > making a document that was readable, so I leveraged our existing YAML
> > to JSON conversion mechanisms to write the description in YAML and
> > then load the object tree into JSONSchema to validate our requests.
> >
> > I did end up getting basic validation of the outer structure (just the
> > Request/Response message, not the contents of Parameters) functioning
> > here:
> >
> https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec.go
> > You can see what some of the error messages look like here:
> >
> https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec_test.go
> >
> > The actual code itself isn't useful as is, because it needs to have
> > the schema validation stuff pulled out into its own file, etc. But it
> > does give a proof-of-concept of basic message validation. I'm not
> > super excited by some of the error messages
> > (gojsonschema.ResultError.Description is very nice by itself for
> > missing keys, extra properties, etc, but not enough information for
> > invalid values, while ResultError.String() is overly verbose in the
> > opposite way.)
> >
> > I'd like to get a bit of feedback on whether people would find this
> > interesting, especially if we brought that all the way into the Param
> > structs as well. I haven't done any benchmarking to determine if this
> > is a large overhead or not. But the golfing community seems determined
> > to not do a Strict Unmarshal function, and seems to be recommending
> > using a Schema instead.
> >
> > I'm not completely sold either way, though I do find the YAML format
> > for the schema description to be far more human readable than the JSON
> > format.
> >
> > Thoughts?
> >
> > John
> > =:->
>
> --
> Katherine
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>



-- 
Reed O'Brien
✉ reed.obr...@canonical.com
✆ 415-562-6797
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Interactive bootstrap has landed

[Andreas Hasenack]

On Wed, Jul 27, 2016 at 2:19 PM, Nate Finch 
wrote:

> If you run "juju bootstrap" or "juju bootstrap --upload-tools", you will
> now be placed into an interactive mode that will ask you for details about
> the cloud you'd like to bootstrap on.   Currently this is limited to clouds
> juju already knows about and only with credentials juju already knows
> about.  We intend to expand this to let you
>

Does this kick in only when run exactly like that, without further
parameters?
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Schema for Juju RPC messages

[Katherine Cox-Buday]

I am a huge fan of publishing schema's and then generating SDKs and 
documentation websites off of them. It brings your API closer to language 
agnostic, removes the burden on us having to hand-craft both our API client and 
our documentation site, and makes validation implicit. I'm not up on what the 
best tools for doing this are, but Swagger[1] seems to be popular.

[1] - http://swagger.io/

John A Meinel  writes:

> We've had some requests from people trying to drive Juju that it would
> actually be really nice if we were stricter with the messages that we
> accept on our API. Specifically, as we've changed the API methods,
> they had several hard-to-debug problems because they were passing a
> parameter that was named incorrectly, and Juju was not giving them any
> indication that something was wrong.
>
> I put together a (very hackish) test branch, to see if I could use
> JSONSchema to define our request and response messages, and give nicer
> error messages (rather than silent acceptance). As I was working with
> JSON, I realized the extra " and { characters really got in the way of
> making a document that was readable, so I leveraged our existing YAML
> to JSON conversion mechanisms to write the description in YAML and
> then load the object tree into JSONSchema to validate our requests.
>
> I did end up getting basic validation of the outer structure (just the
> Request/Response message, not the contents of Parameters) functioning
> here:
> https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec.go
> You can see what some of the error messages look like here:
> https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec_test.go
>
> The actual code itself isn't useful as is, because it needs to have
> the schema validation stuff pulled out into its own file, etc. But it
> does give a proof-of-concept of basic message validation. I'm not
> super excited by some of the error messages
> (gojsonschema.ResultError.Description is very nice by itself for
> missing keys, extra properties, etc, but not enough information for
> invalid values, while ResultError.String() is overly verbose in the
> opposite way.)
>
> I'd like to get a bit of feedback on whether people would find this
> interesting, especially if we brought that all the way into the Param
> structs as well. I haven't done any benchmarking to determine if this
> is a large overhead or not. But the golfing community seems determined
> to not do a Strict Unmarshal function, and seems to be recommending
> using a Schema instead.
>
> I'm not completely sold either way, though I do find the YAML format
> for the schema description to be far more human readable than the JSON
> format.
>
> Thoughts?
>
> John
> =:->

-- 
Katherine

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Small script to connect to Juju's mongo in LXD

[John Meinel]

Ah sorry. My email client treats everything after your signature as "quiet"
values so it didn't jump out at me. I see it now.

John
=:->

On Jul 27, 2016 18:45, "Katherine Cox-Buday" <
katherine.cox-bu...@canonical.com> wrote:

> It's in the pastebin I linked.
>
> But yes, I find it extremely useful for debugging. I could see how having
> it handy at client sites would be super useful.
>
> John Meinel  writes:
>
> > Did you intend to attach the script to the email? It does sound like
> > something useful. I know when we were investigating at some client
> > sites we had a small snippet of a bash function to dig the content out
> > of agent.conf and launch mongo with the right options. It would be
> > nice to have that in a more official place so it doesn't get
> > forgotten.
> >
> > John
> > =:->
> >
> > On Wed, Jul 27, 2016 at 6:19 PM, Katherine Cox-Buday
> >  wrote:
> >
> > I frequently need to connect to Juju's Mongo instance to poke
> > around and see if something I've done is having the desired
> > effect. Back when we were using LXC, I had a script that would
> > pull the password from agent.conf and open a shell. When we
> > switched to LXD my script broke, and I never updated it. I finally
> > got frustrated enough to modify[1] it, and thought others might
> > find this useful for poking around Mongo.
> >
> > Let me know if you have any suggestions.
> >
> > --
> > Katherine
> >
> > [1] - http://pastebin.ubuntu.com/21155985/
> >
> > --
> > Juju-dev mailing list
> > Juju-dev@lists.ubuntu.com
> > Modify settings or unsubscribe at:
> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
> >
> >
>
> --
> Katherine
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Small script to connect to Juju's mongo in LXD

[Katherine Cox-Buday]

It's in the pastebin I linked.

But yes, I find it extremely useful for debugging. I could see how having it 
handy at client sites would be super useful.

John Meinel  writes:

> Did you intend to attach the script to the email? It does sound like
> something useful. I know when we were investigating at some client
> sites we had a small snippet of a bash function to dig the content out
> of agent.conf and launch mongo with the right options. It would be
> nice to have that in a more official place so it doesn't get
> forgotten.
>
> John
> =:->
>
> On Wed, Jul 27, 2016 at 6:19 PM, Katherine Cox-Buday
>  wrote:
>
> I frequently need to connect to Juju's Mongo instance to poke
> around and see if something I've done is having the desired
> effect. Back when we were using LXC, I had a script that would
> pull the password from agent.conf and open a shell. When we
> switched to LXD my script broke, and I never updated it. I finally
> got frustrated enough to modify[1] it, and thought others might
> find this useful for poking around Mongo.
> 
> Let me know if you have any suggestions.
> 
> --
> Katherine
> 
> [1] - http://pastebin.ubuntu.com/21155985/
> 
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
>
>

-- 
Katherine

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Small script to connect to Juju's mongo in LXD

[John Meinel]

Did you intend to attach the script to the email? It does sound like
something useful. I know when we were investigating at some client sites we
had a small snippet of a bash function to dig the content out of agent.conf
and launch mongo with the right options. It would be nice to have that in a
more official place so it doesn't get forgotten.

John
=:->


On Wed, Jul 27, 2016 at 6:19 PM, Katherine Cox-Buday <
katherine.cox-bu...@canonical.com> wrote:

> I frequently need to connect to Juju's Mongo instance to poke around and
> see if something I've done is having the desired effect. Back when we were
> using LXC, I had a script that would pull the password from agent.conf and
> open a shell. When we switched to LXD my script broke, and I never updated
> it. I finally got frustrated enough to modify[1] it, and thought others
> might find this useful for poking around Mongo.
>
> Let me know if you have any suggestions.
>
> --
> Katherine
>
> [1] - http://pastebin.ubuntu.com/21155985/
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Schema for Juju RPC messages

[John A Meinel]

We've had some requests from people trying to drive Juju that it would
actually be really nice if we were stricter with the messages that we
accept on our API. Specifically, as we've changed the API methods, they had
several hard-to-debug problems because they were passing a parameter that
was named incorrectly, and Juju was not giving them any indication that
something was wrong.

I put together a (very hackish) test branch, to see if I could use
JSONSchema to define our request and response messages, and give nicer
error messages (rather than silent acceptance). As I was working with JSON,
I realized the extra " and { characters really got in the way of making a
document that was readable, so I leveraged our existing YAML to JSON
conversion mechanisms to write the description in YAML and then load the
object tree into JSONSchema to validate our requests.

I did end up getting basic validation of the outer structure (just the
Request/Response message, not the contents of Parameters) functioning here:

https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec.go
You can see what some of the error messages look like here:

https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec_test.go

The actual code itself isn't useful as is, because it needs to have the
schema validation stuff pulled out into its own file, etc. But it does give
a proof-of-concept of basic message validation. I'm not super excited by
some of the error messages (gojsonschema.ResultError.Description is very
nice by itself for missing keys, extra properties, etc, but not enough
information for invalid values, while ResultError.String() is overly
verbose in the opposite way.)

I'd like to get a bit of feedback on whether people would find this
interesting, especially if we brought that all the way into the Param
structs as well. I haven't done any benchmarking to determine if this is a
large overhead or not. But the golfing community seems determined to not do
a Strict Unmarshal function, and seems to be recommending using a Schema
instead.

I'm not completely sold either way, though I do find the YAML format for
the schema description to be far more human readable than the JSON format.

Thoughts?

John
=:->


message-schema.json
Description: application/json


message-schema.yaml
Description: Binary data
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Small script to connect to Juju's mongo in LXD

[Katherine Cox-Buday]

I frequently need to connect to Juju's Mongo instance to poke around and see if 
something I've done is having the desired effect. Back when we were using LXC, 
I had a script that would pull the password from agent.conf and open a shell. 
When we switched to LXD my script broke, and I never updated it. I finally got 
frustrated enough to modify[1] it, and thought others might find this useful 
for poking around Mongo.

Let me know if you have any suggestions.

-- 
Katherine

[1] - http://pastebin.ubuntu.com/21155985/

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Critical bugs need fixing

[Curtis Hovey-Canonical]

As seen at
https://bugs.launchpad.net/juju-core/+bugs?field.importance=Critical

We still have 14 Critical bugs, 7 have not been started. The last two
still need assignee.

I'll call out" munna/juju2 is too slow to deploy windows images"
https://bugs.launchpad.net/juju-core/+bug/1605777
Even though the munna mass 1.9 is very slow. juju 1.25 consistently
passes, Juju 2 has not passed in weeks. The test results show Juju 2
cannot deploy windows workloads.

-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju stable 1.25.6 is released

[Mark Shuttleworth]

On 27/07/16 11:35, Martin Packman wrote:
> A stable release of Juju, juju-core 1.25.6, is now available.
> This release replaces version 1.25.5.

Thank you!

Mark

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju stable 1.25.6 is released

[Adam Collard]

Can we use hyperlinks to Launchpad bugs in these release emails? It's a
small papercut but every time I want to dig in on some of the bugs to work
out the potential impact and have to go through a copy/paste step.

Thanks!

On Wed, 27 Jul 2016 at 10:35 Martin Packman 
wrote:

> # juju-core 1.25.6
>
> A stable release of Juju, juju-core 1.25.6, is now available.
> This release replaces version 1.25.5.
>
>
> ## Getting Juju
>
> juju-core 1.25.6 is available for Xenial and backported to earlier
> series in the following PPA:
>
> https://launchpad.net/~juju/+archive/stable
>
> Windows, Centos, and OS X users will find installers at:
>
> https://launchpad.net/juju-core/+milestone/1.25.5
>
>
> ## Notable Changes
>
> This releases addresses stability and performance issues reported by our
> users over the past months. Thanks everyone for continuing to push the
> limits and filing bugs.
>
>
> ## Resolved issues
>
>   * Machine agent failed to register ip addresses, borks agent
> Lp 1537585
>
>   * "invalid entity name or password" error with valid credentials.
> Lp 1514874
>
>   * 'juju help' provider information is out of date
> Lp 1519877
>
>   * Cached local charms should be deleted when their service is
> removed
> Lp 1580418
>
>   * Cannot obtain provisioning script
> Lp 1559299
>
>   * Uniter-hook-execution error prevents  "resolve" unit.
> Lp 1486712
>
>   * Goroutine panic launching container on xenial
> Lp 1592210
>
>   * Trusty juju 1.25.5 ha availability issues
> Lp 1575448
>
>   * Maas provider bridge script on trusty does not handle lacp bonds
> Lp 1594855
>
>   * Concurent map access in joyent
> Lp 1554251
>
>   * Juju run command failing
> Lp 1598964
>
>
> Finally
>
> We encourage everyone to subscribe the mailing list at
> juju-dev@lists.ubuntu.com, or join us on #juju-dev on freenode.
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Juju stable 1.25.6 is released

[Martin Packman]

# juju-core 1.25.6

A stable release of Juju, juju-core 1.25.6, is now available.
This release replaces version 1.25.5.


## Getting Juju

juju-core 1.25.6 is available for Xenial and backported to earlier
series in the following PPA:

https://launchpad.net/~juju/+archive/stable

Windows, Centos, and OS X users will find installers at:

https://launchpad.net/juju-core/+milestone/1.25.5


## Notable Changes

This releases addresses stability and performance issues reported by our
users over the past months. Thanks everyone for continuing to push the
limits and filing bugs.


## Resolved issues

  * Machine agent failed to register ip addresses, borks agent
Lp 1537585

  * "invalid entity name or password" error with valid credentials.
Lp 1514874

  * 'juju help' provider information is out of date
Lp 1519877

  * Cached local charms should be deleted when their service is
removed
Lp 1580418

  * Cannot obtain provisioning script
Lp 1559299

  * Uniter-hook-execution error prevents  "resolve" unit.
Lp 1486712

  * Goroutine panic launching container on xenial
Lp 1592210

  * Trusty juju 1.25.5 ha availability issues
Lp 1575448

  * Maas provider bridge script on trusty does not handle lacp bonds
Lp 1594855

  * Concurent map access in joyent
Lp 1554251

  * Juju run command failing
Lp 1598964


Finally

We encourage everyone to subscribe the mailing list at
juju-dev@lists.ubuntu.com, or join us on #juju-dev on freenode.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: No help Topics in 2.0

[Mark Shuttleworth]

Good catch Nate.

Mark

On 27/07/16 05:08, Nate Finch wrote:
> Juju 1.x had a ton of help in the CLI, right at your fingertips:
>
> $ juju1 help topics
> azure-provider  How to configure a Windows Azure provider
> basics  Basic commands
> commandsBasic help for all commands
> constraints How to use commands with constraints
> ec2-providerHow to configure an Amazon EC2 provider
> global-options  Options common to all commands
> glossaryGlossary of terms
> hpcloud-providerHow to configure an HP Cloud provider
> jujuWhat is Juju?
> juju-systemsAbout Juju Environment Systems (JES)
> local-provider  How to configure a local (LXC) provider
> logging How Juju handles logging
> maas-provider   How to configure a MAAS provider
> openstack-provider  How to configure an OpenStack provider
> placement   How to use placement directives
> plugins Show Juju plugins
> spaces  How to configure more complex networks using spaces
> topics  Topic list
> users   About users in Juju
>
> Almost all of this has been removed in 2.0:
>
> $ juju help topics
> basics  Basic Help Summary
> commandsBasic help for all commands
> global-options  Options common to all commands
> topics  Topic list
>
> There are a lot of parts of the CLI that *need* that extra level of
> documentation.  I'm all for dropping a link to the website at the
> bottom of some help docs for further info, but dropping the help
> entirely seems like a mistake.  Some things like constraints and
> placement are used in multiple commands, but there's no way to know
> from the CLI how to entire valid values for them.  Things like the
> providers are not explained anywhere in the CLI now, as far  as I can
> tell. 
>
> The plaintext docs are the easiest thing for us to update and
> maintain... they live right in our own repo and let us update them
> whenever the code updates, and they're the easiest thing for users to
> get to... let's not drop that on the floor.
>
> -Nate
>
>
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev