Blind retries [Was: Opaque automatic hook retries from API]

[Free Ekanayaka]

Stuart Bishop  writes:

> I find destroy-service/remove-application is particularly problematic,
> because the doomed units don't know they are being destroyed but rather is
> informed about departing one relation at a time (which is inherently racy,
> because the units the doomed service are related too will process their
> relation-departed hooks almost immediately and stop talking to the doomed
> service, while the doomed service still thinks it can access their
> resources while it falls apart one piece at a time).

Yes. I noticed this issue too, and I think it's a valid Juju bug. I'm
not sure what the best fix would be, but it probably involves some
streamlining of the stop-unit logic (and associated hook sequencing).

[...]
> One of the reasons test suites are currently flaky is that there are
> race conditions we have no reasonable way of solving, such as a
> database restarting itself while a hook on another unit is attempting
> to use it.

In theory this should be rule 0 of programming: handle errors (such as
your code failing to talk to a database). This is of course easier said
than done, but it's been the case forever.

Blind retries are by no means a silver bullet, just because (at least
conceptually) there's no way around at looking at the actual issue at
hand, when deciding how to handle it (e.g. retry).

If you are 100% confident that your code is "idempotent" (for some
definition that makes sense in your case), a blind retry mechanism might
simply mean that your code will take a bit longer to bubble up a failure
(for instance because it's stubbornly retrying a failure condition that
has no way out).

However it's often difficult to judge if some piece of logic is really
idempotent (expecially if the logic encompasses a lot of moving parts,
like a hook run, as opposed to some granular API call). So there's
always the *some* risk that a blind retry could do something unwanted or
even harmful.

If you want to be perfectly safe you should look at the failure at hand
and make sure you understand, before doing anything.

YMMV re real-world statistics of whether this argument is actually
relevant (e.g. "blind retry is good enough for me").

This is by no means an easy topic and it's one of the hard parts of
programming, as exemplified by this recent juju-dev thread:

https://lists.ubuntu.com/archives/juju-dev/2016-October/006091.html

It's also an area where some stardardization of failure modes in
distributed systems would probably help developing some better
automation than blind retry or even some form of AI/learning (the HTTP
spec and RESTful architectures were arguably designed with that in
mind).

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: A (Very) Minimal Charm

[Free Ekanayaka]

>
> As a developer, with an xfs-backed
>

s/xfs/zfs/
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: A (Very) Minimal Charm

[Free Ekanayaka]

On 15 December 2016 at 07:59, John Meinel  wrote:

> Right, the issue for test/development iterations is that "machine
> requested to booted in cloud" for LXD is a lot closer to 10s. Especially if
> you set "enable-os-refresh-update: false" and "enable-os-upgrade: false",
> which are also likely to be set in a testing environment.
>

+1

To re-iterate what previously stated: the point is not production
deployments, but rather development and testing cycles. The experience will
be compared to things like docker & friends where the feedback look happens
in seconds, not minutes or dozens of seconds.

As a developer, with an xfs-backed local juju lxd provider and something
like apt-cacher-ng or deb-squid-proxy installed I expect and want "juju
deploy ubuntu" (or any other minimal reactive-based charm) to take seconds.
Anything less than that gets in the way of doing proper TDD.

Free
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: A (Very) Minimal Charm

[Free Ekanayaka]

On 1 December 2016 at 14:39, Marco Ceppi <marco.ce...@canonical.com> wrote:

> On Thu, Dec 1, 2016 at 8:28 AM Free Ekanayaka <
> free.ekanay...@canonical.com> wrote:
>
>> On 1 December 2016 at 13:53, Marco Ceppi <marco.ce...@canonical.com>
>> wrote:
>>
>> On Thu, Dec 1, 2016 at 5:00 AM Adam Collard <adam.coll...@canonical.com>
>> wrote:
>>
>> On Thu, 1 Dec 2016 at 04:02 Nate Finch <nate.fi...@canonical.com> wrote:
>>
>> On IRC, someone was lamenting the fact that the Ubuntu charm takes longer
>> to deploy now, because it has been updated to exercise more of Juju's
>> features.  My response was - just make a minimal charm, it's easy.  And
>> then of course, I had to figure out how minimal you can get.  Here it is:
>>
>>
>> This is neat, but doesn't detract from the bloat in the ubuntu charm.
>>
>>
>> I'm happy to work though changes to the Ubuntu charm to decrease "bloat".
>>
>>
>> IMHO the bloat in the ubuntu charm isn't from support for Juju features,
>> but the switch to reactive plus conflicts in layer-base wanting to a)
>> support lots of toolchains to allow layers above it to be slimmer and b) be
>> a suitable base for "just deploy me" ubuntu.
>>
>>
>> But it is to support the reactive framework, where we utilize newer Juju
>> features, like status and application-version to make the charm rich
>> despite it's minimal goal set.
>>
>>
>> Yeah, and I think this is a good thing.
>>
>>
>> Honestly, a handful of cached wheelhouses and some apt packages don't
>> strike me as bloat
>>
>>
>> No it's not per-se. However I think this highlights a more general issue
>> with the current implementation of the reactive stack. It's not only the
>> ubuntu charm that has slowed done, it's any reactive-based charm, because
>> the steps required to "setup" reactive take longer than they used to.
>>
>
> The problem we're hitting with wheelhouses is exactly the one that snaps
> aim to solve:
>
> - apt packages are not cross distro, and we have reactive centos charms
> - apt packages lag a bit meaning we can't get consistent features even
> between trusty and xenial, let alone xenial and tip
>
> I see a couple of (possibly alternative) ways to improve the situation:
>>
>> 1) Make sure the dependencies of the base reactive layer are packaged,
>> that should be much faster than pip install, and fall back to pip only for
>> what's not there (i.e. dependencies added by the consumers of the base
>> layer). Also, package the base layer itself.
>>
>
> I'm very keen on a development in the snap world, where an unconfined
> "classic" style package would be available. This means we could snap up all
> the dependencies of the basic layer for every architecture and skip the
> setup phase for reactive. I think this is probably our best bet going
> forward.
>

Sounds good.
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: A (Very) Minimal Charm

[Free Ekanayaka]

On 1 December 2016 at 13:53, Marco Ceppi  wrote:

> On Thu, Dec 1, 2016 at 5:00 AM Adam Collard 
> wrote:
>
>> On Thu, 1 Dec 2016 at 04:02 Nate Finch  wrote:
>>
>> On IRC, someone was lamenting the fact that the Ubuntu charm takes longer
>> to deploy now, because it has been updated to exercise more of Juju's
>> features.  My response was - just make a minimal charm, it's easy.  And
>> then of course, I had to figure out how minimal you can get.  Here it is:
>>
>> It's just a directory with a metadata.yaml in it with these contents:
>>
>> name: min
>> summary: nope
>> description: nope
>> series:
>>   - xenial
>>
>> (obviously you can set the series to whatever you want)
>> No other files or directories are needed.
>>
>>
>> This is neat, but doesn't detract from the bloat in the ubuntu charm.
>>
>
> I'm happy to work though changes to the Ubuntu charm to decrease "bloat".
>
>
>> IMHO the bloat in the ubuntu charm isn't from support for Juju features,
>> but the switch to reactive plus conflicts in layer-base wanting to a)
>> support lots of toolchains to allow layers above it to be slimmer and b) be
>> a suitable base for "just deploy me" ubuntu.
>>
>
> But it is to support the reactive framework, where we utilize newer Juju
> features, like status and application-version to make the charm rich
> despite it's minimal goal set.
>

Yeah, and I think this is a good thing.


> Honestly, a handful of cached wheelhouses and some apt packages don't
> strike me as bloat
>

No it's not per-se. However I think this highlights a more general issue
with the current implementation of the reactive stack. It's not only the
ubuntu charm that has slowed done, it's any reactive-based charm, because
the steps required to "setup" reactive take longer than they used to.

I see a couple of (possibly alternative) ways to improve the situation:

1) Make sure the dependencies of the base reactive layer are packaged, that
should be much faster than pip install, and fall back to pip only for
what's not there (i.e. dependencies added by the consumers of the base
layer). Also, package the base layer itself.

2) Add support for images, so when you deploy some vanilla charm there's an
associated "pre-built" image that will be very fast. I guess this is in the
juju road map anyways.

We always need to keep in mind that this experience will be compared with
things like Kubernetes and Docker, and speed-y deployments really unlock
velocity when iterating on charm development (think for instance running
integration tests).
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: New helper library for unit-testing charms

[Free Ekanayaka]

On 12 October 2016 at 16:29, Cory Johns  wrote:

> Free,
>
> This looks fantastic!  Can I request that you cross-post this over to the
> main Juju mailing list, as it would be very helpful for the community as a
> whole?
>

Glad you like it! I forwarded it to j...@lists.canonical.com too.
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

New helper library for unit-testing charms

[Free Ekanayaka]

Hello,

I published a couple of new Python packages aimed at making unit tests for
charms easier and nicer:

https://github.com/freeekanayaka/charm-test
https://github.com/freeekanayaka/systemfixtures

They implement the classic approach [0] of replacing the "boundaries" of a
charm (hook tools, processes, filesystem, network, etc) with realistic
fakes, so your unit tests run actual code as much as possible and you don't
have to mock internal APIs that directly or indirectly interact with the
boundaries (often leading to mocking explosion).

Besides the package documentation, you might want to have a look to the
jenkins charm reactive rewrite branch [1] to see some real-world use.

If there's interest around this, we could move the code under the juju
GitHub tree, or even merge it to the charm.unit package [2].

Cheers,

Free

[0] http://martinfowler.com/articles/mocksArentStubs.html
[1] https://github.com/freeekanayaka/layer-jenkins
[2] https://github.com/juju-solutions/charms.unit
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju stable 1.20.6 is released.

[Free Ekanayaka]

Hi,

I get this when trying to bootstrap:

WARNING failed to find 1.20.6 tools, will attempt to use 1.20.5
Launching instance
WARNING picked arbitrary tools {1.20.5-trusty-amd64
https://juju-dist.s3.amazonaws.com/rc-testing/tools/releases/juju-1.20.5-trusty-amd64.tgz
052df415c790904cd93204ef1841e4e8005c6be78a8157c899173467664dc2b1 8224282}

I have juju-core 1.20.6-0ubuntu1~14.10.1~juju1 installed on an i386 machine
and I'm bootstrapping against a MAAS provider.

Do I need to upload the 1.20.6 tools somewhere?

Thanks,

Free

On Fri, Aug 29, 2014 at 8:37 PM, Aaron Bentley aaron.bent...@canonical.com
wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 juju-core 1.20.6

 A new stable release of Juju, juju-core 1.20.6, is now available.
 This release replaces 1.20.5.

 juju-core 1.20.6 is available for utopic and backported to earlier
 series in the following PPA:

 https://launchpad.net/~juju/+archive/stable

 Noteworthy

 This releases addresses stability issues, particularly with LXC.

 Resolved issues

 * arguments no longer passed to plugins if you don't have an
   environment set
   Lp 1359170

 * ERROR Nonce already used
   Lp 1190986

 * Juju status returns private IP in 'public-ip' field.
   Lp 1348287

 * juju status unit nil pointer reference.
   Lp 1357482

 * lxc containers created, juju can't seen or communicate with them
   Lp 1357552

 * cannot get tools from machine for lxc container
   Lp 1359800

 * Tools download can fail and is not always retried
   Lp 1360994

 * Containers not starting on machine
   Lp 1362360

 * Update the add machine api to accept both environment name and UUID
   Lp 1360063

 Finally

 We encourage everyone to subscribe the mailing list at
 juju-...@lists.canonical.com, or join us on #juju-dev on freenode.
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1

 iQEcBAEBAgAGBQJUAMh2AAoJEK84cMOcf+9h1sEH/A8jCI1SX7ZIDOY0orw9QXc2
 E9L9pvazW4AhMgLHXrSPsF9xGWJFfbOupwKxYK3ItDAQgjRQlxTTgAr1PdeRIzIG
 4ZpjbsyXrzOZHN6M7FFlhBL3A2lz1MN2mL6a5PqwK4yQ0HeDgbNAghEn1fEd/OyM
 z8nWkfmrSbTFrKInk6B2EQSBAM5vBwS6JaS3A0lw8tgrnCLtUDChvq/x7LG2Hqls
 iAWhHztgXv3XvytMN4wCjLVSrcM4pvzGtfOZ9q5iPmf2s+2h9p7KhQIpdZriPHN7
 fg/UYqHJV3ai+gfSmmeUF7Zlr/vOsRnIIDU/od59jtF79JAX5TmulMRUdF3m400=
 =nZdE
 -END PGP SIGNATURE-

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju stable 1.20.6 is released.

[Free Ekanayaka]

On Mon, Sep 1, 2014 at 10:48 PM, Free Ekanayaka 
free.ekanay...@canonical.com wrote:

 Hi,

 I get this when trying to bootstrap:

 WARNING failed to find 1.20.6 tools, will attempt to use 1.20.5
 Launching instance
 WARNING picked arbitrary tools {1.20.5-trusty-amd64
 https://juju-dist.s3.amazonaws.com/rc-testing/tools/releases/juju-1.20.5-trusty-amd64.tgz
 052df415c790904cd93204ef1841e4e8005c6be78a8157c899173467664dc2b1 8224282}

 I have juju-core 1.20.6-0ubuntu1~14.10.1~juju1 installed on an i386
 machine and I'm bootstrapping against a MAAS provider.

 Do I need to upload the 1.20.6 tools somewhere?


Solved. It was because I had this in my environments.yaml:

tools-metadata-url: https://juju-dist.s3.amazonaws.com/rc-testing/tools

dropping it made the trick (thanks Andreas).

Sorry for the noise.

Cheers,

Free


 On Fri, Aug 29, 2014 at 8:37 PM, Aaron Bentley 
 aaron.bent...@canonical.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 juju-core 1.20.6

 A new stable release of Juju, juju-core 1.20.6, is now available.
 This release replaces 1.20.5.

 juju-core 1.20.6 is available for utopic and backported to earlier
 series in the following PPA:

 https://launchpad.net/~juju/+archive/stable

 Noteworthy

 This releases addresses stability issues, particularly with LXC.

 Resolved issues

 * arguments no longer passed to plugins if you don't have an
   environment set
   Lp 1359170

 * ERROR Nonce already used
   Lp 1190986

 * Juju status returns private IP in 'public-ip' field.
   Lp 1348287

 * juju status unit nil pointer reference.
   Lp 1357482

 * lxc containers created, juju can't seen or communicate with them
   Lp 1357552

 * cannot get tools from machine for lxc container
   Lp 1359800

 * Tools download can fail and is not always retried
   Lp 1360994

 * Containers not starting on machine
   Lp 1362360

 * Update the add machine api to accept both environment name and UUID
   Lp 1360063

 Finally

 We encourage everyone to subscribe the mailing list at
 juju-...@lists.canonical.com, or join us on #juju-dev on freenode.
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1

 iQEcBAEBAgAGBQJUAMh2AAoJEK84cMOcf+9h1sEH/A8jCI1SX7ZIDOY0orw9QXc2
 E9L9pvazW4AhMgLHXrSPsF9xGWJFfbOupwKxYK3ItDAQgjRQlxTTgAr1PdeRIzIG
 4ZpjbsyXrzOZHN6M7FFlhBL3A2lz1MN2mL6a5PqwK4yQ0HeDgbNAghEn1fEd/OyM
 z8nWkfmrSbTFrKInk6B2EQSBAM5vBwS6JaS3A0lw8tgrnCLtUDChvq/x7LG2Hqls
 iAWhHztgXv3XvytMN4wCjLVSrcM4pvzGtfOZ9q5iPmf2s+2h9p7KhQIpdZriPHN7
 fg/UYqHJV3ai+gfSmmeUF7Zlr/vOsRnIIDU/od59jtF79JAX5TmulMRUdF3m400=
 =nZdE
 -END PGP SIGNATURE-

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev