Re: [Ecosystem-engineering] The future of Charm Helpers

[Billy Olsen]

Cory,

Yeah, my understanding is that the namespace support in Python 3 is far
improved, and there was some support for it in Python 2.7 which still had
some unique issues from time to time. I'll play around with it a bit and if
I find anything worth mentioning I'll report back. At the very least, it
can go into a known issues/limitations list.

Part of the main reasons I offered this as only a mild concern is that I'm
not fully aware if oslo had made different choices in how namespaces were
being handled that directly impacted the compatibility between python 2 and
python 3 and using namespaced projects.

Thanks!

- Billy



On Mon, Nov 23, 2015 at 12:39 PM, Cory Johns 
wrote:

> Billy,
>
> I also notice that oslo used the following to define the namespace
> packages:
>
> __import__('pkg_resources').declare_namespace(__name__)
>
> My reading indicates that the preferred way to handle namespace packages
> in Python 2 (which is future-compatible with Python 3) is:
>
> from pkgutil import extend_path
> __path__ = extend_path(__path__, __name__)
>
> I tested this (https://github.com/johnsca/nspkg) and it seems to address
> the issues you had with oslo, even in Python 2.  (Note that I did also
> manually test the --system-site-packages + virtualenv case, though I didn't
> commit that code to test.sh in that repo.)
>
> This is the approach we've been using with the charms.X namespace, so I'm
> optimistic that we won't have the same issues you did with oslo.  And, as
> noted in my previous email, we'll probably be switching to Python 3  very
> soon anyway.  However, further testing is always welcome.
>
>
> On Mon, Nov 23, 2015 at 2:01 PM, Cory Johns 
> wrote:
>
>>
>> On Mon, Nov 23, 2015 at 1:37 AM, Billy Olsen 
>> wrote:
>>
>>> Secondly, I'm mildly concerned with the namespace of choice (using the
>>> shared charms. as the parent namespace). There may be a magical python 3ism
>>> that resolves the mixed development + packaged use of common code (think
>>> pip, virtualenvs, etc), but there were some issues that the oslo components
>>> within OpenStack ran into with a shared common namespace ((some are in a
>>> blog here
>>> ,
>>> and the spec to remove the namespaces within the oslo packages is here
>>> ).
>>> As the libraries are broken up, as I believe they should be, we need to
>>> make sure we've carefully considered how we expect some of these flows to
>>> work and make sure they work (and preferably well). Maybe its not really an
>>> issue, but I'd love to be convinced of that.
>>>
>>
>> I do think that namespace package support has been much improved in
>> Python 3 (in particular, 3.3 and above), but I must admit that I had not
>> run into nor been aware of the issues with namespace packages under earlier
>> versions of Python.  However, there has already been discussion of making
>> all layered / reactive charms Python 3 anyway, so maybe we can do some
>> quick tests to determine if those issues have been resolved with the new
>> namespace package support?
>>
>
>


-- 
Billy Olsen

billy.ol...@canonical.com
Software Engineer
Canonical USA
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Cory Johns]

Billy,

I also notice that oslo used the following to define the namespace packages:

__import__('pkg_resources').declare_namespace(__name__)

My reading indicates that the preferred way to handle namespace packages in
Python 2 (which is future-compatible with Python 3) is:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

I tested this (https://github.com/johnsca/nspkg) and it seems to address
the issues you had with oslo, even in Python 2.  (Note that I did also
manually test the --system-site-packages + virtualenv case, though I didn't
commit that code to test.sh in that repo.)

This is the approach we've been using with the charms.X namespace, so I'm
optimistic that we won't have the same issues you did with oslo.  And, as
noted in my previous email, we'll probably be switching to Python 3  very
soon anyway.  However, further testing is always welcome.


On Mon, Nov 23, 2015 at 2:01 PM, Cory Johns 
wrote:

>
> On Mon, Nov 23, 2015 at 1:37 AM, Billy Olsen 
> wrote:
>
>> Secondly, I'm mildly concerned with the namespace of choice (using the
>> shared charms. as the parent namespace). There may be a magical python 3ism
>> that resolves the mixed development + packaged use of common code (think
>> pip, virtualenvs, etc), but there were some issues that the oslo components
>> within OpenStack ran into with a shared common namespace ((some are in a
>> blog here
>> ,
>> and the spec to remove the namespaces within the oslo packages is here
>> ).
>> As the libraries are broken up, as I believe they should be, we need to
>> make sure we've carefully considered how we expect some of these flows to
>> work and make sure they work (and preferably well). Maybe its not really an
>> issue, but I'd love to be convinced of that.
>>
>
> I do think that namespace package support has been much improved in Python
> 3 (in particular, 3.3 and above), but I must admit that I had not run into
> nor been aware of the issues with namespace packages under earlier versions
> of Python.  However, there has already been discussion of making all
> layered / reactive charms Python 3 anyway, so maybe we can do some quick
> tests to determine if those issues have been resolved with the new
> namespace package support?
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Cory Johns]

On Mon, Nov 23, 2015 at 1:37 AM, Billy Olsen 
wrote:

> Secondly, I'm mildly concerned with the namespace of choice (using the
> shared charms. as the parent namespace). There may be a magical python 3ism
> that resolves the mixed development + packaged use of common code (think
> pip, virtualenvs, etc), but there were some issues that the oslo components
> within OpenStack ran into with a shared common namespace ((some are in a
> blog here
> ,
> and the spec to remove the namespaces within the oslo packages is here
> ).
> As the libraries are broken up, as I believe they should be, we need to
> make sure we've carefully considered how we expect some of these flows to
> work and make sure they work (and preferably well). Maybe its not really an
> issue, but I'd love to be convinced of that.
>

I do think that namespace package support has been much improved in Python
3 (in particular, 3.3 and above), but I must admit that I had not run into
nor been aware of the issues with namespace packages under earlier versions
of Python.  However, there has already been discussion of making all
layered / reactive charms Python 3 anyway, so maybe we can do some quick
tests to determine if those issues have been resolved with the new
namespace package support?
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[James Page]

Hi Marco

On Sun, Nov 22, 2015 at 7:23 PM, Marco Ceppi 
wrote:

> Hello everyone,
>
> I'm rebooting this conversation because it never fully came to a
> resolution and since this topic was discussed a lot has happened in the
> Charm Ecosystem. I still hold firm, and a lot of charmers I've spoken with
> agree, that charm helpers has become the opposite which it originally
> helped to solve - a concise and tasteful way to bind Python to Juju. I've
> been considering ways to resolve this, providing a clear way for users to
> author Python charms, while not diminishing the large breadth of helpers
> and methods already created.
>
> A new approach I'd like to present is a clean break from the
> "charm-helpers" name and a transition to a new library, `charms.helper`.
> This name follows the same scheme that the reactive framework does,
> `charms.reactive` and is a way we can continue the practice of producing
> helpers around the charm ecosystem without the need of a monolithic code
> base.
>
> Under this proposal, `charmhelpers.core.hookenv` would more or less become
> `charms.helper` and items like `charmhelpers.contrib.openstack` would be
> moved to their own upstream project and be available as `charms.openstack`.
> They will instead depend on `charms.helper` for previous `hookenv` methods.
> This is a cleaner namespace that still providing the discoverability
> (search pypi index for `charms` and you'll see the ecosystem basically owns
> that space) desired from the current source tree.
>

I'm +1 on this approach and I think we should just bit the bullet and get
it done.


>
> This clean break will allow us to correct a few naming mistmatches and
> allow us to incubate a transition period where charm authors can use and
> try these but the current charm-helpers library has charms.helper as a
> dependency and map current `charmhelpers.core.hookenv` to the new
> `charms.helper`. I've already started work on a strawman to demonstrate how
> charms.helper could look and will share that later this week.
>
> With the new charm build pattern and reactive framework this would fit in
> nicely as we continue on a "charming 2.0" march for quick, easy, and
> concise ways to create charms. I welcome a continued discussion on the
> subject with the hope we can reach a consensus soon on the best way
> forward. One thing is clear, the current way of maintaining charm-helpers
> is neither scalable or manageable.
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Billy Olsen]

I'm very much in favor of breaking up charm helpers as well. I think it
ultimately acknowledges how charm developers have chosen to use the library
(even if not quite correct), but also allows for a theoretically more
organized core library. A few thoughts come up as I read this...

First, the hookenv and other parts of charmhelpers is something that is so
essential to writing a python-based charm these days, I would personally
making a move for the cornerstone python charm library to use the
charms.core namespace, though I get why charms.helper would be used as a
close-to replacement. Honestly, I think just reboot and put it in the
charms.core namespace - you want to write a charm in python, you need this
one library and that's it. Others provide more value, but this one is the
core piece.

Secondly, I'm mildly concerned with the namespace of choice (using the
shared charms. as the parent namespace). There may be a magical python 3ism
that resolves the mixed development + packaged use of common code (think
pip, virtualenvs, etc), but there were some issues that the oslo components
within OpenStack ran into with a shared common namespace ((some are in a
blog here
, and
the spec to remove the namespaces within the oslo packages is here
).
As the libraries are broken up, as I believe they should be, we need to
make sure we've carefully considered how we expect some of these flows to
work and make sure they work (and preferably well). Maybe its not really an
issue, but I'd love to be convinced of that.

Thanks,
Billy

--
Billy Olsen
Software Engineer
Canonical Ltd

On Sun, Nov 22, 2015 at 4:37 PM, Adam Israel 
wrote:

> I am very much in favor of this move forward. I’ve recently worked on
> converting the charm-benchmark package to charms.benchmark; I see where
> having cleaner namespaces make will make every charm author’s life easier.
>
> That said, I know that transitioning to this new model is an epic
> undertaking, especially with the changes coming in the next LTS, i.e., no
> python 2 by default. To that end, I’d propose some kind of compatibility
> report be generated (as part of the upgraded CI, perhaps) that notifies
> charm authors of upcoming changes and how their charm fares against the new
> requirements. The last thing I want to see as a ~charmer is Xenial come to
> pass and having to engage in firefighter mode to fix incompatibilities.
>
>
> Adam Israel - Software Engineer
> Canonical Ltd.
> http://juju.ubuntu.com/ - Automate your Cloud Infrastructure
>
> > On Nov 22, 2015, at 2:23 PM, Marco Ceppi 
> wrote:
> >
> > Hello everyone,
> >
> > I'm rebooting this conversation because it never fully came to a
> resolution and since this topic was discussed a lot has happened in the
> Charm Ecosystem. I still hold firm, and a lot of charmers I've spoken with
> agree, that charm helpers has become the opposite which it originally
> helped to solve - a concise and tasteful way to bind Python to Juju. I've
> been considering ways to resolve this, providing a clear way for users to
> author Python charms, while not diminishing the large breadth of helpers
> and methods already created.
> >
> > A new approach I'd like to present is a clean break from the
> "charm-helpers" name and a transition to a new library, `charms.helper`.
> This name follows the same scheme that the reactive framework does,
> `charms.reactive` and is a way we can continue the practice of producing
> helpers around the charm ecosystem without the need of a monolithic code
> base.
> >
> > Under this proposal, `charmhelpers.core.hookenv` would more or less
> become `charms.helper` and items like `charmhelpers.contrib.openstack`
> would be moved to their own upstream project and be available as
> `charms.openstack`. They will instead depend on `charms.helper` for
> previous `hookenv` methods. This is a cleaner namespace that still
> providing the discoverability (search pypi index for `charms` and you'll
> see the ecosystem basically owns that space) desired from the current
> source tree.
> >
> > This clean break will allow us to correct a few naming mistmatches and
> allow us to incubate a transition period where charm authors can use and
> try these but the current charm-helpers library has charms.helper as a
> dependency and map current `charmhelpers.core.hookenv` to the new
> `charms.helper`. I've already started work on a strawman to demonstrate how
> charms.helper could look and will share that later this week.
> >
> > With the new charm build pattern and reactive framework this would fit
> in nicely as we continue on a "charming 2.0" march for quick, easy, and
> concise ways to create charms. I welcome a continued discussion on the
> subject with the hope we can reach a consensus soon on the best way
> forward. One thing is clear, the current way of maintaining charm-helpers

Re: [Ecosystem-engineering] The future of Charm Helpers

[Adam Israel]

I am very much in favor of this move forward. I’ve recently worked on 
converting the charm-benchmark package to charms.benchmark; I see where having 
cleaner namespaces make will make every charm author’s life easier.

That said, I know that transitioning to this new model is an epic undertaking, 
especially with the changes coming in the next LTS, i.e., no python 2 by 
default. To that end, I’d propose some kind of compatibility report be 
generated (as part of the upgraded CI, perhaps) that notifies charm authors of 
upcoming changes and how their charm fares against the new requirements. The 
last thing I want to see as a ~charmer is Xenial come to pass and having to 
engage in firefighter mode to fix incompatibilities.


Adam Israel - Software Engineer
Canonical Ltd.
http://juju.ubuntu.com/ - Automate your Cloud Infrastructure

> On Nov 22, 2015, at 2:23 PM, Marco Ceppi  wrote:
> 
> Hello everyone,
> 
> I'm rebooting this conversation because it never fully came to a resolution 
> and since this topic was discussed a lot has happened in the Charm Ecosystem. 
> I still hold firm, and a lot of charmers I've spoken with agree, that charm 
> helpers has become the opposite which it originally helped to solve - a 
> concise and tasteful way to bind Python to Juju. I've been considering ways 
> to resolve this, providing a clear way for users to author Python charms, 
> while not diminishing the large breadth of helpers and methods already 
> created.
> 
> A new approach I'd like to present is a clean break from the "charm-helpers" 
> name and a transition to a new library, `charms.helper`. This name follows 
> the same scheme that the reactive framework does, `charms.reactive` and is a 
> way we can continue the practice of producing helpers around the charm 
> ecosystem without the need of a monolithic code base.
> 
> Under this proposal, `charmhelpers.core.hookenv` would more or less become 
> `charms.helper` and items like `charmhelpers.contrib.openstack` would be 
> moved to their own upstream project and be available as `charms.openstack`. 
> They will instead depend on `charms.helper` for previous `hookenv` methods. 
> This is a cleaner namespace that still providing the discoverability (search 
> pypi index for `charms` and you'll see the ecosystem basically owns that 
> space) desired from the current source tree.
> 
> This clean break will allow us to correct a few naming mistmatches and allow 
> us to incubate a transition period where charm authors can use and try these 
> but the current charm-helpers library has charms.helper as a dependency and 
> map current `charmhelpers.core.hookenv` to the new `charms.helper`. I've 
> already started work on a strawman to demonstrate how charms.helper could 
> look and will share that later this week.
> 
> With the new charm build pattern and reactive framework this would fit in 
> nicely as we continue on a "charming 2.0" march for quick, easy, and concise 
> ways to create charms. I welcome a continued discussion on the subject with 
> the hope we can reach a consensus soon on the best way forward. One thing is 
> clear, the current way of maintaining charm-helpers is neither scalable or 
> manageable.
> 
> Thanks,
> Marco Ceppi
> -- 
> Mailing list: https://launchpad.net/~ecosystem-engineering
> Post to : ecosystem-engineer...@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~ecosystem-engineering
> More help   : https://help.launchpad.net/ListHelp


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Cory Johns]

On Thu, Aug 13, 2015 at 8:12 AM, Stuart Bishop 
wrote:

> I think the documentation at https://pythonhosted.org/charmhelpers/ is
> key here. To aid new authors, we need a reference guide of all the
> helpful tools. Even if charm-helpers is split up, I think that we
> still need some way of generating this reference as a cohesive whole.
> If not, we are back where we started with the original problem
> charm-helpers was created to solve - a pile of libraries that
> interoperated poorly and developed in isolation, often wastefully
> reimplementing each others work.
>

Agreed.  We definitely need a cohesive set of documentation that surfaces
work done by various charm authors.  I definitely do *not* feel like the
current documentation does that.  In fact, there were quite a few
charmhelpers sub-packages that were not even included in the documentation
until recently.  Additionally, many of the doc pages are either
sparse-to-empty or giant unreadable walls-of-text (though I tried to help
the latter by introducing the autosummarytable at the start of the bigger
core pages).

I think jujucharms.com/docs needs to be a part of this picture, as well.
It seems like the right place to have a Greater List of Charm Helpers,
including all sub-packages, alternate-language packages, and highlights of
new or interesting helpers.

I personally don't see any advantages in splitting charm-helpers into
> two separate branches. I think it will increase the maintenance burden
> and raise the barriers to entry. Splitting it even further, into core,
> extras, contrib_a, contrib_b, etc. even more so. From my experience,
> this approach just spreads the maintainers too thin, with some
> components badly maintained and others totally unmaintained.
>

I think that it changes the maintenance burden but overall i don't think it
increases it.  I'm honestly tired of maintaining forks of charm-helpers for
various projects, and having already split things out of contrib for those
projects, it has become much easier to manage.


> I believe they should be integrated into an umbrella project, such as
>
charm-helpers. If not they will all wither and die. I don't see any of
> them being able to support a community. There is no point me releasing
> cool stuff of interest to other charmers as a separate branch or
> package, as they will never discover it. And if I did, bug fixes or
> suggestions will be rare, because it is perceived to be my project and
> not their project.
>

We have a unified Juju and Charming community, and we can work to
proselytize each other's work.  You can also post to the mailing list,
attend charm schools and Juju Office Hours to talk about them, write blog
posts, etc.  The projects don't have to live in a shared code-base to
address that.


> For the Pythonic side, I'm mainly thinking of charmhelpers.context
> which I have up for review and have used extensively in the PostgreSQL
> charm rewrite. I got sick of writing relation_set/relation_get etc.
> everywhere, so wrote a higher level API that exposes a big chunk of
> the hook environment as dictionary like objects. I personally think it
> makes charm code much clearer, and if people agree then I think it
> should be in charmhelpers and charmers encouraged to use it over the
> low level API provided by hookenv and Juju. But if this was a 3rd
> party library, it would never gain traction and I doubt I would get
> any feedback on the design at all. I think it is helpful and
> interesting, but not enough for people to go out of their way do
> discover and use. And it is certainly not the sort of thing to port to
> other languages - you could do something similar, but the design would
> necessarily be different.
>

Excellent, I will definitely take a look at this, because I agree with you
and I look forward to seeing how you address it.

However, there have been several attempts (to varying degrees) at this
already in charm-helpers and none of them have gained all that much
traction, so I don't think that's really an argument in favor of
charm-helpers.  I think mentioning it here on the list is much better than
just firing-and-forgetting a review that is easily missed by the
community-at-large.
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Cory Johns]

On Wed, Aug 12, 2015 at 7:38 AM, Marco Ceppi 
wrote:

> I can see how this helps for discoverability for those already pretty
> intimate with the code base, but a lot of the key complaints we've gotten
> around new authors getting started is there is just an overwhelming amount
> of code.
>

I honestly can't understand the argument that the current situation lends
itself to discoverability in any fashion.  I contribute to charmhelpers and
work on many charms, and I have pretty much never looked at contrib because
it's a giant mess.

However, I don't actually think that splitting it will help discoverability
in and of itself, because that just takes one big mess and breaks it in to
many smaller messes.  But I do think it's a step in the right direction,
because it gives the owners of each piece far more freedom to iterate on
their part to make it better.

Then we'd want to have a central location where we collect and list the
individual packages (namespace packages could help here, but an easy
registration process, built in to the tooling, would work, as well).  This
list would link to those packages' documentation (encouraging them to
improve their docs; I'm looking at you,
http://pythonhosted.org/charmhelpers/api/charmhelpers.contrib.ssl.html) and
could even have call-outs for particularly useful packages.


> And your test harnesses are already setup for you. It is easy to add
>> new cool stuff, and its tests. And those tests get run under both
>> Python 2 and Python 3 with the same versions of dependencies and your
>> cool stuff is much more likely to work with other peoples cool stuff.
>> I know for a fact if things get spit out they will have less rigorous
>> testing, and we will end up with tech debt Py2 only modules for
>> example, or conflicting dependencies.
>>
>
> If we provide similar scaffolding as we have now for those projects I
> don't see why they wouldn't continue this way, if people don't take pride
> in their projects they either don't get used or get forked and made better
>

Agreed.  We could create a charm-tools createlib or similar command that
provides you with a skeleton with all the scaffolding set up for you.  As
long as the individual libraries are discoverable (namespaced) or
registered (via a make target in the scaffolding, perhaps), then we could
(semi-)automatically add them to our testing infrastructure.


> On Wed, Aug 12, 2015 at 6:11 AM Stuart Bishop 
> wrote:
>
>
>> But I don't see the large tree as a problem. Its not unwieldy. And you
>> don't have to worry about anything dumped into contrib, except from
>> the perspective of if it is worth migrating it to the core area and
>> how much work it would be (documentation, ensuring consistency,
>> maintainability etc.). I don't see the large tree as a problem because
>> you next talk about packaging.
>>
>
The large tree isn't a problem from the user's perspective, it's a problem
for the people trying to maintain their specific bits of contrib.  It
creates a barrier to entry that encourages long-lived forks of
charm-helpers and perpetuates the charm-helpers-sync terribleness.


> The main reason I will not be using your proposed packages for quite
>> some time is that I like to add things to charm-helpers and fix bugs
>> while working on my charms. I can't do that if I need to wait for a
>> release cycle, even if you could get it down to 24 hours. I will
>> remain using charm-helpers-sync.py, until I can switch to nested git
>> branches, but that is fine. Packages will still make lots of other
>> people's lives easier.
>>
>
Right.  This is exactly *why* we want to split charm-helpers.
 charm-helpers-sync is just about the worst way to solve the dev cycle
problem that there is.  You mentioned git submodules as an alternative,
which would be better, as is jujuresources, "make sdist" and moving a
tarball, composer, and any number of other options.  But the point is that
if we can pare down the number of things in charmhelpers, it makes it much
easier for any given component to be iterated on.  The number of changes to
charm-helpers would settle down drastically once it was pared down, making
the dev cycle and packaging issues much more tractable.


> I tend to think
>> charmhelpers.core should provide wrappers around the juju hook
>> environment, matching the documented juju hook environment behavior,
>> and required low level primitives you need for writing charms. It is
>> the foundation for the rest of charm-helpers, which can provide higher
>> level and friendlier interfaces. I believe that these higher level
>> tools should be Pythonic. So your cli wrappers can publish
>> charmhelpers.core for people insane or uneducated enough to write Bash
>> charms, and the trivial charmhelpers.core would be trivially ported to
>> other high level languages. But porting the Python decorators provided
>> by charmhelpers.coordinator or the Pythonic datastructures provided by
>> charmhelpers.context would not be helpful. Nor would porting

Re: [Ecosystem-engineering] The future of Charm Helpers

[Cory Johns]

Have we considered using Python namespace packages (
https://pythonhosted.org/setuptools/setuptools.html#namespace-packages) as
the spiritual replacement for contrib?  I have already started using the
"charms" namespace in charms.reactive, and I think it could give us a nice
balance between having specialized helpers be independently owned and
maintained, while still giving a (loose) sense of cohesion to those
packages and making it clear that they are all tools for Charming.

I'd also like to note that the upcoming "composer" addition to charm-tools
includes some very nice features that will make dependency management of
charm-helpers much easier.  Specifically, a layer can include a .pypi file
that contains a Python requirement string which, during composition of the
layers (prior to releasing to the store) will cause the package to be
inlined into the charm (sort of like a simplified charm-helpers-sync).  The
key feature here is that you can use  semantic version ranges, so that if
you specify "charmhelpers>=2.0.0,<3.0.0" then whenever you re-compose
(refresh) your charm, you will automatically get backwards compatible
updates within the 2.x version, but will never get backwards incompatible
changes (as long as charmhelpers follows SemVer).  And this all happens at
"compile time" for the charm, so run-time deployments are 100% consistent
and repeatable.

Back on track, I personally lean toward keeping unitdata in charmhelpers
simply because we have historically had a big problem with many charms
using many different means to persist data and I think we need to make it
as easy as possible to have a single, consistent, recommended way to do
that.  I think that including unitdata in charmhelpers and exposing it via
the CLI addresses that (though I'm not averse to another solution to that
core problem).

On Tue, Aug 11, 2015 at 10:15 AM, Marco Ceppi 
wrote:

> On Tue, Aug 11, 2015 at 10:00 AM Matt Bruzek 
> wrote:
>
>> Thanks for the write up Marco.
>>
>> I am particularly interested in growing the CLI aspect of charm-helpers
>> to aid authors who still use bash charms.  Also cleaning up the docs so
>> they make sense and is easier to understand what some methods do.
>>
>> The idea of contrib was so that we could try out a few methods and if it
>> worked well and was useful we pull that into core.  Is there any other
>> contrib features that would be pulled in? How would someone propose a new
>> feature in this new model?
>>
>
> That was the idea for using contrib. Proposing new features into "core"
> will be like most other projects, either opening an issue/bug on the
> charm-helpers project to outline what you want added or submitting code as
> a merge request for review.
>
> For those who want the contrib lifestyle of developing helpers and using
> them in charms but not submitting them yet, branching on launchpad or any
> other VCS and having pip install from a VCS is a viable workflow[0] and one
> I think we should promote.
>
> Marco
>
> [0] https://pip.pypa.io/en/latest/reference/pip_install.html#vcs-support
>
>
>>
>> Thanks,
>>
>>- Matt Bruzek 
>>
>> On Tue, Aug 11, 2015 at 8:42 AM, Marco Ceppi 
>> wrote:
>>
>>> Hello everyone,
>>>
>>> There have been a lot of conversations had over the past few months
>>> about charm-helpers and how to proceed forward with the project. As a
>>> result of these conversations I'd like to surface the following items for
>>> discussion:
>>>
>>> # Trimming down charm-helpers
>>>
>>> The first item, and arguably the largest is a complete reorganization of
>>> the current charm-helpers code base. Originally, charm helpers was setup to
>>> allow a path for developers to place unstable code (contrib directory)
>>> where they could iterate without a guarantee of API stability. However,
>>> this quickly grew to include code that simply didn't make sense as a "core"
>>> feature of charm-helpers. This includes a large set of code for domain
>>> specific charm ecosystems (bigdata, openstack, storage) as well as quickly
>>> fixes to missing features in Juju (leader election).
>>>
>>> The amount of collaborative code in contrib is fantastic! However, it's
>>> making the release and distribution of charm-helpers more complicated. With
>>> an inability to keep up with documentation for methods contrib and the
>>> "volatile" nature of "unstable API", I'd like to propose we completely
>>> remove the contrib directory and instead urge code maintainers in contrib
>>> to create their own libraries for the feature and code they wish to
>>> maintain going forward which simply depends on charm-helpers for the code
>>> in core.
>>>
>>> This will allow groups like bigdata, openstack charmers, storage, and
>>> others to maintain their own release and review process for helpers
>>> specific to their solutions, as well as their own means of delivery (be it
>>> charm-helpers-sync, pypi, debian, or another means). This will also GREATLY
>>> slim down the charm-helpers library to instead include

Re: [Ecosystem-engineering] The future of Charm Helpers

[Marco Ceppi]

On Tue, Aug 11, 2015 at 10:00 AM Matt Bruzek 
wrote:

> Thanks for the write up Marco.
>
> I am particularly interested in growing the CLI aspect of charm-helpers to
> aid authors who still use bash charms.  Also cleaning up the docs so they
> make sense and is easier to understand what some methods do.
>
> The idea of contrib was so that we could try out a few methods and if it
> worked well and was useful we pull that into core.  Is there any other
> contrib features that would be pulled in? How would someone propose a new
> feature in this new model?
>

That was the idea for using contrib. Proposing new features into "core"
will be like most other projects, either opening an issue/bug on the
charm-helpers project to outline what you want added or submitting code as
a merge request for review.

For those who want the contrib lifestyle of developing helpers and using
them in charms but not submitting them yet, branching on launchpad or any
other VCS and having pip install from a VCS is a viable workflow[0] and one
I think we should promote.

Marco

[0] https://pip.pypa.io/en/latest/reference/pip_install.html#vcs-support


>
> Thanks,
>
>- Matt Bruzek 
>
> On Tue, Aug 11, 2015 at 8:42 AM, Marco Ceppi 
> wrote:
>
>> Hello everyone,
>>
>> There have been a lot of conversations had over the past few months about
>> charm-helpers and how to proceed forward with the project. As a result of
>> these conversations I'd like to surface the following items for discussion:
>>
>> # Trimming down charm-helpers
>>
>> The first item, and arguably the largest is a complete reorganization of
>> the current charm-helpers code base. Originally, charm helpers was setup to
>> allow a path for developers to place unstable code (contrib directory)
>> where they could iterate without a guarantee of API stability. However,
>> this quickly grew to include code that simply didn't make sense as a "core"
>> feature of charm-helpers. This includes a large set of code for domain
>> specific charm ecosystems (bigdata, openstack, storage) as well as quickly
>> fixes to missing features in Juju (leader election).
>>
>> The amount of collaborative code in contrib is fantastic! However, it's
>> making the release and distribution of charm-helpers more complicated. With
>> an inability to keep up with documentation for methods contrib and the
>> "volatile" nature of "unstable API", I'd like to propose we completely
>> remove the contrib directory and instead urge code maintainers in contrib
>> to create their own libraries for the feature and code they wish to
>> maintain going forward which simply depends on charm-helpers for the code
>> in core.
>>
>> This will allow groups like bigdata, openstack charmers, storage, and
>> others to maintain their own release and review process for helpers
>> specific to their solutions, as well as their own means of delivery (be it
>> charm-helpers-sync, pypi, debian, or another means). This will also GREATLY
>> slim down the charm-helpers library to instead include one of two things:
>>
>> - API in Python for communicating with Juju
>> - Simple collection of tools in Python to solve shared universal problems
>>
>> Which roughly translates to hookenv.py, host.py, and fetch tools. Once
>> contrib is removed and relocated I'd like to also propose flattening the
>> library namespaces from
>>
>> charmhelpers.core.hookenv -> charmhelpers.hookenv
>> charmhelpers.core.host -> charmhelpers.host
>> charmhelpers.core.files -> charmhelpers.host
>> charmhelpers.core.fstab -> charmhelpers.host
>> charmhelpers.core.sysctl -> charmhelpers.host
>> charmhelpers.core.kv -> separate project?
>>
>> Everything else, outside of contrib would remain the same. While I think
>> it's important to slim down charm-helpers I'm not convinced the above
>> outlined rename/code merges are the right or best way forward. I welcome
>> feedback and a discussion around this.
>>
>> # Packaging charm-helpers
>>
>> Currently, charm-helpers are available via either bazaar source or PyPI.
>> With a more stable, slim, and maintainable code base we would like to
>> package charm-helpers still on PyPI with semantic versioning, make a
>> dist/wheel[0] available for embedding (instead of charm-helpers-sycn), and
>> in Debian and have them available via a PPA but also potentially in the
>> archive proper.
>>
>> The hardest challenge will be keeping the archive as up to date as
>> possible with releases of charm-helpers. Since charms and Juju versions are
>> decoupled from packages "frozen" in the archive, just having a version of
>> charm-helpers in the archive for the version of Juju in that archive won't
>> necessarily last the test of time. Someone deploying a charm now from a new
>> version of Juju against trusty would get the 1.21 equivalent of
>> charm-helpers which may not include the new leadership or extended status
>> methods causing the charm to fail erroneously.
>>
>> Having a PPA can alleviate this, but PyPI and PPAs may interfere with
>

Re: [Ecosystem-engineering] The future of Charm Helpers

[Matt Bruzek]

Thanks for the write up Marco.

I am particularly interested in growing the CLI aspect of charm-helpers to
aid authors who still use bash charms.  Also cleaning up the docs so they
make sense and is easier to understand what some methods do.

The idea of contrib was so that we could try out a few methods and if it
worked well and was useful we pull that into core.  Is there any other
contrib features that would be pulled in? How would someone propose a new
feature in this new model?

Thanks,

   - Matt Bruzek 

On Tue, Aug 11, 2015 at 8:42 AM, Marco Ceppi 
wrote:

> Hello everyone,
>
> There have been a lot of conversations had over the past few months about
> charm-helpers and how to proceed forward with the project. As a result of
> these conversations I'd like to surface the following items for discussion:
>
> # Trimming down charm-helpers
>
> The first item, and arguably the largest is a complete reorganization of
> the current charm-helpers code base. Originally, charm helpers was setup to
> allow a path for developers to place unstable code (contrib directory)
> where they could iterate without a guarantee of API stability. However,
> this quickly grew to include code that simply didn't make sense as a "core"
> feature of charm-helpers. This includes a large set of code for domain
> specific charm ecosystems (bigdata, openstack, storage) as well as quickly
> fixes to missing features in Juju (leader election).
>
> The amount of collaborative code in contrib is fantastic! However, it's
> making the release and distribution of charm-helpers more complicated. With
> an inability to keep up with documentation for methods contrib and the
> "volatile" nature of "unstable API", I'd like to propose we completely
> remove the contrib directory and instead urge code maintainers in contrib
> to create their own libraries for the feature and code they wish to
> maintain going forward which simply depends on charm-helpers for the code
> in core.
>
> This will allow groups like bigdata, openstack charmers, storage, and
> others to maintain their own release and review process for helpers
> specific to their solutions, as well as their own means of delivery (be it
> charm-helpers-sync, pypi, debian, or another means). This will also GREATLY
> slim down the charm-helpers library to instead include one of two things:
>
> - API in Python for communicating with Juju
> - Simple collection of tools in Python to solve shared universal problems
>
> Which roughly translates to hookenv.py, host.py, and fetch tools. Once
> contrib is removed and relocated I'd like to also propose flattening the
> library namespaces from
>
> charmhelpers.core.hookenv -> charmhelpers.hookenv
> charmhelpers.core.host -> charmhelpers.host
> charmhelpers.core.files -> charmhelpers.host
> charmhelpers.core.fstab -> charmhelpers.host
> charmhelpers.core.sysctl -> charmhelpers.host
> charmhelpers.core.kv -> separate project?
>
> Everything else, outside of contrib would remain the same. While I think
> it's important to slim down charm-helpers I'm not convinced the above
> outlined rename/code merges are the right or best way forward. I welcome
> feedback and a discussion around this.
>
> # Packaging charm-helpers
>
> Currently, charm-helpers are available via either bazaar source or PyPI.
> With a more stable, slim, and maintainable code base we would like to
> package charm-helpers still on PyPI with semantic versioning, make a
> dist/wheel[0] available for embedding (instead of charm-helpers-sycn), and
> in Debian and have them available via a PPA but also potentially in the
> archive proper.
>
> The hardest challenge will be keeping the archive as up to date as
> possible with releases of charm-helpers. Since charms and Juju versions are
> decoupled from packages "frozen" in the archive, just having a version of
> charm-helpers in the archive for the version of Juju in that archive won't
> necessarily last the test of time. Someone deploying a charm now from a new
> version of Juju against trusty would get the 1.21 equivalent of
> charm-helpers which may not include the new leadership or extended status
> methods causing the charm to fail erroneously.
>
> Having a PPA can alleviate this, but PyPI and PPAs may interfere with
> firewalls and policies where Juju is being deployed. Opinions on how to
> help with this are appreciated. Otherwise, we may simply recommend authors
> to use a bit of bootstrap code which attempts to install from PyPI, with a
> fallback to install an embedded version of charmhelpers that was included
> in the charm as a dist/wheel.
>
> # Charm-helpers for other languages
>
> We already have charm-helpers written for three languages! Python[0],
> PowerShell[1], and Rust-lang[2]. By sliming down the charm-helpers we can
> define a minimum requirement that each new charm-helper language should
> expose. This will enable us to document charm-helpers more thoroughly and
> include code examples for each language supported in our d