Re: [Ecosystem-engineering] The future of Charm Helpers

[James Page]

Hi Marco

On Sun, Nov 22, 2015 at 7:23 PM, Marco Ceppi 
wrote:

> Hello everyone,
>
> I'm rebooting this conversation because it never fully came to a
> resolution and since this topic was discussed a lot has happened in the
> Charm Ecosystem. I still hold firm, and a lot of charmers I've spoken with
> agree, that charm helpers has become the opposite which it originally
> helped to solve - a concise and tasteful way to bind Python to Juju. I've
> been considering ways to resolve this, providing a clear way for users to
> author Python charms, while not diminishing the large breadth of helpers
> and methods already created.
>
> A new approach I'd like to present is a clean break from the
> "charm-helpers" name and a transition to a new library, `charms.helper`.
> This name follows the same scheme that the reactive framework does,
> `charms.reactive` and is a way we can continue the practice of producing
> helpers around the charm ecosystem without the need of a monolithic code
> base.
>
> Under this proposal, `charmhelpers.core.hookenv` would more or less become
> `charms.helper` and items like `charmhelpers.contrib.openstack` would be
> moved to their own upstream project and be available as `charms.openstack`.
> They will instead depend on `charms.helper` for previous `hookenv` methods.
> This is a cleaner namespace that still providing the discoverability
> (search pypi index for `charms` and you'll see the ecosystem basically owns
> that space) desired from the current source tree.
>

I'm +1 on this approach and I think we should just bit the bullet and get
it done.


>
> This clean break will allow us to correct a few naming mistmatches and
> allow us to incubate a transition period where charm authors can use and
> try these but the current charm-helpers library has charms.helper as a
> dependency and map current `charmhelpers.core.hookenv` to the new
> `charms.helper`. I've already started work on a strawman to demonstrate how
> charms.helper could look and will share that later this week.
>
> With the new charm build pattern and reactive framework this would fit in
> nicely as we continue on a "charming 2.0" march for quick, easy, and
> concise ways to create charms. I welcome a continued discussion on the
> subject with the hope we can reach a consensus soon on the best way
> forward. One thing is clear, the current way of maintaining charm-helpers
> is neither scalable or manageable.
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Cory Johns]

On Mon, Nov 23, 2015 at 1:37 AM, Billy Olsen 
wrote:

> Secondly, I'm mildly concerned with the namespace of choice (using the
> shared charms. as the parent namespace). There may be a magical python 3ism
> that resolves the mixed development + packaged use of common code (think
> pip, virtualenvs, etc), but there were some issues that the oslo components
> within OpenStack ran into with a shared common namespace ((some are in a
> blog here
> ,
> and the spec to remove the namespaces within the oslo packages is here
> ).
> As the libraries are broken up, as I believe they should be, we need to
> make sure we've carefully considered how we expect some of these flows to
> work and make sure they work (and preferably well). Maybe its not really an
> issue, but I'd love to be convinced of that.
>

I do think that namespace package support has been much improved in Python
3 (in particular, 3.3 and above), but I must admit that I had not run into
nor been aware of the issues with namespace packages under earlier versions
of Python.  However, there has already been discussion of making all
layered / reactive charms Python 3 anyway, so maybe we can do some quick
tests to determine if those issues have been resolved with the new
namespace package support?
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Billy Olsen]

Cory,

Yeah, my understanding is that the namespace support in Python 3 is far
improved, and there was some support for it in Python 2.7 which still had
some unique issues from time to time. I'll play around with it a bit and if
I find anything worth mentioning I'll report back. At the very least, it
can go into a known issues/limitations list.

Part of the main reasons I offered this as only a mild concern is that I'm
not fully aware if oslo had made different choices in how namespaces were
being handled that directly impacted the compatibility between python 2 and
python 3 and using namespaced projects.

Thanks!

- Billy



On Mon, Nov 23, 2015 at 12:39 PM, Cory Johns 
wrote:

> Billy,
>
> I also notice that oslo used the following to define the namespace
> packages:
>
> __import__('pkg_resources').declare_namespace(__name__)
>
> My reading indicates that the preferred way to handle namespace packages
> in Python 2 (which is future-compatible with Python 3) is:
>
> from pkgutil import extend_path
> __path__ = extend_path(__path__, __name__)
>
> I tested this (https://github.com/johnsca/nspkg) and it seems to address
> the issues you had with oslo, even in Python 2.  (Note that I did also
> manually test the --system-site-packages + virtualenv case, though I didn't
> commit that code to test.sh in that repo.)
>
> This is the approach we've been using with the charms.X namespace, so I'm
> optimistic that we won't have the same issues you did with oslo.  And, as
> noted in my previous email, we'll probably be switching to Python 3  very
> soon anyway.  However, further testing is always welcome.
>
>
> On Mon, Nov 23, 2015 at 2:01 PM, Cory Johns 
> wrote:
>
>>
>> On Mon, Nov 23, 2015 at 1:37 AM, Billy Olsen 
>> wrote:
>>
>>> Secondly, I'm mildly concerned with the namespace of choice (using the
>>> shared charms. as the parent namespace). There may be a magical python 3ism
>>> that resolves the mixed development + packaged use of common code (think
>>> pip, virtualenvs, etc), but there were some issues that the oslo components
>>> within OpenStack ran into with a shared common namespace ((some are in a
>>> blog here
>>> ,
>>> and the spec to remove the namespaces within the oslo packages is here
>>> ).
>>> As the libraries are broken up, as I believe they should be, we need to
>>> make sure we've carefully considered how we expect some of these flows to
>>> work and make sure they work (and preferably well). Maybe its not really an
>>> issue, but I'd love to be convinced of that.
>>>
>>
>> I do think that namespace package support has been much improved in
>> Python 3 (in particular, 3.3 and above), but I must admit that I had not
>> run into nor been aware of the issues with namespace packages under earlier
>> versions of Python.  However, there has already been discussion of making
>> all layered / reactive charms Python 3 anyway, so maybe we can do some
>> quick tests to determine if those issues have been resolved with the new
>> namespace package support?
>>
>
>


-- 
Billy Olsen

billy.ol...@canonical.com
Software Engineer
Canonical USA
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Adam Israel]

I am very much in favor of this move forward. I’ve recently worked on 
converting the charm-benchmark package to charms.benchmark; I see where having 
cleaner namespaces make will make every charm author’s life easier.

That said, I know that transitioning to this new model is an epic undertaking, 
especially with the changes coming in the next LTS, i.e., no python 2 by 
default. To that end, I’d propose some kind of compatibility report be 
generated (as part of the upgraded CI, perhaps) that notifies charm authors of 
upcoming changes and how their charm fares against the new requirements. The 
last thing I want to see as a ~charmer is Xenial come to pass and having to 
engage in firefighter mode to fix incompatibilities.


Adam Israel - Software Engineer
Canonical Ltd.
http://juju.ubuntu.com/ - Automate your Cloud Infrastructure

> On Nov 22, 2015, at 2:23 PM, Marco Ceppi  wrote:
> 
> Hello everyone,
> 
> I'm rebooting this conversation because it never fully came to a resolution 
> and since this topic was discussed a lot has happened in the Charm Ecosystem. 
> I still hold firm, and a lot of charmers I've spoken with agree, that charm 
> helpers has become the opposite which it originally helped to solve - a 
> concise and tasteful way to bind Python to Juju. I've been considering ways 
> to resolve this, providing a clear way for users to author Python charms, 
> while not diminishing the large breadth of helpers and methods already 
> created.
> 
> A new approach I'd like to present is a clean break from the "charm-helpers" 
> name and a transition to a new library, `charms.helper`. This name follows 
> the same scheme that the reactive framework does, `charms.reactive` and is a 
> way we can continue the practice of producing helpers around the charm 
> ecosystem without the need of a monolithic code base.
> 
> Under this proposal, `charmhelpers.core.hookenv` would more or less become 
> `charms.helper` and items like `charmhelpers.contrib.openstack` would be 
> moved to their own upstream project and be available as `charms.openstack`. 
> They will instead depend on `charms.helper` for previous `hookenv` methods. 
> This is a cleaner namespace that still providing the discoverability (search 
> pypi index for `charms` and you'll see the ecosystem basically owns that 
> space) desired from the current source tree.
> 
> This clean break will allow us to correct a few naming mistmatches and allow 
> us to incubate a transition period where charm authors can use and try these 
> but the current charm-helpers library has charms.helper as a dependency and 
> map current `charmhelpers.core.hookenv` to the new `charms.helper`. I've 
> already started work on a strawman to demonstrate how charms.helper could 
> look and will share that later this week.
> 
> With the new charm build pattern and reactive framework this would fit in 
> nicely as we continue on a "charming 2.0" march for quick, easy, and concise 
> ways to create charms. I welcome a continued discussion on the subject with 
> the hope we can reach a consensus soon on the best way forward. One thing is 
> clear, the current way of maintaining charm-helpers is neither scalable or 
> manageable.
> 
> Thanks,
> Marco Ceppi
> -- 
> Mailing list: https://launchpad.net/~ecosystem-engineering
> Post to : ecosystem-engineer...@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~ecosystem-engineering
> More help   : https://help.launchpad.net/ListHelp


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Ecosystem-engineering] The future of Charm Helpers

[Cory Johns]

On Wed, Aug 12, 2015 at 7:38 AM, Marco Ceppi marco.ce...@canonical.com
wrote:

 I can see how this helps for discoverability for those already pretty
 intimate with the code base, but a lot of the key complaints we've gotten
 around new authors getting started is there is just an overwhelming amount
 of code.


I honestly can't understand the argument that the current situation lends
itself to discoverability in any fashion.  I contribute to charmhelpers and
work on many charms, and I have pretty much never looked at contrib because
it's a giant mess.

However, I don't actually think that splitting it will help discoverability
in and of itself, because that just takes one big mess and breaks it in to
many smaller messes.  But I do think it's a step in the right direction,
because it gives the owners of each piece far more freedom to iterate on
their part to make it better.

Then we'd want to have a central location where we collect and list the
individual packages (namespace packages could help here, but an easy
registration process, built in to the tooling, would work, as well).  This
list would link to those packages' documentation (encouraging them to
improve their docs; I'm looking at you,
http://pythonhosted.org/charmhelpers/api/charmhelpers.contrib.ssl.html) and
could even have call-outs for particularly useful packages.


 And your test harnesses are already setup for you. It is easy to add
 new cool stuff, and its tests. And those tests get run under both
 Python 2 and Python 3 with the same versions of dependencies and your
 cool stuff is much more likely to work with other peoples cool stuff.
 I know for a fact if things get spit out they will have less rigorous
 testing, and we will end up with tech debt Py2 only modules for
 example, or conflicting dependencies.


 If we provide similar scaffolding as we have now for those projects I
 don't see why they wouldn't continue this way, if people don't take pride
 in their projects they either don't get used or get forked and made better


Agreed.  We could create a charm-tools createlib or similar command that
provides you with a skeleton with all the scaffolding set up for you.  As
long as the individual libraries are discoverable (namespaced) or
registered (via a make target in the scaffolding, perhaps), then we could
(semi-)automatically add them to our testing infrastructure.


 On Wed, Aug 12, 2015 at 6:11 AM Stuart Bishop stuart.bis...@canonical.com
 wrote:


 But I don't see the large tree as a problem. Its not unwieldy. And you
 don't have to worry about anything dumped into contrib, except from
 the perspective of if it is worth migrating it to the core area and
 how much work it would be (documentation, ensuring consistency,
 maintainability etc.). I don't see the large tree as a problem because
 you next talk about packaging.


The large tree isn't a problem from the user's perspective, it's a problem
for the people trying to maintain their specific bits of contrib.  It
creates a barrier to entry that encourages long-lived forks of
charm-helpers and perpetuates the charm-helpers-sync terribleness.


 The main reason I will not be using your proposed packages for quite
 some time is that I like to add things to charm-helpers and fix bugs
 while working on my charms. I can't do that if I need to wait for a
 release cycle, even if you could get it down to 24 hours. I will
 remain using charm-helpers-sync.py, until I can switch to nested git
 branches, but that is fine. Packages will still make lots of other
 people's lives easier.


Right.  This is exactly *why* we want to split charm-helpers.
 charm-helpers-sync is just about the worst way to solve the dev cycle
problem that there is.  You mentioned git submodules as an alternative,
which would be better, as is jujuresources, make sdist and moving a
tarball, composer, and any number of other options.  But the point is that
if we can pare down the number of things in charmhelpers, it makes it much
easier for any given component to be iterated on.  The number of changes to
charm-helpers would settle down drastically once it was pared down, making
the dev cycle and packaging issues much more tractable.


 I tend to think
 charmhelpers.core should provide wrappers around the juju hook
 environment, matching the documented juju hook environment behavior,
 and required low level primitives you need for writing charms. It is
 the foundation for the rest of charm-helpers, which can provide higher
 level and friendlier interfaces. I believe that these higher level
 tools should be Pythonic. So your cli wrappers can publish
 charmhelpers.core for people insane or uneducated enough to write Bash
 charms, and the trivial charmhelpers.core would be trivially ported to
 other high level languages. But porting the Python decorators provided
 by charmhelpers.coordinator or the Pythonic datastructures provided by
 charmhelpers.context would not be helpful. Nor would porting
 charmhelpers.core.templating, 

Re: [Ecosystem-engineering] The future of Charm Helpers

[Marco Ceppi]

On Tue, Aug 11, 2015 at 10:00 AM Matt Bruzek matthew.bru...@canonical.com
wrote:

 Thanks for the write up Marco.

 I am particularly interested in growing the CLI aspect of charm-helpers to
 aid authors who still use bash charms.  Also cleaning up the docs so they
 make sense and is easier to understand what some methods do.

 The idea of contrib was so that we could try out a few methods and if it
 worked well and was useful we pull that into core.  Is there any other
 contrib features that would be pulled in? How would someone propose a new
 feature in this new model?


That was the idea for using contrib. Proposing new features into core
will be like most other projects, either opening an issue/bug on the
charm-helpers project to outline what you want added or submitting code as
a merge request for review.

For those who want the contrib lifestyle of developing helpers and using
them in charms but not submitting them yet, branching on launchpad or any
other VCS and having pip install from a VCS is a viable workflow[0] and one
I think we should promote.

Marco

[0] https://pip.pypa.io/en/latest/reference/pip_install.html#vcs-support



 Thanks,

- Matt Bruzek matthew.bru...@canonical.com

 On Tue, Aug 11, 2015 at 8:42 AM, Marco Ceppi marco.ce...@canonical.com
 wrote:

 Hello everyone,

 There have been a lot of conversations had over the past few months about
 charm-helpers and how to proceed forward with the project. As a result of
 these conversations I'd like to surface the following items for discussion:

 # Trimming down charm-helpers

 The first item, and arguably the largest is a complete reorganization of
 the current charm-helpers code base. Originally, charm helpers was setup to
 allow a path for developers to place unstable code (contrib directory)
 where they could iterate without a guarantee of API stability. However,
 this quickly grew to include code that simply didn't make sense as a core
 feature of charm-helpers. This includes a large set of code for domain
 specific charm ecosystems (bigdata, openstack, storage) as well as quickly
 fixes to missing features in Juju (leader election).

 The amount of collaborative code in contrib is fantastic! However, it's
 making the release and distribution of charm-helpers more complicated. With
 an inability to keep up with documentation for methods contrib and the
 volatile nature of unstable API, I'd like to propose we completely
 remove the contrib directory and instead urge code maintainers in contrib
 to create their own libraries for the feature and code they wish to
 maintain going forward which simply depends on charm-helpers for the code
 in core.

 This will allow groups like bigdata, openstack charmers, storage, and
 others to maintain their own release and review process for helpers
 specific to their solutions, as well as their own means of delivery (be it
 charm-helpers-sync, pypi, debian, or another means). This will also GREATLY
 slim down the charm-helpers library to instead include one of two things:

 - API in Python for communicating with Juju
 - Simple collection of tools in Python to solve shared universal problems

 Which roughly translates to hookenv.py, host.py, and fetch tools. Once
 contrib is removed and relocated I'd like to also propose flattening the
 library namespaces from

 charmhelpers.core.hookenv - charmhelpers.hookenv
 charmhelpers.core.host - charmhelpers.host
 charmhelpers.core.files - charmhelpers.host
 charmhelpers.core.fstab - charmhelpers.host
 charmhelpers.core.sysctl - charmhelpers.host
 charmhelpers.core.kv - separate project?

 Everything else, outside of contrib would remain the same. While I think
 it's important to slim down charm-helpers I'm not convinced the above
 outlined rename/code merges are the right or best way forward. I welcome
 feedback and a discussion around this.

 # Packaging charm-helpers

 Currently, charm-helpers are available via either bazaar source or PyPI.
 With a more stable, slim, and maintainable code base we would like to
 package charm-helpers still on PyPI with semantic versioning, make a
 dist/wheel[0] available for embedding (instead of charm-helpers-sycn), and
 in Debian and have them available via a PPA but also potentially in the
 archive proper.

 The hardest challenge will be keeping the archive as up to date as
 possible with releases of charm-helpers. Since charms and Juju versions are
 decoupled from packages frozen in the archive, just having a version of
 charm-helpers in the archive for the version of Juju in that archive won't
 necessarily last the test of time. Someone deploying a charm now from a new
 version of Juju against trusty would get the 1.21 equivalent of
 charm-helpers which may not include the new leadership or extended status
 methods causing the charm to fail erroneously.

 Having a PPA can alleviate this, but PyPI and PPAs may interfere with
 firewalls and policies where Juju is being deployed. Opinions on how to
 help with this are