Re: The future of Charm Helpers

2015-11-23 Thread James Page 
On Mon, Nov 23, 2015 at 9:49 AM, Merlijn Sebrechts <
merlijn.sebrec...@gmail.com> wrote:

> Hi James
>
>
> > Requiring that anything under the charms namespace is reactive aware is
> fine - but lets make that 'aware' not 'required' please.
>
> I'm not sure I understand what you are saying. Could you explain this a
> bit more?
>
> For more context: What I propose is that everything under the Charms
> namespace is reactive aware. Code does not get included in that namespace
> if it isn't reactive aware. From my point of view this would pose no
> problem to the ecosystems that are not yet reactive aware since they can
> still use their current charmhelpers.xyz code. I think devs will assume
> that everything in the charms namespace works together well.
>


Well charmhelpers.openstack will want to use charms.hookenv for example -
so charms.hookenv should be reactive aware, but should also be functional
without the reactive framework - does that make things clearer?



>
> 2015-11-23 10:40 GMT+01:00 James Page :
>
>> Hi Merlijn
>>
>> On Mon, Nov 23, 2015 at 9:32 AM, Merlijn Sebrechts <
>> merlijn.sebrec...@gmail.com> wrote:
>>
>>> I'm all for requiring everything under the charms namespace to be
>>> reactive aware.
>>>
>>> It is hard for people who are less involved to make sense of all the
>>> different ways to Charm. It's even harder because some ways get deprecated
>>> faster than they get adopted (services framework). I think it's vital to
>>> have a very clear message to charmers about what the recommended way is. We
>>> shouldn't confuse them by exposing deprecated and risky ways to charm.
>>> Let's bite the bullet and use this namespace change to start over with a
>>> clear message.
>>>
>>> I'd say it like this: use bash or use reactive. Everything else is nice
>>> to have but they shouldn't stumble upon it by accident.
>>>
>>
>>
>> I agree with the clear messaging here, but please lets not dead-end all
>> of the charms that don't use reactive; The OpenStack charmers team and the
>> ecosystem of partners who have charms that integrate into the OpenStack
>> charm set have a large codebase which is not reactive based and won't be in
>> the near term.
>>
>> Requiring that anything under the charms namespace is reactive aware is
>> fine - but lets make that 'aware' not 'required' please.
>>
>>
>>>
>>>
>>>
>>> 2015-11-23 9:47 GMT+01:00 Stuart Bishop :
>>>
 On 23 November 2015 at 02:23, Marco Ceppi 
 wrote:

 > Under this proposal, `charmhelpers.core.hookenv` would more or less
 become
 > `charms.helper` and items like `charmhelpers.contrib.openstack` would
 be
 > moved to their own upstream project and be available as
 `charms.openstack`.
 > They will instead depend on `charms.helper` for previous `hookenv`
 methods.
 > This is a cleaner namespace that still providing the discoverability
 (search
 > pypi index for `charms` and you'll see the ecosystem basically owns
 that
 > space) desired from the current source tree.

 > With the new charm build pattern and reactive framework this would
 fit in
 > nicely as we continue on a "charming 2.0" march for quick, easy, and
 concise
 > ways to create charms. I welcome a continued discussion on the
 subject with
 > the hope we can reach a consensus soon on the best way forward. One
 thing is
 > clear, the current way of maintaining charm-helpers is neither
 scalable or
 > manageable.

 I don't think it matters what you do with the low level hookenv
 library, as reactive charms should be using a higher level library
 that sets states appropriately (and mixing calls just means state and
 hook environment will get out of sync).

 I think it is worth doing this in tandem with creating
 charms.reactive.hookenv. It is really, really useful having handlers
 watching for states like 'leadership.set.foo' or 'config.changed.bar'
 or 'workloadstatus.blocked', but if layers start using the lower level
 API then state will get out of sync with the hook environment.

 Or should everything under the charms namespace be reactive framework
 aware, with charms.reactive just being where the engine is stored?

 --
 Stuart Bishop 

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju

>>>
>>>
>>>
>>> --
>>> Juju mailing list
>>> Juju@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>
>>>
>>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: The future of Charm Helpers

2015-11-23 Thread Stuart Bishop 
On 23 November 2015 at 02:23, Marco Ceppi  wrote:

> Under this proposal, `charmhelpers.core.hookenv` would more or less become
> `charms.helper` and items like `charmhelpers.contrib.openstack` would be
> moved to their own upstream project and be available as `charms.openstack`.
> They will instead depend on `charms.helper` for previous `hookenv` methods.
> This is a cleaner namespace that still providing the discoverability (search
> pypi index for `charms` and you'll see the ecosystem basically owns that
> space) desired from the current source tree.

> With the new charm build pattern and reactive framework this would fit in
> nicely as we continue on a "charming 2.0" march for quick, easy, and concise
> ways to create charms. I welcome a continued discussion on the subject with
> the hope we can reach a consensus soon on the best way forward. One thing is
> clear, the current way of maintaining charm-helpers is neither scalable or
> manageable.

I don't think it matters what you do with the low level hookenv
library, as reactive charms should be using a higher level library
that sets states appropriately (and mixing calls just means state and
hook environment will get out of sync).

I think it is worth doing this in tandem with creating
charms.reactive.hookenv. It is really, really useful having handlers
watching for states like 'leadership.set.foo' or 'config.changed.bar'
or 'workloadstatus.blocked', but if layers start using the lower level
API then state will get out of sync with the hook environment.

Or should everything under the charms namespace be reactive framework
aware, with charms.reactive just being where the engine is stored?

-- 
Stuart Bishop 

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: The future of Charm Helpers

2015-08-12 Thread Christopher Glass 
Small comment while mentioning charm-helpers:

What I would really like to see is the project's tests running in CI,
before things start moving around. The last few times we ran them we
discovered many were broken (I suspect people need to put out fires
and don't bother running the tests before committing).

Could we consider moving to i.e. github.com with travis? It's there,
it works, and somebody else is maintaining it (for free, no less).

- Chris

On Tue, Aug 11, 2015 at 3:42 PM, Marco Ceppi marco.ce...@canonical.com wrote:
 Hello everyone,

 There have been a lot of conversations had over the past few months about
 charm-helpers and how to proceed forward with the project. As a result of
 these conversations I'd like to surface the following items for discussion:

 # Trimming down charm-helpers

 The first item, and arguably the largest is a complete reorganization of the
 current charm-helpers code base. Originally, charm helpers was setup to
 allow a path for developers to place unstable code (contrib directory) where
 they could iterate without a guarantee of API stability. However, this
 quickly grew to include code that simply didn't make sense as a core
 feature of charm-helpers. This includes a large set of code for domain
 specific charm ecosystems (bigdata, openstack, storage) as well as quickly
 fixes to missing features in Juju (leader election).

 The amount of collaborative code in contrib is fantastic! However, it's
 making the release and distribution of charm-helpers more complicated. With
 an inability to keep up with documentation for methods contrib and the
 volatile nature of unstable API, I'd like to propose we completely
 remove the contrib directory and instead urge code maintainers in contrib to
 create their own libraries for the feature and code they wish to maintain
 going forward which simply depends on charm-helpers for the code in core.

 This will allow groups like bigdata, openstack charmers, storage, and others
 to maintain their own release and review process for helpers specific to
 their solutions, as well as their own means of delivery (be it
 charm-helpers-sync, pypi, debian, or another means). This will also GREATLY
 slim down the charm-helpers library to instead include one of two things:

 - API in Python for communicating with Juju
 - Simple collection of tools in Python to solve shared universal problems

 Which roughly translates to hookenv.py, host.py, and fetch tools. Once
 contrib is removed and relocated I'd like to also propose flattening the
 library namespaces from

 charmhelpers.core.hookenv - charmhelpers.hookenv
 charmhelpers.core.host - charmhelpers.host
 charmhelpers.core.files - charmhelpers.host
 charmhelpers.core.fstab - charmhelpers.host
 charmhelpers.core.sysctl - charmhelpers.host
 charmhelpers.core.kv - separate project?

 Everything else, outside of contrib would remain the same. While I think
 it's important to slim down charm-helpers I'm not convinced the above
 outlined rename/code merges are the right or best way forward. I welcome
 feedback and a discussion around this.

 # Packaging charm-helpers

 Currently, charm-helpers are available via either bazaar source or PyPI.
 With a more stable, slim, and maintainable code base we would like to
 package charm-helpers still on PyPI with semantic versioning, make a
 dist/wheel[0] available for embedding (instead of charm-helpers-sycn), and
 in Debian and have them available via a PPA but also potentially in the
 archive proper.

 The hardest challenge will be keeping the archive as up to date as possible
 with releases of charm-helpers. Since charms and Juju versions are decoupled
 from packages frozen in the archive, just having a version of
 charm-helpers in the archive for the version of Juju in that archive won't
 necessarily last the test of time. Someone deploying a charm now from a new
 version of Juju against trusty would get the 1.21 equivalent of
 charm-helpers which may not include the new leadership or extended status
 methods causing the charm to fail erroneously.

 Having a PPA can alleviate this, but PyPI and PPAs may interfere with
 firewalls and policies where Juju is being deployed. Opinions on how to help
 with this are appreciated. Otherwise, we may simply recommend authors to use
 a bit of bootstrap code which attempts to install from PyPI, with a fallback
 to install an embedded version of charmhelpers that was included in the
 charm as a dist/wheel.

 # Charm-helpers for other languages

 We already have charm-helpers written for three languages! Python[0],
 PowerShell[1], and Rust-lang[2]. By sliming down the charm-helpers we can
 define a minimum requirement that each new charm-helper language should
 expose. This will enable us to document charm-helpers more thoroughly and
 include code examples for each language supported in our docs. Also, by
 having a narrowly defined scope of methods, it enables other communities to
 create helpers for their languages.

 The

Re: The future of Charm Helpers

2015-08-12 Thread Christopher Glass 
I personally don't care where the CI runs, as long as it runs :)

Adding to that, it should run *for every merge proposal*, not just on
trunk (when it's too late). Travis is a time and cost effective way to
achieve this, but I'd be more than happy with whatever alternative
solution.

- Chris

On Wed, Aug 12, 2015 at 1:12 PM, Marco Ceppi marco.ce...@canonical.com wrote:
 I don't have a problem moving to github for the project. However, we have a
 lot of people contributing to the library already in launchpad and I would
 hate to disrupt that flow. I'm not sure if there's an advantage to moving to
 github at this moment in time if only to use travis. I'd be interested in
 more contributors feedback about moving before doing anything. Working with
 the Juju QA team to add a Jenkins job for test landing is a great idea, one
 we should certainly do now while we figure the rest of this out.

 On Wed, Aug 12, 2015 at 6:59 AM Christopher Glass trib...@gmail.com wrote:

 Small comment while mentioning charm-helpers:

 What I would really like to see is the project's tests running in CI,
 before things start moving around. The last few times we ran them we
 discovered many were broken (I suspect people need to put out fires
 and don't bother running the tests before committing).

 Could we consider moving to i.e. github.com with travis? It's there,
 it works, and somebody else is maintaining it (for free, no less).

 - Chris

 On Tue, Aug 11, 2015 at 3:42 PM, Marco Ceppi marco.ce...@canonical.com
 wrote:
  Hello everyone,
 
  There have been a lot of conversations had over the past few months
  about
  charm-helpers and how to proceed forward with the project. As a result
  of
  these conversations I'd like to surface the following items for
  discussion:
 
  # Trimming down charm-helpers
 
  The first item, and arguably the largest is a complete reorganization of
  the
  current charm-helpers code base. Originally, charm helpers was setup to
  allow a path for developers to place unstable code (contrib directory)
  where
  they could iterate without a guarantee of API stability. However, this
  quickly grew to include code that simply didn't make sense as a core
  feature of charm-helpers. This includes a large set of code for domain
  specific charm ecosystems (bigdata, openstack, storage) as well as
  quickly
  fixes to missing features in Juju (leader election).
 
  The amount of collaborative code in contrib is fantastic! However, it's
  making the release and distribution of charm-helpers more complicated.
  With
  an inability to keep up with documentation for methods contrib and the
  volatile nature of unstable API, I'd like to propose we completely
  remove the contrib directory and instead urge code maintainers in
  contrib to
  create their own libraries for the feature and code they wish to
  maintain
  going forward which simply depends on charm-helpers for the code in
  core.
 
  This will allow groups like bigdata, openstack charmers, storage, and
  others
  to maintain their own release and review process for helpers specific to
  their solutions, as well as their own means of delivery (be it
  charm-helpers-sync, pypi, debian, or another means). This will also
  GREATLY
  slim down the charm-helpers library to instead include one of two
  things:
 
  - API in Python for communicating with Juju
  - Simple collection of tools in Python to solve shared universal
  problems
 
  Which roughly translates to hookenv.py, host.py, and fetch tools. Once
  contrib is removed and relocated I'd like to also propose flattening the
  library namespaces from
 
  charmhelpers.core.hookenv - charmhelpers.hookenv
  charmhelpers.core.host - charmhelpers.host
  charmhelpers.core.files - charmhelpers.host
  charmhelpers.core.fstab - charmhelpers.host
  charmhelpers.core.sysctl - charmhelpers.host
  charmhelpers.core.kv - separate project?
 
  Everything else, outside of contrib would remain the same. While I think
  it's important to slim down charm-helpers I'm not convinced the above
  outlined rename/code merges are the right or best way forward. I welcome
  feedback and a discussion around this.
 
  # Packaging charm-helpers
 
  Currently, charm-helpers are available via either bazaar source or PyPI.
  With a more stable, slim, and maintainable code base we would like to
  package charm-helpers still on PyPI with semantic versioning, make a
  dist/wheel[0] available for embedding (instead of charm-helpers-sycn),
  and
  in Debian and have them available via a PPA but also potentially in the
  archive proper.
 
  The hardest challenge will be keeping the archive as up to date as
  possible
  with releases of charm-helpers. Since charms and Juju versions are
  decoupled
  from packages frozen in the archive, just having a version of
  charm-helpers in the archive for the version of Juju in that archive
  won't
  necessarily last the test of time. Someone deploying a charm now from a
  new
  version of Juju against

Re: The future of Charm Helpers

2015-08-12 Thread Adam Collard 
On Wed, 12 Aug 2015 at 12:12 Marco Ceppi marco.ce...@canonical.com wrote:

 Working with the Juju QA team to add a Jenkins job for test landing is a
 great idea, one we should certainly do now while we figure the rest of this
 out.


FYI there's an already an outstanding request with Juju QA team to set up
CI for charm-helpers.
--

 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: The future of Charm Helpers

2015-08-12 Thread Marco Ceppi 
On Wed, Aug 12, 2015 at 7:17 AM Adam Collard adam.coll...@canonical.com
wrote:

 On Wed, 12 Aug 2015 at 12:12 Marco Ceppi marco.ce...@canonical.com
 wrote:

 Working with the Juju QA team to add a Jenkins job for test landing is a
 great idea, one we should certainly do now while we figure the rest of this
 out.


 FYI there's an already an outstanding request with Juju QA team to set up
 CI for charm-helpers.


We should be able to help speed that along, we have access to create new
jobs in our Charm Testing jenkins slave. We'll take a look next week on how
quickly we can get a simple merge request approval/testing job setup and
reporting results back to the MP


 --

 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: The future of Charm Helpers

2015-08-12 Thread Marco Ceppi 
On Wed, Aug 12, 2015 at 6:11 AM Stuart Bishop stuart.bis...@canonical.com
wrote:

 On 11 August 2015 at 20:42, Marco Ceppi marco.ce...@canonical.com wrote:


  # Trimming down charm-helpers
 
  The first item, and arguably the largest is a complete reorganization of
 the
  current charm-helpers code base. Originally, charm helpers was setup to
  allow a path for developers to place unstable code (contrib directory)
 where
  they could iterate without a guarantee of API stability. However, this
  quickly grew to include code that simply didn't make sense as a core
  feature of charm-helpers. This includes a large set of code for domain
  specific charm ecosystems (bigdata, openstack, storage) as well as
 quickly
  fixes to missing features in Juju (leader election).
 
  The amount of collaborative code in contrib is fantastic! However, it's
  making the release and distribution of charm-helpers more complicated.
 With
  an inability to keep up with documentation for methods contrib and the
  volatile nature of unstable API, I'd like to propose we completely
  remove the contrib directory and instead urge code maintainers in
 contrib to
  create their own libraries for the feature and code they wish to maintain
  going forward which simply depends on charm-helpers for the code in core.
 
  This will allow groups like bigdata, openstack charmers, storage, and
 others
  to maintain their own release and review process for helpers specific to
  their solutions, as well as their own means of delivery (be it
  charm-helpers-sync, pypi, debian, or another means). This will also
 GREATLY
  slim down the charm-helpers library to instead include one of two things:
 
  - API in Python for communicating with Juju
  - Simple collection of tools in Python to solve shared universal problems
 
  Which roughly translates to hookenv.py, host.py, and fetch tools. Once
  contrib is removed and relocated I'd like to also propose flattening the
  library namespaces from
 
  charmhelpers.core.hookenv - charmhelpers.hookenv
  charmhelpers.core.host - charmhelpers.host
  charmhelpers.core.files - charmhelpers.host
  charmhelpers.core.fstab - charmhelpers.host
  charmhelpers.core.sysctl - charmhelpers.host
  charmhelpers.core.kv - separate project?
 
  Everything else, outside of contrib would remain the same. While I think
  it's important to slim down charm-helpers I'm not convinced the above
  outlined rename/code merges are the right or best way forward. I welcome
  feedback and a discussion around this.

 I really like the big tree, including contrib.

 I discover cool things in there, which I would not if the cool things
 were hidden away in another branch I know nothing about. More and more
 teams get created around the globe and communication between them
 grows less and less. A shared repository helps in a small way.


I can see how this helps for discoverability for those already pretty
intimate with the code base, but a lot of the key complaints we've gotten
around new authors getting started is there is just an overwhelming amount
of code. If contrib just became charmutils or some other big umbrella I'd
be better with that, though I still don't think it solves the problems of
versioning. If the BIg Data charmers want to publish a backwards
incompatible change how does that get broadcast from a contrib directory?
Most all of the modules in contrib have merit, I think having them placed
in a tree does a disservice to the code created. Most that I've come across
are actually pretty generic and could just be standalone python modules the
rest of the python community uses (ssl, for example).


 And your test harnesses are already setup for you. It is easy to add
 new cool stuff, and its tests. And those tests get run under both
 Python 2 and Python 3 with the same versions of dependencies and your
 cool stuff is much more likely to work with other peoples cool stuff.
 I know for a fact if things get spit out they will have less rigorous
 testing, and we will end up with tech debt Py2 only modules for
 example, or conflicting dependencies.


If we provide similar scaffolding as we have now for those projects I don't
see why they wouldn't continue this way, if people don't take pride in
their projects they either don't get used or get forked and made better


 But I don't see the large tree as a problem. Its not unwieldy. And you
 don't have to worry about anything dumped into contrib, except from
 the perspective of if it is worth migrating it to the core area and
 how much work it would be (documentation, ensuring consistency,
 maintainability etc.). I don't see the large tree as a problem because
 you next talk about packaging.


From my perspective charm helpers should be core juju components mapped to
that language, tools to assist with writing charms in that language
(decorators, etc), and modules to help charms adhere to the charm policy.
I'd let things like host.py continue because they're small convenience
features.