On Wed, Aug 12, 2015 at 6:11 AM Stuart Bishop <stuart.bis...@canonical.com> wrote:
> On 11 August 2015 at 20:42, Marco Ceppi <marco.ce...@canonical.com> wrote: > > > > # Trimming down charm-helpers > > > > The first item, and arguably the largest is a complete reorganization of > the > > current charm-helpers code base. Originally, charm helpers was setup to > > allow a path for developers to place unstable code (contrib directory) > where > > they could iterate without a guarantee of API stability. However, this > > quickly grew to include code that simply didn't make sense as a "core" > > feature of charm-helpers. This includes a large set of code for domain > > specific charm ecosystems (bigdata, openstack, storage) as well as > quickly > > fixes to missing features in Juju (leader election). > > > > The amount of collaborative code in contrib is fantastic! However, it's > > making the release and distribution of charm-helpers more complicated. > With > > an inability to keep up with documentation for methods contrib and the > > "volatile" nature of "unstable API", I'd like to propose we completely > > remove the contrib directory and instead urge code maintainers in > contrib to > > create their own libraries for the feature and code they wish to maintain > > going forward which simply depends on charm-helpers for the code in core. > > > > This will allow groups like bigdata, openstack charmers, storage, and > others > > to maintain their own release and review process for helpers specific to > > their solutions, as well as their own means of delivery (be it > > charm-helpers-sync, pypi, debian, or another means). This will also > GREATLY > > slim down the charm-helpers library to instead include one of two things: > > > > - API in Python for communicating with Juju > > - Simple collection of tools in Python to solve shared universal problems > > > > Which roughly translates to hookenv.py, host.py, and fetch tools. Once > > contrib is removed and relocated I'd like to also propose flattening the > > library namespaces from > > > > charmhelpers.core.hookenv -> charmhelpers.hookenv > > charmhelpers.core.host -> charmhelpers.host > > charmhelpers.core.files -> charmhelpers.host > > charmhelpers.core.fstab -> charmhelpers.host > > charmhelpers.core.sysctl -> charmhelpers.host > > charmhelpers.core.kv -> separate project? > > > > Everything else, outside of contrib would remain the same. While I think > > it's important to slim down charm-helpers I'm not convinced the above > > outlined rename/code merges are the right or best way forward. I welcome > > feedback and a discussion around this. > > I really like the big tree, including contrib. > > I discover cool things in there, which I would not if the cool things > were hidden away in another branch I know nothing about. More and more > teams get created around the globe and communication between them > grows less and less. A shared repository helps in a small way. > > I can see how this helps for discoverability for those already pretty intimate with the code base, but a lot of the key complaints we've gotten around new authors getting started is there is just an overwhelming amount of code. If contrib just became charmutils or some other big umbrella I'd be better with that, though I still don't think it solves the problems of versioning. If the BIg Data charmers want to publish a backwards incompatible change how does that get broadcast from a contrib directory? Most all of the modules in contrib have merit, I think having them placed in a tree does a disservice to the code created. Most that I've come across are actually pretty generic and could just be standalone python modules the rest of the python community uses (ssl, for example). > And your test harnesses are already setup for you. It is easy to add > new cool stuff, and its tests. And those tests get run under both > Python 2 and Python 3 with the same versions of dependencies and your > cool stuff is much more likely to work with other peoples cool stuff. > I know for a fact if things get spit out they will have less rigorous > testing, and we will end up with tech debt Py2 only modules for > example, or conflicting dependencies. > If we provide similar scaffolding as we have now for those projects I don't see why they wouldn't continue this way, if people don't take pride in their projects they either don't get used or get forked and made better > But I don't see the large tree as a problem. Its not unwieldy. And you > don't have to worry about anything dumped into contrib, except from > the perspective of if it is worth migrating it to the core area and > how much work it would be (documentation, ensuring consistency, > maintainability etc.). I don't see the large tree as a problem because > you next talk about packaging. > > >From my perspective charm helpers should be core juju components mapped to that language, tools to assist with writing charms in that language (decorators, etc), and modules to help charms adhere to the charm policy. I'd let things like host.py continue because they're small convenience features. If that's the library we put forth and the guidelines around what should be in each language specific library documentation, replication, and comprehension of the library are much easier. > > > # Packaging charm-helpers > > > > Currently, charm-helpers are available via either bazaar source or PyPI. > > With a more stable, slim, and maintainable code base we would like to > > package charm-helpers still on PyPI with semantic versioning, make a > > dist/wheel[0] available for embedding (instead of charm-helpers-sycn), > and > > in Debian and have them available via a PPA but also potentially in the > > archive proper. > > Sounds cool. You don't have to package the entire tree. Just package > the core section. > > If you are really enthusiastic, you could have the contrib modules > packaged separately if they provide a working setup.py. But wouldn't > bother. There will be lots of cases where we can't use the packages, > and using unstable contrib modules would just be one of them. > > The main reason I will not be using your proposed packages for quite > some time is that I like to add things to charm-helpers and fix bugs > while working on my charms. I can't do that if I need to wait for a > release cycle, even if you could get it down to 24 hours. I will > remain using charm-helpers-sync.py, until I can switch to nested git > branches, but that is fine. Packages will still make lots of other > people's lives easier. > One of my goals is to burn charm-helpers-sync with fire. The amount of noise it adds to charm reviews is tedious at best and unruly at it's worst. There's no way for us to validate if the user just modified a file in the tree or actually sync'd from the upstream. Providing a way to generate a wheel from the charm-helpers source tree is how I'd like to get past this so we can continue to allow the "embed charm helpers" workflow while addressing these issue. A 24 hour release cycle would be great but not likely. I think a weekly patch release cadence is more obtainable, but I see your point about working with charms. Cory also made some good points about how the composer feature for charm-tools being developed can help alleviate this as well. > > The hardest challenge will be keeping the archive as up to date as > possible > > with releases of charm-helpers. Since charms and Juju versions are > decoupled > > from packages "frozen" in the archive, just having a version of > > charm-helpers in the archive for the version of Juju in that archive > won't > > necessarily last the test of time. Someone deploying a charm now from a > new > > version of Juju against trusty would get the 1.21 equivalent of > > charm-helpers which may not include the new leadership or extended status > > methods causing the charm to fail erroneously. > > > > Having a PPA can alleviate this, but PyPI and PPAs may interfere with > > firewalls and policies where Juju is being deployed. Opinions on how to > help > > with this are appreciated. Otherwise, we may simply recommend authors to > use > > a bit of bootstrap code which attempts to install from PyPI, with a > fallback > > to install an embedded version of charmhelpers that was included in the > > charm as a dist/wheel. > > Could we use simplestreams for this? Or whatever the mechanism is that > Juju uses to install the correct version of the juju tools tarball > when provisioning a unit? > This is a great idea, I'll look into this more since it's a common convention already with Juju. I think Juju growing the notion of charm-helpers as a first class citizen and the charm either declaring what helpers it needs (language/version) or something similar could be really useful. There's also been talk of using juju resources for this when that feature goes live, which would be a simplestream for embedded objects in the charm. > deb packages are unsuitable, as a subordinate charm may require a > different version to the host charm. I think it has to be a wheel or > tarball, and installed in a unit specific location rather than > stuffing it into the system Python library. > You're right, I think we may just need to abandon debian packaging for this all together. > > Its not just bootstrapping on install either. upgrade-charm will need > to update the package too before it is imported. > > > > # Charm-helpers for other languages > > > > We already have charm-helpers written for three languages! Python[0], > > PowerShell[1], and Rust-lang[2]. By sliming down the charm-helpers we can > > define a minimum requirement that each new charm-helper language should > > expose. This will enable us to document charm-helpers more thoroughly and > > include code examples for each language supported in our docs. Also, by > > having a narrowly defined scope of methods, it enables other communities > to > > create helpers for their languages. > > Slimming down. I've bitched before about too much stuff being in > charmhelpers.core, and 'helped' by landing my own stuff I don't think > should be core in the parent package. I tend to think > charmhelpers.core should provide wrappers around the juju hook > environment, matching the documented juju hook environment behavior, > and required low level primitives you need for writing charms. It is > the foundation for the rest of charm-helpers, which can provide higher > level and friendlier interfaces. I believe that these higher level > tools should be Pythonic. So your cli wrappers can publish > charmhelpers.core for people insane or uneducated enough to write Bash > charms, and the trivial charmhelpers.core would be trivially ported to > other high level languages. But porting the Python decorators provided > by charmhelpers.coordinator or the Pythonic datastructures provided by > charmhelpers.context would not be helpful. Nor would porting > charmhelpers.core.templating, since it is currently Jinja2 specific > (and why it should be charmhelpers.templating). > > I agree with the first half of this statement, there is too much in core and core should be basically hookenv. I think, however, that most of the other top level modules you've talked about should just be their own packages. I'll address your bash observations in the next few chunks. > > > The PowerShell library was written by the folks at CloubBase, I had the > > chance to sit down with them a few months back and review how their > > structure looks. It's already very far along, includes unit testing, and > > matches the relative structure we have had in the Python charm helpers > for > > some time. Going forward I'd like to continue this trend and sent some > > guidelines around future charm-helper libraries for other languages. > > > > # More exposure via the CLI > > > > Finally, we have a CLI interface for the charm-helpers python library. > > Recently there have been several merges which add exposure to additional > > methods and features of python charm-helpers. I'd like to continue this > tend > > and add a CLI hook for everything in charm-helpers, within reason. A > recent > > addition, by Cory Johns, enabled the internal key-value store in > > charm-helpers to be accesible via CLI. While making things like the > > "config_get" method available by CLI is silly, other key and core methods > > that exist as code in Python only and aren't a 1:1 bridge to Juju > commands > > should also be made available. This will allow authors creating first > drafts > > of charms in bash to access these higher methods. > > Are you being helpful or are you being an enabler? At the risk of > derailing the discussion, I do wonder if we are doing people a > disservice by pretending that good, robust, maintainable charms can be > written in Bash. Sure, it is technically possible, but even simple > charms are complex, people capable or writing complex code in Bash are > rare, and the probability finding a team able to maintain it > approaches zero. We have spent a lot of time rewriting Bash charms in > Python purely to fix bugs and add features, because it was easier. Not > bitter, at all. > Yes, we all agree that charms in bash are not maintainable for large software projects overtime. Most of our big hitting charms that started in bash have all been rewritten. We will of course highlight in the documentation that more...robust languages should be used when writing charms. However, one major point you're missing and one I failed to highlight is that by making methods available via a CLI, other libraries do not have to re-implement the code inside of them. Instead they just build a shim around shelling out to that command. Since Python can be installed on every distro we support this works well for the PowerShell and Rust libraries we currently have. > > > # Conclusion > > > > I welcome discussion, dissent, and suggestions around these topics. They > are > > by no means set in stone but are the assumed plan at the moment. > > Ha ;) > Thanks for the feedback! I appreciate the replies and contributions to the project thus far > > > TL;DR: I want to make charm-helpers very slim, versioned, packaged, > > documented, and easily portable to other languages. > > TL;DR: Cool. Save work by keeping contrib where it is and just not > packaging it. Consider charmhelpers.core low level and charmhelpers > root high level, with only charmhelpers.core exposed by cli. Move > unsuitable parts of charmhelpers.core to the top level module. > TL;DR: Sweet. I think we're mostly on the same page but I still have some concerns having all the current code remain in the project and it's an opinion I hold and is feedback I've gotten from existing charm authors and charm-helper maintainers Marco Ceppi
-- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
