Re: Introducing, juju resources!

[Adam Collard]

Hi Moonstone!

On Fri, 12 Feb 2016 at 21:27 Katherine Cox-Buday <
katherine.cox-bu...@canonical.com> wrote:

> Moonstone have been working hard on a new feature coming up in Juju 2.0
> called "Juju Resources", and we're now at a point where we can share the
> goodness and call for bugs/feedback! As noted in the video linked below,
> the feature isn't quite complete, so expect some rough edges, and for some
> of the CLI details to shift just a bit.
>

After watching the demo (thanks for sharing!) it struck me that use of the
update-status hook to demonstrate this was pointing to a missing feature
(IMHO).

Is there a hook that fires when a resource is made available? If I have an
install hook that requires a resource, I could make it transition to
blocked when that resource is missing. But when that resource is provided,
how do I know to try again?

Thanks,

Adam
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Introducing, juju resources!

[Aaron Bentley]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 2016-02-14 01:40 PM, Rick Harding wrote:
> 
> 
> On Sun, Feb 14, 2016 at 12:50 PM Aaron Bentley Yes, you can work
> around this with tarfiles, but why do we want to? It's a pain to
> build a tar file every time a single dependency changes.  It's
> easier to use S3 instead for a particular use case, but it doesn't
> solve the general case.
> 
> 
> I'd push back a little bit here though. We do this in a couple of
> python web applications. We keep a repo of 'download-cache' which
> is the current pypi .tar.gz and use that built into a single
> download-cache tarball for deliver in the charms. This would be one
> resource. The second resource would then be the actual python
> application. It would have the makefile, requirements.txt file,
> source code, etc. It would depend on the download-cache file being
> there and so with the two, you'd manage both the source code and
> the dependencies as two different resource files that are all
> updated individually from the actual charm itself.

I would call that "working around this with tarfiles".  I still think
that it's more convenient to use S3 than to have to update the
dependencies tarfile every time a resource changes, and I think that
being less convenient that alternatives will hinder uptake.

Aaron
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJWwMw9AAoJEK84cMOcf+9hX3EH+gKybA2e5bWJBvCOURgv6Ep2
esVvugT7h5bW84zJUX7He+OZLM+PQV9RcWo10A+SjN3U+AVS+zgIHIlXXTDmLb4N
orP8nHsgTptYuv52kNmrycASY+QDUTBrr1KhxtUjPobMO2DW4dwrbXRzWKD0U2rY
1oHH5KE6CjnRRFJGnXik6OOoP9Xskty3sOguBw9KVISjLlh4k0kl3EKRS00ED7KY
PgBC/EKjD1+mLFt+TA9m+Ds6y/dSco0yWVNgbjBZ18ftxY6g4J4LCd1Sj5zXW03q
Ic/iNWj3gI89dV1TGVx7VNqSex7lKS1AYODjeZC3saPcJ3oIRV97WxA7D66wRQw=
=p32x
-END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Introducing, juju resources!

[Rick Harding]

On Sun, Feb 14, 2016 at 12:50 PM Aaron Bentley 
wrote:

> Another use case is when you want to create a charm deploys Python
> code.  You want it to use "resources" instead of downloading
> dependencies from PyPI.
>
> 1. The list of resources can change over time.  You don't want the
> charm to have to change every time the deployed software adds/removes
> a dependency.
>
> 2. The version numbers are embedded in the filenames.  You don't want
> the charm to have to change ever time deployed software changes its
> version numbers.
>
> Yes, you can work around this with tarfiles, but why do we want to?
> It's a pain to build a tar file every time a single dependency
> changes.  It's easier to use S3 instead for a particular use case, but
> it doesn't solve the general case.
>

I'd push back a little bit here though. We do this in a couple of python
web applications. We keep a repo of 'download-cache' which is the current
pypi .tar.gz and use that built into a single download-cache tarball for
deliver in the charms. This would be one resource. The second resource
would then be the actual python application. It would have the makefile,
requirements.txt file, source code, etc. It would depend on the
download-cache file being there and so with the two, you'd manage both the
source code and the dependencies as two different resource files that are
all updated individually from the actual charm itself.

So if a dependency is updated you can just update the source/download cache
resources and not the charm. If the source code is updated and no deps are
updated you just update the source code.

Again, there's room to improve the idea for the 'many resources' ideas, but
the current work moves things forward into a nice direction reducing the
need for fat charms and allowing some flexibility.
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Introducing, juju resources!

[Aaron Bentley]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 2016-02-13 09:10 AM, Rick Harding wrote:
> Your read is correct. You must declare the resources. It's helpful
> to users to know what to stick in there and for the charm to be
> able to handle different items. In your case, a single tar file of
> the plugins you'd like to enable could be sent up and the charm
> would be able to untar, loop through them, etc. Ideally the charm
> would be uploaded to the store with a standard set of plugins and
> folks could then customize which ones they wanted by creating their
> own tar of plugins.

Another use case is when you want to create a charm deploys Python
code.  You want it to use "resources" instead of downloading
dependencies from PyPI.

1. The list of resources can change over time.  You don't want the
charm to have to change every time the deployed software adds/removes
a dependency.

2. The version numbers are embedded in the filenames.  You don't want
the charm to have to change ever time deployed software changes its
version numbers.

Yes, you can work around this with tarfiles, but why do we want to?
It's a pain to build a tar file every time a single dependency
changes.  It's easier to use S3 instead for a particular use case, but
it doesn't solve the general case.

Aaron
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJWwL5XAAoJEK84cMOcf+9hVLQIAI5q7epn8UKtAILu9E41PoPK
CzVE+8gkWHEOZ9fOpfC8rXGVl1MmF279vx7o5eEYaXq8LNIL4yQx4rg4BahnkIhn
oO5LwPLXTp6c+XJ6muofqQZtvm80HJO3AcsuYbdrQoBDrQrNqjPr4yMXZi0BJ1c1
iGmqNTgl4IWfYJtxMGmzV7qBLrjepG17lClHSNFpMz4rpftZje7H0utPsoFvFRcU
ya0LiGiKjTgkBfPWuopgO6cBZ8Y0FVPV94tyLc67PNrjSYDYKeSrfMCAMdHZ4Gh9
kMM4pnEjX0lT01d8A8PcecbmUDRJ1NLHnzDW1VeJxDXVrzA/vIMF+QWc0EUJKCo=
=BEaN
-END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Introducing, juju resources!

[Rick Harding]

On Fri, Feb 12, 2016 at 11:06 PM Aaron Bentley 
wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> On 2016-02-12 04:26 PM, Katherine Cox-Buday wrote:
> > Moonstone have been working hard on a new feature coming up in Juju
> > 2.0 called "Juju Resources", and we're now at a point where we can
> > share the goodness and call for bugs/feedback!
>
> I'm glad to see you folks are working on this. On the QA side, we've
> been wanting Juju to support some kind of native blob storage,
> basically since we started.
>
> Is is necessary to enumerate the names in the charm? I'd rather we
> didn't. One of our use cases is plugins for Jenkins. The Jenkins charm
> has has a setting that tells it which plugins to install, but there's
> no predetermined list. It would be nice if the plugins could be
> retrieved in a Juju-native way, but it would be painful to have to
> update the charm every time we added a new plugin.  And it would be
> bad if people ended up forking the charm just to adjust the list of
> plugins.
>
> We could bundle the plugins in a single tarfile, but then updating the
> tarfile would be annoying. For plugins, it's best if each plugin is
> its own file and there are no restrictions on the filenames.


Your read is correct. You must declare the resources. It's helpful to users
to know what to stick in there and for the charm to be able to handle
different items. In your case, a single tar file of the plugins you'd like
to enable could be sent up and the charm would be able to untar, loop
through them, etc. Ideally the charm would be uploaded to the store with a
standard set of plugins and folks could then customize which ones they
wanted by creating their own tar of plugins.

I'll make sure we keep this in mind though. We've started with one file per
resource, and maybe there's an idea here of "one or more" where you might
have a charm that takes a data set file. You can deploy with a data set, or
multiple files of data that all need to be processed, rather than building
into a single file. We'll have to think it through for a future iteration
of resources.

Rick
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Introducing, juju resources!

[Nate Finch]

One idea is to have a resource type of "directory" and upload everything in
that directory.  The current code has only one resource type - file.   But
it's copied to a directory on the unit named by the resource. So it would
be pretty easy to jnstsd copy a number of files to that directory.

Of course, this would be a resources v2 feature. I'm sure there's rework
that would be needed on the back end, but it certainly seems doable.  We
also had ideas for making a github repo as a resource, which might be an
even better UX, depending on your use case.

On Sat, Feb 13, 2016, 9:10 AM Rick Harding 
wrote:

> On Fri, Feb 12, 2016 at 11:06 PM Aaron Bentley <
> aaron.bent...@canonical.com> wrote:
>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA256
>>
>> On 2016-02-12 04:26 PM, Katherine Cox-Buday wrote:
>> > Moonstone have been working hard on a new feature coming up in Juju
>> > 2.0 called "Juju Resources", and we're now at a point where we can
>> > share the goodness and call for bugs/feedback!
>>
>> I'm glad to see you folks are working on this. On the QA side, we've
>> been wanting Juju to support some kind of native blob storage,
>> basically since we started.
>>
>> Is is necessary to enumerate the names in the charm? I'd rather we
>> didn't. One of our use cases is plugins for Jenkins. The Jenkins charm
>> has has a setting that tells it which plugins to install, but there's
>> no predetermined list. It would be nice if the plugins could be
>> retrieved in a Juju-native way, but it would be painful to have to
>> update the charm every time we added a new plugin.  And it would be
>> bad if people ended up forking the charm just to adjust the list of
>> plugins.
>>
>> We could bundle the plugins in a single tarfile, but then updating the
>> tarfile would be annoying. For plugins, it's best if each plugin is
>> its own file and there are no restrictions on the filenames.
>
>
> Your read is correct. You must declare the resources. It's helpful to
> users to know what to stick in there and for the charm to be able to handle
> different items. In your case, a single tar file of the plugins you'd like
> to enable could be sent up and the charm would be able to untar, loop
> through them, etc. Ideally the charm would be uploaded to the store with a
> standard set of plugins and folks could then customize which ones they
> wanted by creating their own tar of plugins.
>
> I'll make sure we keep this in mind though. We've started with one file
> per resource, and maybe there's an idea here of "one or more" where you
> might have a charm that takes a data set file. You can deploy with a data
> set, or multiple files of data that all need to be processed, rather than
> building into a single file. We'll have to think it through for a future
> iteration of resources.
>
> Rick
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Introducing, juju resources!

[Katherine Cox-Buday]

Hey Gabbey!

On 02/12/2016 03:57 PM, Gabriel Samfira wrote:

Is there a limit on the filesize I can upload?

Not to my knowledge; they are streamed to the controller.
What happens if multiple users try to upload the same file (does it 
get stored twice, or does juju just increment a reference counter)?
For the moment, the bytes will always be uploaded so that it can 
calculate a fingerprint. If the fingerprint is the same, it won't store 
a new copy. I'm not sure if it will consider the duped version a new 
revision or not; I'll have to check on that.


Do I need to attach more storage to the controller if I need to upload 
a bunch of ISO files (3-4 ISO files of about 3 GB each)? On openstack, 
the default is a small instance with 20 GB of disk space.
We're storing it in Juju's blobstore which has deduping, so I believe 
you'll need some amount less than the sum of the size of all your files. 
But yes, ultimately it's cached on the controller so you'll want to bump 
the size of that up if you're planning on storing lots of resources.

Thanks for this great feature!

And thanks for the great questions :)
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Introducing, juju resources!

[Gabriel Samfira]

Outstanding stuff!

I know this is still early :D. I will probably be abusing this feature a lot in 
the comming weeks, and want to get as familiar with it as possible. So here are 
a few questions:

Is there a limit on the filesize I can upload?
What happens if multiple users try to upload the same file (does it get stored 
twice, or does juju just increment a reference counter)?
Do I need to attach more storage to the controller if I need to upload a bunch 
of ISO files (3-4 ISO files of about 3 GB each)? On openstack, the default is a 
small instance with 20 GB of disk space.

Thanks for this great feature!

Gabriel

On Vi, 2016-02-12 at 15:26 -0600, Katherine Cox-Buday wrote:
Hey all!

Moonstone have been working hard on a new feature coming up in Juju 2.0 called 
"Juju Resources", and we're now at a point where we can share the goodness and 
call for bugs/feedback! As noted in the video linked below, the feature isn't 
quite complete, so expect some rough edges, and for some of the CLI details to 
shift just a bit.

This has landed in master, but if you don't feel like compiling you can install 
the devel PPA. Then, on ~Feb. 18th 2.0-beta1 will drop and you'll be able to 
play around with everything we've highlighted!

To see what all the buzz is about, have a look here:

https://www.youtube.com/watch?v=SS3AQO3ZN9Y

Thanks, and happy Jujuing! :)

-
Katherine

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Introducing, juju resources!

[Aaron Bentley]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 2016-02-12 04:26 PM, Katherine Cox-Buday wrote:
> Moonstone have been working hard on a new feature coming up in Juju
> 2.0 called "Juju Resources", and we're now at a point where we can
> share the goodness and call for bugs/feedback!

I'm glad to see you folks are working on this. On the QA side, we've
been wanting Juju to support some kind of native blob storage,
basically since we started.

Is is necessary to enumerate the names in the charm? I'd rather we
didn't. One of our use cases is plugins for Jenkins. The Jenkins charm
has has a setting that tells it which plugins to install, but there's
no predetermined list. It would be nice if the plugins could be
retrieved in a Juju-native way, but it would be painful to have to
update the charm every time we added a new plugin.  And it would be
bad if people ended up forking the charm just to adjust the list of
plugins.

We could bundle the plugins in a single tarfile, but then updating the
tarfile would be annoying. For plugins, it's best if each plugin is
its own file and there are no restrictions on the filenames.

Aaron
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJWvquhAAoJEK84cMOcf+9h/k8IAMtadIceMAUOibSXSuw96WqM
n3jv6Vd6/UzorzieA66yCrquW+gkscnnDuLedEoRCFLn5lJZT2cWpPcnI/tdonB6
l3bF5ViIrrebvf+wWvc7r3Ep3Y/3NBKYtznwbjfHJuJoZcDWTKR/6lgflD5GAomX
gZyZhP6U/+dE0TfhFBoG3hsqkRludBSI7fzIlw8X32mBclvJlTtvWikuhnwbvhI/
ao+9VCR7iFumUX7wfDksNrRKLrA9rxitlKRhTxVUsgEbtGImwu6B/G8eOZ58kFIy
C9QuO5MwW3BCwgJ9NyiQTCXUcOtbSyRHSfnb/BwbbI3A9i5DABDwIcCrIKc9Zmo=
=R0j6
-END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev