Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/22/2014 04:39 AM, Fox, Kevin M wrote: Simply having a git repository does not imply that its source. In fact, if its considered compiled (minified), I'm thinking the debian rules would prevent sourcing from it? Thanks, Kevin Indeed, we don't package minified sources. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/21/2014 08:31 PM, Donald Stufft wrote: On Nov 21, 2014, at 3:59 AM, Thomas Goirand z...@debian.org wrote: I'm not sure I understand the meaning behind this question. bower install angular downloads a bower package called angular. Isn't there is a simple URL that I may use with wget? I don't really want to use bower directly, I just would like to have a look to the content of the bower package. You can’t. Bower doesn’t have “traditional” packages where you take a directory and archive it using tar/zip/whatever and then upload it to some repo. Bower has a registry which maps names to git URLs and then the bower CLI looks up that mapping, fetches the git repository and then uses that as the input to the “look at metadata and do stuff with files” part of the package manager instead of the output of an un-unarchival command. Then this makes Bower a very bad candidate to debianize stuff. We'll have a moving target with a constantly changing git from upstream, meaning that we'll have all sorts of surprise in the gate. Frankly, this Bower thing scares me, and I don't really understand why we're not continuing to use XStatic stuff, which was really convenient and has been proven to work during the Juno cycle. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/25/2014 03:40 PM, Richard Jones wrote: On Wed Nov 26 2014 at 3:36:27 AM Thomas Goirand z...@debian.org wrote: On 11/21/2014 08:31 PM, Donald Stufft wrote: On Nov 21, 2014, at 3:59 AM, Thomas Goirand z...@debian.org wrote: I'm not sure I understand the meaning behind this question. bower install angular downloads a bower package called angular. Isn't there is a simple URL that I may use with wget? I don't really want to use bower directly, I just would like to have a look to the content of the bower package. You can’t. Bower doesn’t have “traditional” packages where you take a directory and archive it using tar/zip/whatever and then upload it to some repo. Bower has a registry which maps names to git URLs and then the bower CLI looks up that mapping, fetches the git repository and then uses that as the input to the “look at metadata and do stuff with files” part of the package manager instead of the output of an un-unarchival command. Then this makes Bower a very bad candidate to debianize stuff. We'll have a moving target with a constantly changing git from upstream, meaning that we'll have all sorts of surprise in the gate. It's no more constantly moving than any other project. Bower versions are tied to git tags. In fact, since debian etc. usually go to the repository rather than use release tarballs, bower *improves* things by requiring the tags, making it easier for you to isolate the version in the repository that you need, whereas otherwise people just have to remember to tag and often don't :) Frankly, this Bower thing scares me, and I don't really understand why we're not continuing to use XStatic stuff, which was really convenient and has been proven to work during the Juno cycle. We're doing this so we don't have the additional burden of creating and maintaining the xstatic packages. Also, it's _very_ standard javascript tooling. It's what javascript devs use. So sooner or later there's going to need to be a proper story for it in Debian if Debian wants to continue to be able to provide value around applications written in javascript. Might as well be the trailblazers, no? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/21/2014 01:51 PM, Richard Jones wrote: On 21 November 2014 16:12, Thomas Goirand z...@debian.org mailto:z...@debian.org wrote: On 11/21/2014 10:52 AM, Richard Jones wrote: On 11/18/2014 04:22 PM, Radomir Dopieralski wrote: If we use Bower, we don't need to use Xstatic. It would be pure overhead. Bower already takes care of tracking releases and versions, and of bundling the files. All we need is a simple line in the settings.py telling Django where it puts all the files -- we don't really need Xstatic just for that. The packagers can then take those Bower packages and turn them into system packages, and just add/change the paths in settings.py to where they put the files. All in one place. The issue is that there's often not just a single path, but a full directory structure to address. That is easily managed with a Debian xstatic package, not sure how it would be with Bower. I'm not sure what the difference is (unless it's just related to the Debian-specific historical issue you raise below). xstatic and bower are remarkably similar a directory to be packaged and some meta-data describing it. Let me explain again then. Let's say there's python-xstatic-foo, and libjs-foo in Debian. If the directory structure of libjs-foo is very different from xstatic-foo, I can address that issue with symlinks within the xstatic package. Just changing the BASE_DIR may not be enough, as libjs-foo may have files organized in a very different way than in the upstream package for foo. OK, so python-xstatic-foo can depend on libjs-foo and just symlink, fair enough. I'm still not sure what makes bower unique in this respect, I was under the impression that I wouldn't be able to do the same symlink thing with Bower. If I am, then great! although it'd be nice to avoid creating additional packages just to symlink things around for bower, which I think is what you're getting at. Just to make sure: we're not moving away from the current already existing xstatic packages are we? Also, yes, if I can avoid to have a bower package, that'd be great. But not sure how. It worked with the XStatic packages, and my main concern about switching to bower is exactly that: how is it going to work compared to XStatic stuff. Again; bower is not npm! Jasmine is a command-line program which is packaged by npm. Bower packages bundles of stuff that are included in web applications. bower itself, a command-line tool, is packaged by npm, but itself manages other packages which are not command-line things, but just bundles of css, javascript, images, fonts, etc. that are resources for web applications to use. Sure. But how do I download a bower package then? I'm not sure I understand the meaning behind this question. bower install angular downloads a bower package called angular. Isn't there is a simple URL that I may use with wget? I don't really want to use bower directly, I just would like to have a look to the content of the bower package. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 21/11/14 06:12, Thomas Goirand wrote:
Let's say there's python-xstatic-foo, and libjs-foo in Debian. If the
directory structure of libjs-foo is very different from xstatic-foo, I
can address that issue with symlinks within the xstatic package. Just
changing the BASE_DIR may not be enough, as libjs-foo may have files
organized in a very different way than in the upstream package for foo.
You can do it without xstatic even more easily, without having to muck
around with symlinks. Django has this STATICFILES_DIRS setting variable,
that you can set, that lets you specify on what URL each static
directory should be available. For instance, I use it currently to work
around the issue with jquery-ui changing directory structure between
versions:
if xstatic.main.XStatic(xstatic.pkg.jquery_ui).version.startswith('1.10.'):
# The 1.10.x versions already contain the 'ui' directory.
STATICFILES_DIRS.append(
('horizon/lib/jquery-ui',
xstatic.main.XStatic(xstatic.pkg.jquery_ui).base_dir))
else:
# Newer versions dropped the directory, add it to keep the path the
same.
STATICFILES_DIRS.append(
('horizon/lib/jquery-ui/ui',
xstatic.main.XStatic(xstatic.pkg.jquery_ui).base_dir))
But in your Debian package, you can completely drop xstatic and just use
absolute paths to the filesystem explicitly.
--
Radomir Dopieralski
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 21, 2014, at 3:59 AM, Thomas Goirand z...@debian.org wrote: I'm not sure I understand the meaning behind this question. bower install angular downloads a bower package called angular. Isn't there is a simple URL that I may use with wget? I don't really want to use bower directly, I just would like to have a look to the content of the bower package. You can’t. Bower doesn’t have “traditional” packages where you take a directory and archive it using tar/zip/whatever and then upload it to some repo. Bower has a registry which maps names to git URLs and then the bower CLI looks up that mapping, fetches the git repository and then uses that as the input to the “look at metadata and do stuff with files” part of the package manager instead of the output of an un-unarchival command. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 2014-11-21 07:31:36 -0500 (-0500), Donald Stufft wrote: You can’t. Bower doesn’t have “traditional” packages where you take a directory and archive it using tar/zip/whatever and then upload it to some repo. Bower has a registry which maps names to git URLs and then the bower CLI looks up that mapping, fetches the git repository and then uses that as the input to the “look at metadata and do stuff with files” part of the package manager instead of the output of an un-unarchival command. This raises interesting free software philosophy/license questions... how do I redistribute (or even examine) the source of a bower-managed package? Is there a way without actually reverse-engineering the toolchain? -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 21, 2014, at 11:32 AM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-11-21 07:31:36 -0500 (-0500), Donald Stufft wrote: You can’t. Bower doesn’t have “traditional” packages where you take a directory and archive it using tar/zip/whatever and then upload it to some repo. Bower has a registry which maps names to git URLs and then the bower CLI looks up that mapping, fetches the git repository and then uses that as the input to the “look at metadata and do stuff with files” part of the package manager instead of the output of an un-unarchival command. This raises interesting free software philosophy/license questions... how do I redistribute (or even examine) the source of a bower-managed package? Is there a way without actually reverse-engineering the toolchain? Well it’s a git repository, so you could just clone it and look at it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 2014-11-21 11:39:00 -0500 (-0500), Donald Stufft wrote: Well it’s a git repository, so you could just clone it and look at it. Aha, your earlier description made it sound like Bower was a file registry mapping to various random contents from a bunch of revision control repositories to assemble any one package. If Bower packages generally map back to one repository per package (even if there are multiple packages per repository) then that seems much more sane to deal with. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 21, 2014, at 11:57 AM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-11-21 11:39:00 -0500 (-0500), Donald Stufft wrote: Well it’s a git repository, so you could just clone it and look at it. Aha, your earlier description made it sound like Bower was a file registry mapping to various random contents from a bunch of revision control repositories to assemble any one package. If Bower packages generally map back to one repository per package (even if there are multiple packages per repository) then that seems much more sane to deal with. -- Jeremy Stanley Yea sorry, the bower registry (aka bower PyPI) is a mapping of name: git URL. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Simply having a git repository does not imply that its source. In fact, if its considered compiled (minified), I'm thinking the debian rules would prevent sourcing from it? Thanks, Kevin From: Donald Stufft [don...@stufft.io] Sent: Friday, November 21, 2014 8:39 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon On Nov 21, 2014, at 11:32 AM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-11-21 07:31:36 -0500 (-0500), Donald Stufft wrote: You can’t. Bower doesn’t have “traditional” packages where you take a directory and archive it using tar/zip/whatever and then upload it to some repo. Bower has a registry which maps names to git URLs and then the bower CLI looks up that mapping, fetches the git repository and then uses that as the input to the “look at metadata and do stuff with files” part of the package manager instead of the output of an un-unarchival command. This raises interesting free software philosophy/license questions... how do I redistribute (or even examine) the source of a bower-managed package? Is there a way without actually reverse-engineering the toolchain? Well it’s a git repository, so you could just clone it and look at it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
All the bower components I've used have included full source alongside any minified versions. On Sat, 22 Nov 2014 07:40 Fox, Kevin M kevin@pnnl.gov wrote: Simply having a git repository does not imply that its source. In fact, if its considered compiled (minified), I'm thinking the debian rules would prevent sourcing from it? Thanks, Kevin From: Donald Stufft [don...@stufft.io] Sent: Friday, November 21, 2014 8:39 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon On Nov 21, 2014, at 11:32 AM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-11-21 07:31:36 -0500 (-0500), Donald Stufft wrote: You can’t. Bower doesn’t have “traditional” packages where you take a directory and archive it using tar/zip/whatever and then upload it to some repo. Bower has a registry which maps names to git URLs and then the bower CLI looks up that mapping, fetches the git repository and then uses that as the input to the “look at metadata and do stuff with files” part of the package manager instead of the output of an un-unarchival command. This raises interesting free software philosophy/license questions... how do I redistribute (or even examine) the source of a bower-managed package? Is there a way without actually reverse-engineering the toolchain? Well it’s a git repository, so you could just clone it and look at it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/17/2014 06:54 PM, Radomir Dopieralski wrote:
- A tool, probably a script, that would help packaging the Bower
packages into DEB/RPM packages. I suspect the Debian/Fedora packagers
already have a semi-automatic solution for that.
Nop. Bower isn't even packaged in Debian. Though I may try to do it
(when I'm done with other Mirantis stuff like packaging Fuel for Debian...).
On 11/18/2014 07:59 AM, Richard Jones wrote:
I was envisaging us creating a tool which generates xstatic packages
from bower packages. I'm not the first to think along these
lines
http://lists.openstack.org/pipermail/openstack-dev/2014-March/031042.html
I think that's a very good idea!
I wrote the tool today, and you can find it here:
https://github.com/r1chardj0n3s/flaming-shame
AWESOME ! :)
Then now, everyone is happy. Thank you.
On 11/18/2014 04:22 PM, Radomir Dopieralski wrote:
If we use Bower, we don't need to use Xstatic. It would be pure
overhead. Bower already takes care of tracking releases and versions,
and of bundling the files. All we need is a simple line in the
settings.py telling Django where it puts all the files -- we don't
really need Xstatic just for that. The packagers can then take those
Bower packages and turn them into system packages, and just add/change
the paths in settings.py to where they put the files. All in one
place.
The issue is that there's often not just a single path, but a full
directory structure to address. That is easily managed with a Debian
xstatic package, not sure how it would be with Bower.
On 11/18/2014 06:36 PM, Richard Jones wrote:
I guess I got the message that turning bower packages into system
packages was something that the Linux packagers were not keen on.
What I'm not a fan of, is that we'll have external dependencies being
bumped all the time, with unexpected consequences. At least, with
xstatic packages, we control what's going on (though I understand the
overhead work problem).
By the way, I went into bower.io as I wanted to have a look. How do I
download a binary package for let's say jasmin? When searching, it just
links to github...
On 11/19/2014 12:14 AM, Radomir Dopieralski wrote:
We would replace that with:
STATICFILES_DIRS = [
('horizon/lib/angular',
os.path.join(BASE_DIR, 'bower_modules/angular'),
...
]
This would only work if upstream package directory structure is the same
as the one in the distribution. For historical reason, that's
unfortunately often not the case (sometimes because we want to keep
backward compatibility in the distro because of reverse dependencies),
and just changing the path wont make it work.
On 11/19/2014 03:43 AM, Richard Jones wrote:
+1 to all that, except I'd recommend using django-bower to handle the
static collection stuff. It's not documented but django-bower has a
setting BOWER_COMPONENTS_ROOT which would make the above transition much
simpler. You leave it alone for local dev, and packagers setup
BOWER_COMPONENTS_ROOT to '/usr/lib/javascript/' or wherever.
s/lib/share/
However, I'm almost sure that wont be enough to make it work. For
example, in Debian, we have /usr/share/javascript/angular.js, not just
/usr/share/javascript/angular. So django-bower would be searching on the
wrong path.
On 11/19/2014 12:25 PM, Richard Jones wrote:
In their view, bower components don't need to be in global-requirements:
- there are no other projects that use bower components, so we don't
need to ensure cross-project compatibility
- we can vet new versions of bower components as part of standard
Horizon change review
Maybe that's right for the OpenStack project, but that is a problem at
least for me, as I wont be constantly looking at Horizon dependencies,
just at the global-requirements.txt. So I'm afraid I may miss some new
stuff, and miss the deadline for the next release if I don't pay
attention to it. :(
Anyway, that's not so bad, I can try to adapt, but I just wanted to
raise my concern so that everyone knows about it.
Last thing: I'm currently a bit afraid of what will happen, as I don't
know the tools (bower and such). I wish I had a bit more time to test
them out, but I don't... :( So, I'm just raising my concerns even if
sometimes they may have no root, in the hope that we all find the best
solution that fits everyone. I hope the way I did in this thread is ok.
Cheers,
Thomas
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Fri Nov 21 2014 at 4:06:51 AM Thomas Goirand z...@debian.org wrote:
On 11/17/2014 06:54 PM, Radomir Dopieralski wrote:
- A tool, probably a script, that would help packaging the Bower
packages into DEB/RPM packages. I suspect the Debian/Fedora packagers
already have a semi-automatic solution for that.
Nop. Bower isn't even packaged in Debian. Though I may try to do it
(when I'm done with other Mirantis stuff like packaging Fuel for
Debian...).
Just to be clear, it's not bower itself (the command-line tool) that needs
packaging, just the components that bower itself packages.
On 11/18/2014 07:59 AM, Richard Jones wrote:
I was envisaging us creating a tool which generates xstatic packages
from bower packages. I'm not the first to think along these
lines
http://lists.openstack.org/pipermail/openstack-dev/2014-March/031042.html
I think that's a very good idea!
I wrote the tool today, and you can find it here:
https://github.com/r1chardj0n3s/flaming-shame
AWESOME ! :)
Then now, everyone is happy. Thank you.
Well, no, but at least it exists ;)
On 11/18/2014 04:22 PM, Radomir Dopieralski wrote:
If we use Bower, we don't need to use Xstatic. It would be pure
overhead. Bower already takes care of tracking releases and versions,
and of bundling the files. All we need is a simple line in the
settings.py telling Django where it puts all the files -- we don't
really need Xstatic just for that. The packagers can then take those
Bower packages and turn them into system packages, and just add/change
the paths in settings.py to where they put the files. All in one
place.
The issue is that there's often not just a single path, but a full
directory structure to address. That is easily managed with a Debian
xstatic package, not sure how it would be with Bower.
I'm not sure what the difference is (unless it's just related to the
Debian-specific historical issue you raise below). xstatic and bower are
remarkably similar a directory to be packaged and some meta-data describing
it.
On 11/18/2014 06:36 PM, Richard Jones wrote:
I guess I got the message that turning bower packages into system
packages was something that the Linux packagers were not keen on.
What I'm not a fan of, is that we'll have external dependencies being
bumped all the time, with unexpected consequences. At least, with
xstatic packages, we control what's going on (though I understand the
overhead work problem).
The dependencies won't be bumped any faster than reviewers allow, though I
realise that's not necessarily going to make you sleep any easier. Hmm.
By the way, I went into bower.io as I wanted to have a look. How do I
download a binary package for let's say jasmin? When searching, it
just links to github...
Again; bower is not npm! Jasmine is a command-line program which is
packaged by npm. Bower packages bundles of stuff that are included in web
applications. bower itself, a command-line tool, is packaged by npm, but
itself manages other packages which are not command-line things, but just
bundles of css, javascript, images, fonts, etc. that are resources for web
applications to use.
On 11/19/2014 12:14 AM, Radomir Dopieralski wrote:
We would replace that with:
STATICFILES_DIRS = [
('horizon/lib/angular',
os.path.join(BASE_DIR, 'bower_modules/angular'),
...
]
This would only work if upstream package directory structure is the same
as the one in the distribution. For historical reason, that's
unfortunately often not the case (sometimes because we want to keep
backward compatibility in the distro because of reverse dependencies),
and just changing the path wont make it work.
On 11/19/2014 03:43 AM, Richard Jones wrote:
+1 to all that, except I'd recommend using django-bower to handle the
static collection stuff. It's not documented but django-bower has a
setting BOWER_COMPONENTS_ROOT which would make the above transition much
simpler. You leave it alone for local dev, and packagers setup
BOWER_COMPONENTS_ROOT to '/usr/lib/javascript/' or wherever.
s/lib/share/
However, I'm almost sure that wont be enough to make it work. For
example, in Debian, we have /usr/share/javascript/angular.js, not just
/usr/share/javascript/angular. So django-bower would be searching on the
wrong path.
That is unfortunate. It may be that Debian therefore has to special-case
angular to handle that case.
I think the general idea of following the component pattern set by bower
(separate directories with no risk of conflicts, and using the bower.json
meta-data to allow automatic configuration of the component) is too good to
dismiss though. Far less work for everyone, including packagers.
Perhaps the new packages should have bower in their name?
On 11/19/2014 12:25 PM, Richard Jones wrote:
In their view, bower components don't need to be in global-requirements:
- there are no other projects that use bower components, so we
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/21/2014 10:52 AM, Richard Jones wrote: On 11/18/2014 04:22 PM, Radomir Dopieralski wrote: If we use Bower, we don't need to use Xstatic. It would be pure overhead. Bower already takes care of tracking releases and versions, and of bundling the files. All we need is a simple line in the settings.py telling Django where it puts all the files -- we don't really need Xstatic just for that. The packagers can then take those Bower packages and turn them into system packages, and just add/change the paths in settings.py to where they put the files. All in one place. The issue is that there's often not just a single path, but a full directory structure to address. That is easily managed with a Debian xstatic package, not sure how it would be with Bower. I'm not sure what the difference is (unless it's just related to the Debian-specific historical issue you raise below). xstatic and bower are remarkably similar a directory to be packaged and some meta-data describing it. Let me explain again then. Let's say there's python-xstatic-foo, and libjs-foo in Debian. If the directory structure of libjs-foo is very different from xstatic-foo, I can address that issue with symlinks within the xstatic package. Just changing the BASE_DIR may not be enough, as libjs-foo may have files organized in a very different way than in the upstream package for foo. By the way, I went into bower.io http://bower.io as I wanted to have a look. How do I download a binary package for let's say jasmin? When searching, it just links to github... Again; bower is not npm! Jasmine is a command-line program which is packaged by npm. Bower packages bundles of stuff that are included in web applications. bower itself, a command-line tool, is packaged by npm, but itself manages other packages which are not command-line things, but just bundles of css, javascript, images, fonts, etc. that are resources for web applications to use. Sure. But how do I download a bower package then? This would only work if upstream package directory structure is the same as the one in the distribution. For historical reason, that's unfortunately often not the case (sometimes because we want to keep backward compatibility in the distro because of reverse dependencies), and just changing the path wont make it work. On 11/19/2014 03:43 AM, Richard Jones wrote: +1 to all that, except I'd recommend using django-bower to handle the static collection stuff. It's not documented but django-bower has a setting BOWER_COMPONENTS_ROOT which would make the above transition much simpler. You leave it alone for local dev, and packagers setup BOWER_COMPONENTS_ROOT to '/usr/lib/javascript/' or wherever. s/lib/share/ However, I'm almost sure that wont be enough to make it work. For example, in Debian, we have /usr/share/javascript/angular.__js, not just /usr/share/javascript/angular. So django-bower would be searching on the wrong path. That is unfortunate. It may be that Debian therefore has to special-case angular to handle that case. I wasn't making a point about Angular here. It's a general issue we have to take care of addressing correctly. I think the general idea of following the component pattern set by bower (separate directories with no risk of conflicts, and using the bower.json meta-data to allow automatic configuration of the component) is too good to dismiss though. Far less work for everyone, including packagers. Perhaps the new packages should have bower in their name? There's already libjs-angularjs and a bunch of python-xstatic-angular packages in Debian. I'm not sure that it is necessary to *also* have a bower-angularjs packages. Why would I need to do that? Just for the .json file? If that's the case, then couldn't I just add the bower.json file in the existing libjs-* packages? I'm not sure I get the point here... Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 21 November 2014 16:12, Thomas Goirand z...@debian.org wrote: On 11/21/2014 10:52 AM, Richard Jones wrote: On 11/18/2014 04:22 PM, Radomir Dopieralski wrote: If we use Bower, we don't need to use Xstatic. It would be pure overhead. Bower already takes care of tracking releases and versions, and of bundling the files. All we need is a simple line in the settings.py telling Django where it puts all the files -- we don't really need Xstatic just for that. The packagers can then take those Bower packages and turn them into system packages, and just add/change the paths in settings.py to where they put the files. All in one place. The issue is that there's often not just a single path, but a full directory structure to address. That is easily managed with a Debian xstatic package, not sure how it would be with Bower. I'm not sure what the difference is (unless it's just related to the Debian-specific historical issue you raise below). xstatic and bower are remarkably similar a directory to be packaged and some meta-data describing it. Let me explain again then. Let's say there's python-xstatic-foo, and libjs-foo in Debian. If the directory structure of libjs-foo is very different from xstatic-foo, I can address that issue with symlinks within the xstatic package. Just changing the BASE_DIR may not be enough, as libjs-foo may have files organized in a very different way than in the upstream package for foo. OK, so python-xstatic-foo can depend on libjs-foo and just symlink, fair enough. I'm still not sure what makes bower unique in this respect, although it'd be nice to avoid creating additional packages just to symlink things around for bower, which I think is what you're getting at. By the way, I went into bower.io http://bower.io as I wanted to have a look. How do I download a binary package for let's say jasmin? When searching, it just links to github... Again; bower is not npm! Jasmine is a command-line program which is packaged by npm. Bower packages bundles of stuff that are included in web applications. bower itself, a command-line tool, is packaged by npm, but itself manages other packages which are not command-line things, but just bundles of css, javascript, images, fonts, etc. that are resources for web applications to use. Sure. But how do I download a bower package then? I'm not sure I understand the meaning behind this question. bower install angular downloads a bower package called angular. This would only work if upstream package directory structure is the same as the one in the distribution. For historical reason, that's unfortunately often not the case (sometimes because we want to keep backward compatibility in the distro because of reverse dependencies), and just changing the path wont make it work. On 11/19/2014 03:43 AM, Richard Jones wrote: +1 to all that, except I'd recommend using django-bower to handle the static collection stuff. It's not documented but django-bower has a setting BOWER_COMPONENTS_ROOT which would make the above transition much simpler. You leave it alone for local dev, and packagers setup BOWER_COMPONENTS_ROOT to '/usr/lib/javascript/' or wherever. s/lib/share/ However, I'm almost sure that wont be enough to make it work. For example, in Debian, we have /usr/share/javascript/angular.__js, not just /usr/share/javascript/angular. So django-bower would be searching on the wrong path. That is unfortunate. It may be that Debian therefore has to special-case angular to handle that case. I wasn't making a point about Angular here. It's a general issue we have to take care of addressing correctly. I think the general idea of following the component pattern set by bower (separate directories with no risk of conflicts, and using the bower.json meta-data to allow automatic configuration of the component) is too good to dismiss though. Far less work for everyone, including packagers. Perhaps the new packages should have bower in their name? There's already libjs-angularjs and a bunch of python-xstatic-angular packages in Debian. I'm not sure that it is necessary to *also* have a bower-angularjs packages. Why would I need to do that? Just for the .json file? If that's the case, then couldn't I just add the bower.json file in the existing libjs-* packages? I'm not sure I get the point here... The angular component that bower installs looks like this: $ ls -CAF bower_components/angular .bower.json angular-csp.css angular.min.js angular.min.js.map package.json README.md angular.js angular.min.js.gzip bower.json The bootstrap component looks like this: $ ls -CAF bower_components/boot/ .bower.json LICENSE bower.json fonts/ js/ package.json Gruntfile.js README.md dist/ grunt/
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 19/11/14 05:25, Richard Jones wrote: I've just had a long discussion with #infra folk about the global-requirements thing, which deviated (quite naturally) into a discussion about packaging (and their thoughts were in line with where Radomir and I are heading). In their view, bower components don't need to be in global-requirements: - there are no other projects that use bower components, so we don't need to ensure cross-project compatibility - we can vet new versions of bower components as part of standard Horizon change review That sounds good to me! Thanks for doing this! Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Perhaps they are there to support older browsers? Thanks, Kevin From: Matthias Runge [mru...@redhat.com] Sent: Wednesday, November 19, 2014 12:27 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon On 18/11/14 14:48, Thomas Goirand wrote: And then, does selenium continues to work for testing Horizon? If so, then the solution could be to send the .dll and .xpi files in non-free, and remove them from Selenium in main. Yes, it still works; that leaves the question, why they are included in the tarball at all. In Fedora, we do not distribute .dll or selenium xpi files with selenium at all. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/19/2014 04:27 PM, Matthias Runge wrote: On 18/11/14 14:48, Thomas Goirand wrote: And then, does selenium continues to work for testing Horizon? If so, then the solution could be to send the .dll and .xpi files in non-free, and remove them from Selenium in main. Yes, it still works; that leaves the question, why they are included in the tarball at all. In Fedora, we do not distribute .dll or selenium xpi files with selenium at all. Matthias Thanks for letting me know. I have opened a bug against the current selenium package in non-free, to ask to have it uploaded in Debian main, without the .xpi file. Let's see how it goes. Cheers, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 19/11/14 17:52, Fox, Kevin M wrote: Perhaps they are there to support older browsers? Probable. Windows dlls are quite uncommon in a Linux distribution. It's a bit unlikely to have an older browser installed in a centrally managed distribution like Fedora. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 18/11/14 00:59, Richard Jones wrote: On 17 November 2014 21:54, Radomir Dopieralski openst...@sheep.art.pl mailto:openst...@sheep.art.pl wrote: - Bower in the development environment, - Bower configuration file in two copies, one for global-requirements, and one for the Horizon's local requirements. Plus a gate job that makes sure no new library or version gets included in the Horizon's before getting into the global-requirements, Could you perhaps elaborate on this? How do you see the workflow working here? Basically we would have an additional file in the global-requirements directory, for listing the JavaScript dependencies. Then we would have a check on the Horizon's gate that would check Horizon's Bower configuration against that global-requirements file. This way we keep the same process for JavaScript libraries as we have for Python libraries: first you submit a patch to the global-requirements and have the dependency accepted by the infra team and packagers (checked against licenses, version conflicts, etc.), and then you add it to Horizon's dependencies. Of course you can submit both patches at once, just the Horizon one will fail the gate until the global-requirements one gets merged. Given that Horizon already integrates with xstatic, it would be messy (and potentially confusing) to include something so it *also* integrated with bower. I was envisaging us creating a tool which generates xstatic packages from bower packages. I'm not the first to think along these lines http://lists.openstack.org/pipermail/openstack-dev/2014-March/031042.html If we use Bower, we don't need to use Xstatic. It would be pure overhead. Bower already takes care of tracking releases and versions, and of bundling the files. All we need is a simple line in the settings.py telling Django where it puts all the files -- we don't really need Xstatic just for that. The packagers can then take those Bower packages and turn them into system packages, and just add/change the paths in settings.py to where they put the files. All in one place. I will be looking into creating such a tool today. The problem is not the work that has to be done for the initial creation of the package. That is one-time effort and as you show, it can be easily automated. The problem is the effort and resources needed to maintain that package. Someone (the author of the package?) has to check for security vulnerabilities, critical bugs, packaging issues, changing licenses, etc. and patch/update the packages accordingly. Also, the more layers of code you have, them more likely you are to have bugs in the. Again, I see no reason to duplicate the effort if the Bower packagers are doing that work for us already (they are, are they?). -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 18 November 2014 19:22, Radomir Dopieralski openst...@sheep.art.pl wrote: On 18/11/14 00:59, Richard Jones wrote: On 17 November 2014 21:54, Radomir Dopieralski openst...@sheep.art.pl mailto:openst...@sheep.art.pl wrote: - Bower in the development environment, - Bower configuration file in two copies, one for global-requirements, and one for the Horizon's local requirements. Plus a gate job that makes sure no new library or version gets included in the Horizon's before getting into the global-requirements, Could you perhaps elaborate on this? How do you see the workflow working here? Basically we would have an additional file in the global-requirements directory, for listing the JavaScript dependencies. Then we would have a check on the Horizon's gate that would check Horizon's Bower configuration against that global-requirements file. This way we keep the same process for JavaScript libraries as we have for Python libraries: first you submit a patch to the global-requirements and have the dependency accepted by the infra team and packagers (checked against licenses, version conflicts, etc.), and then you add it to Horizon's dependencies. Of course you can submit both patches at once, just the Horizon one will fail the gate until the global-requirements one gets merged. Given that Horizon already integrates with xstatic, it would be messy (and potentially confusing) to include something so it *also* integrated with bower. I was envisaging us creating a tool which generates xstatic packages from bower packages. I'm not the first to think along these lines http://lists.openstack.org/pipermail/openstack-dev/2014-March/031042.html If we use Bower, we don't need to use Xstatic. It would be pure overhead. Bower already takes care of tracking releases and versions, and of bundling the files. All we need is a simple line in the settings.py telling Django where it puts all the files -- we don't really need Xstatic just for that. The packagers can then take those Bower packages and turn them into system packages, and just add/change the paths in settings.py to where they put the files. All in one place. I guess I got the message that turning bower packages into system packages was something that the Linux packagers were not keen on. Did I get the message wrong there? If so, *and* we can get the bower stuff through #infra and global-requirements then yes, we should totally try to avoid adding the xstatic layer :) Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Tue, Nov 18, 2014 at 09:36:00PM +1100, Richard Jones wrote: I guess I got the message that turning bower packages into system packages was something that the Linux packagers were not keen on. Did I get the message wrong there? If so, *and* we can get the bower stuff through #infra and global-requirements then yes, we should totally try to avoid adding the xstatic layer :) For me, it's more work to turn packages into bower, or to translate bower packages to system packages. But that is purely because we don't have bower packaged currently in Fedora. I would vote for the cleaner way (whatever that is). XStatic is obviously not the cleanest way, but a good compromise in most situations. Matthias -- Matthias Runge mru...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/17/2014 03:22 PM, Matthias Runge wrote: On 17/11/14 02:07, Richard Jones wrote: Except that selenium is non-free: it's in the non-free repository of Debian, because it contains a pre-built .xpi plugin for firefox, which itself contains pre-built .so and .dll files. Hasn't this issue already been addressed? Horizon currently uses Selenium-based integration tests. For Fedora, we found, that simply removing bundled .dll and .xpi files still leaves selenium intact. And then, does selenium continues to work for testing Horizon? If so, then the solution could be to send the .dll and .xpi files in non-free, and remove them from Selenium in main. Cheers, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 18/11/14 12:54, Matthias Runge wrote:
On Tue, Nov 18, 2014 at 09:36:00PM +1100, Richard Jones wrote:
I guess I got the message that turning bower packages into system packages
was something that the Linux packagers were not keen on. Did I get the
message wrong there? If so, *and* we can get the bower stuff through #infra
and global-requirements then yes, we should totally try to avoid adding the
xstatic layer :)
For me, it's more work to turn packages into bower, or to translate
bower packages to system packages.
But that is purely because we don't have bower packaged currently in Fedora.
I would vote for the cleaner way (whatever that is).
XStatic is obviously not the cleanest way, but a good compromise in most
situations.
The way I thought about it, we would simply replace the Bower packages
with the corresponding system packages, by just changing the path in the
settings.py file.
Maybe an example would help illustrate it.
Currently we have something like this:
STATICFILES_DIRS = [
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular).base_dir),
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular_cookies).base_dir),
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular_mock).base_dir),
('horizon/lib/bootstrap_datepicker',
xstatic.main.XStatic(xstatic.pkg.bootstrap_datepicker).base_dir),
('bootstrap',
xstatic.main.XStatic(xstatic.pkg.bootstrap_scss).base_dir),
...
]
We would replace that with:
STATICFILES_DIRS = [
('horizon/lib/angular',
os.path.join(BASE_DIR, 'bower_modules/angular'),
...
]
or wherever bower puts the files. Now, the packagers would replace those
lines with:
STATICFILES_DIRS = [
('horizon/lib/angular',
'/usr/lib/javascript/angular'),
...
]
or whatever the packages in their distribution use. It's less work than
having to make a whole xstatic package just to put that single path in
there. The horizon system package would already depend on all the
javascript library packages, so we are sure the files are there.
--
Radomir Dopieralski
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 19 November 2014 03:14, Radomir Dopieralski openst...@sheep.art.pl
wrote:
On 18/11/14 12:54, Matthias Runge wrote:
On Tue, Nov 18, 2014 at 09:36:00PM +1100, Richard Jones wrote:
I guess I got the message that turning bower packages into system
packages
was something that the Linux packagers were not keen on. Did I get the
message wrong there? If so, *and* we can get the bower stuff through
#infra
and global-requirements then yes, we should totally try to avoid adding
the
xstatic layer :)
For me, it's more work to turn packages into bower, or to translate
bower packages to system packages.
But that is purely because we don't have bower packaged currently in
Fedora.
I would vote for the cleaner way (whatever that is).
XStatic is obviously not the cleanest way, but a good compromise in most
situations.
The way I thought about it, we would simply replace the Bower packages
with the corresponding system packages, by just changing the path in the
settings.py file.
Maybe an example would help illustrate it.
Currently we have something like this:
STATICFILES_DIRS = [
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular).base_dir),
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular_cookies).base_dir),
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular_mock).base_dir),
('horizon/lib/bootstrap_datepicker',
xstatic.main.XStatic(xstatic.pkg.bootstrap_datepicker).base_dir),
('bootstrap',
xstatic.main.XStatic(xstatic.pkg.bootstrap_scss).base_dir),
...
]
We would replace that with:
STATICFILES_DIRS = [
('horizon/lib/angular',
os.path.join(BASE_DIR, 'bower_modules/angular'),
...
]
or wherever bower puts the files. Now, the packagers would replace those
lines with:
STATICFILES_DIRS = [
('horizon/lib/angular',
'/usr/lib/javascript/angular'),
...
]
or whatever the packages in their distribution use. It's less work than
having to make a whole xstatic package just to put that single path in
there. The horizon system package would already depend on all the
javascript library packages, so we are sure the files are there.
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
+1 to all that, except I'd recommend using django-bower to handle the
static collection stuff. It's not documented but django-bower has a setting
BOWER_COMPONENTS_ROOT which would make the above transition much simpler.
You leave it alone for local dev, and packagers setup BOWER_COMPONENTS_ROOT
to '/usr/lib/javascript/' or wherever.
bower also has a concept of entry points - there a main value in the
bower.json which identifies the Javascript, CSS, font and other files to
include in the index.html to have the library loaded into your application.
Saves a bunch of manual editing to get things right when you include the
library. Sadly, the current django-bower plugin doesn't use any of that,
though it does handle the collectstatic stuff.
Richard
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
I've just had a long discussion with #infra folk about the
global-requirements thing, which deviated (quite naturally) into a
discussion about packaging (and their thoughts were in line with where
Radomir and I are heading).
In their view, bower components don't need to be in global-requirements:
- there are no other projects that use bower components, so we don't need
to ensure cross-project compatibility
- we can vet new versions of bower components as part of standard Horizon
change review
Richard
On 19 November 2014 06:43, Richard Jones r1chardj0...@gmail.com wrote:
On 19 November 2014 03:14, Radomir Dopieralski openst...@sheep.art.pl
wrote:
On 18/11/14 12:54, Matthias Runge wrote:
On Tue, Nov 18, 2014 at 09:36:00PM +1100, Richard Jones wrote:
I guess I got the message that turning bower packages into system
packages
was something that the Linux packagers were not keen on. Did I get the
message wrong there? If so, *and* we can get the bower stuff through
#infra
and global-requirements then yes, we should totally try to avoid
adding the
xstatic layer :)
For me, it's more work to turn packages into bower, or to translate
bower packages to system packages.
But that is purely because we don't have bower packaged currently in
Fedora.
I would vote for the cleaner way (whatever that is).
XStatic is obviously not the cleanest way, but a good compromise in most
situations.
The way I thought about it, we would simply replace the Bower packages
with the corresponding system packages, by just changing the path in the
settings.py file.
Maybe an example would help illustrate it.
Currently we have something like this:
STATICFILES_DIRS = [
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular).base_dir),
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular_cookies).base_dir),
('horizon/lib/angular',
xstatic.main.XStatic(xstatic.pkg.angular_mock).base_dir),
('horizon/lib/bootstrap_datepicker',
xstatic.main.XStatic(xstatic.pkg.bootstrap_datepicker).base_dir),
('bootstrap',
xstatic.main.XStatic(xstatic.pkg.bootstrap_scss).base_dir),
...
]
We would replace that with:
STATICFILES_DIRS = [
('horizon/lib/angular',
os.path.join(BASE_DIR, 'bower_modules/angular'),
...
]
or wherever bower puts the files. Now, the packagers would replace those
lines with:
STATICFILES_DIRS = [
('horizon/lib/angular',
'/usr/lib/javascript/angular'),
...
]
or whatever the packages in their distribution use. It's less work than
having to make a whole xstatic package just to put that single path in
there. The horizon system package would already depend on all the
javascript library packages, so we are sure the files are there.
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
+1 to all that, except I'd recommend using django-bower to handle the
static collection stuff. It's not documented but django-bower has a setting
BOWER_COMPONENTS_ROOT which would make the above transition much simpler.
You leave it alone for local dev, and packagers setup BOWER_COMPONENTS_ROOT
to '/usr/lib/javascript/' or wherever.
bower also has a concept of entry points - there a main value in the
bower.json which identifies the Javascript, CSS, font and other files to
include in the index.html to have the library loaded into your application.
Saves a bunch of manual editing to get things right when you include the
library. Sadly, the current django-bower plugin doesn't use any of that,
though it does handle the collectstatic stuff.
Richard
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Thomas Goirand z...@debian.org writes: Hi Thomas, On 11/15/2014 05:34 PM, Martin Geisler wrote: I'm sorry if I came across as being hostile towards packagers and distros. I've been running Debian for 15 years and that is because of the work the Debian developers put into making the system work well together at a whole. When it comes to installing software, I only use apt to touch paths outside my home directory. That is to ensure that the integrity of the system isn't compromised. That means that software not yet packaged for Debian has a low change of being installed by me. However, the chances of me installing it improve significantly if I can install it with pip or npm. Simply because this allows me to do a local installation in a home directory -- I know then that I can easily remove the sofware later. Sorry to say it this way, and it's not about you in particular, You're quite right, it's not about me! I'm not about to deploy OpenStack anytime soon so you don't have to sell the packaging solution to me :) My main goal in this discussion was to bring some web development knowledge to the table. It's clear to me that you have a very strong background in Debian packaging -- and (I'm guessing here) not a very strong background in web development. What we care is to find a system that will satisfy both worlds: distributions upstream fast moving development. It is looking like NPM has the best feature and that it would be a winner against Bower and Grunt. As Richard said, npm and bower are not competitors. You use npm to install bower, and you use bower to download Angular, jQuery, Bootstrap and other static files. These are the static files that you will want to include when you finally deploy the web app to your server. Before using Bower, people would simply download Angular from the projects homepage and check it into version control. Bower is not doing much, but using it avoids this bad practice. There is often a kind of compilation step between bower downloading a dependency and the deployment on the webserver: minification and compression of the JavaScript and CSS. Concatenating and minifying the files serve to reduce the number of HTTP requests -- which can make an app much faster. Finally, you use Grunt/Gulp to execute other tools during development. These tools could be a local web server, it could be running the unit tests. Grunt is only a convenience tool here -- think of it as a kind of Makefile that tells you how to lunch various tasks. Again, I'm just trying to bring information to light and let you know the tools of the trade -- how you and OpenStack as a whole decide to use and package them is not my concern. -- Martin Geisler http://google.com/+MartinGeisler pgpU3qC4oRLtU.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 17/11/14 09:53, Martin Geisler wrote: [...] As Richard said, npm and bower are not competitors. You use npm to install bower, and you use bower to download Angular, jQuery, Bootstrap and other static files. These are the static files that you will want to include when you finally deploy the web app to your server. Before using Bower, people would simply download Angular from the projects homepage and check it into version control. Bower is not doing much, but using it avoids this bad practice. There is often a kind of compilation step between bower downloading a dependency and the deployment on the webserver: minification and compression of the JavaScript and CSS. Concatenating and minifying the files serve to reduce the number of HTTP requests -- which can make an app much faster. Finally, you use Grunt/Gulp to execute other tools during development. These tools could be a local web server, it could be running the unit tests. Grunt is only a convenience tool here -- think of it as a kind of Makefile that tells you how to lunch various tasks. Thank you for your explanations. The way I see it, we would need: - Bower in the development environment, - Grunt both in the development environment and packaged (to run tests, etc.), - Bower configuration file in two copies, one for global-requirements, and one for the Horizon's local requirements. Plus a gate job that makes sure no new library or version gets included in the Horizon's before getting into the global-requirements, - A tool, probably a script, that would help packaging the Bower packages into DEB/RPM packages. I suspect the Debian/Fedora packagers already have a semi-automatic solution for that. - A script that would generate a file with all the paths to those packaged libraries, that would get included in Horizon's settings.py What do you think? -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 17 November 2014 21:54, Radomir Dopieralski openst...@sheep.art.pl wrote: On 17/11/14 09:53, Martin Geisler wrote: [...] As Richard said, npm and bower are not competitors. You use npm to install bower, and you use bower to download Angular, jQuery, Bootstrap and other static files. These are the static files that you will want to include when you finally deploy the web app to your server. Before using Bower, people would simply download Angular from the projects homepage and check it into version control. Bower is not doing much, but using it avoids this bad practice. There is often a kind of compilation step between bower downloading a dependency and the deployment on the webserver: minification and compression of the JavaScript and CSS. Concatenating and minifying the files serve to reduce the number of HTTP requests -- which can make an app much faster. Finally, you use Grunt/Gulp to execute other tools during development. These tools could be a local web server, it could be running the unit tests. Grunt is only a convenience tool here -- think of it as a kind of Makefile that tells you how to lunch various tasks. Thank you for your explanations. The way I see it, we would need: - Grunt both in the development environment and packaged (to run tests, etc.), I'm increasingly coming to think that we can avoid grunt. - serve is handled by django runserver - test running is handled by tox - livereload could be handled by https://pypi.python.org/pypi/livereload (it'd be really nice if someone could get support for this rolled in to django-livereload...) - watch is not handled by anything and it would be a shame to miss out on automatic linting/testing, but I think we can live without it So, the tools we need packaged for Linux are: - karma - jasmine (already in Fedora, I believe) - selenium (I believe this is already done in Fedora and Debian) - phantomjs (definitely already done!) - Bower in the development environment, - Bower configuration file in two copies, one for global-requirements, and one for the Horizon's local requirements. Plus a gate job that makes sure no new library or version gets included in the Horizon's before getting into the global-requirements, Could you perhaps elaborate on this? How do you see the workflow working here? Given that Horizon already integrates with xstatic, it would be messy (and potentially confusing) to include something so it *also* integrated with bower. I was envisaging us creating a tool which generates xstatic packages from bower packages. I'm not the first to think along these lines http://lists.openstack.org/pipermail/openstack-dev/2014-March/031042.html I will be looking into creating such a tool today. - A tool, probably a script, that would help packaging the Bower packages into DEB/RPM packages. I suspect the Debian/Fedora packagers already have a semi-automatic solution for that. Yep, that is indeed their problem that they'd have already solved for the existing xstatic packages. - A script that would generate a file with all the paths to those packaged libraries, that would get included in Horizon's settings.py If we stick with xstatic, we don't need this :) Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 18 November 2014 10:59, Richard Jones r1chardj0...@gmail.com wrote: On 17 November 2014 21:54, Radomir Dopieralski openst...@sheep.art.pl wrote: - Bower in the development environment, - Bower configuration file in two copies, one for global-requirements, and one for the Horizon's local requirements. Plus a gate job that makes sure no new library or version gets included in the Horizon's before getting into the global-requirements, Could you perhaps elaborate on this? How do you see the workflow working here? Given that Horizon already integrates with xstatic, it would be messy (and potentially confusing) to include something so it *also* integrated with bower. I was envisaging us creating a tool which generates xstatic packages from bower packages. I'm not the first to think along these lines http://lists.openstack.org/pipermail/openstack-dev/2014-March/031042.html I will be looking into creating such a tool today. I wrote the tool today, and you can find it here: https://github.com/r1chardj0n3s/flaming-shame (github auto-named the repository for me - it's like it KNOWS) I've proposed to Thomas Waldmann that this be included in the xstatic package. It doesn't handle dependencies at all - I'm not sure it should or could sensibly. Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/15/2014 06:27 AM, Gabriel Hurley wrote: So, I propose that a group gets together and defines criteria: we need to accept that the Horizon team (and those knowledgeable about web-app development) know best what tools they need, and they need to produce such a list as a starting point. We then need packagers and maintainers to examine that list and evaluate it for problems (non-free software, irresolvable dependencies, etc.). They're looking strictly for things which are un-packageable, not commenting on the necessity of said software. And we need people (thanks, Monty) willing to build new tools to find a way to turn that list into something the package maintainers can consume without burdening either side. Make sense? - Gabriel I'd be happy to be in this group. Let me sum-up again my opinion. Selenium As I wrote previously, the biggest issue currently for me, is selenium. It is very frustrating that I can't run these unit tests when building package, and potentially loose the opportunity to discover problems. I really would like this to be solved. There's 2 ways of solving it: 1/ Someone works so that the .xpi can be built together with the rest of selenium, and therefore, selenium can leave the non-free repository of Debian and go in main (and also be uploaded in other distros). 2/ We move away from Selenium and decide to use something else like PhantomJS. Tooling for JS dependencies === As for the tooling, we're currently talking about 7 new dependencies, which isn't much. I believe it's preferable to continue to use XStatic, because it has been very convenient in many aspects (I wont list here why again...), and that doing 7 new xstatic packages will not be so much work. But I wouldn't mind if there was some kind of environment used by developers to experience new things faster, if that doesn't affect packaging. Can someone sum-up the other opinions for the other options? Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/15/2014 05:34 PM, Martin Geisler wrote: I'm sorry if I came across as being hostile towards packagers and distros. I've been running Debian for 15 years and that is because of the work the Debian developers put into making the system work well together at a whole. When it comes to installing software, I only use apt to touch paths outside my home directory. That is to ensure that the integrity of the system isn't compromised. That means that software not yet packaged for Debian has a low change of being installed by me. However, the chances of me installing it improve significantly if I can install it with pip or npm. Simply because this allows me to do a local installation in a home directory -- I know then that I can easily remove the sofware later. Sorry to say it this way, and it's not about you in particular, but this debate about apt vs others is largely not interesting for the maintenance of Horizon itself, neither upstream or in distributions. Let me try to sum-up what has been written, so we can move forward. What we care is to find a system that will satisfy both worlds: distributions upstream fast moving development. It is looking like NPM has the best feature and that it would be a winner against Bower and Grunt. At the same time, it's obvious that stuff like NPM cannot be used for deploying Horizon inside a distribution (please, this is *not* open for a debate...). Though it seems convenient for development, just like pip is, because it is very easy to just add a new dependency, and try it out. Using NPM, Horizon contributors wouldn't have to do the work of building and maintaining XStatic packages. However, it has the issue that, unlike with XStatic, these dependencies will *not* land into our global-requirements.txt, and isn't either integrated (yet) with something like devstack. We'd have to find a way to clearly define these dependencies somewhere (like in a global-requirements-js.txt?), and have a way to agree on them and their versions. Did I forget anything? Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 17 November 2014 06:42, Thomas Goirand z...@debian.org wrote: On 11/15/2014 06:27 AM, Gabriel Hurley wrote: So, I propose that a group gets together and defines criteria: we need to accept that the Horizon team (and those knowledgeable about web-app development) know best what tools they need, and they need to produce such a list as a starting point. We then need packagers and maintainers to examine that list and evaluate it for problems (non-free software, irresolvable dependencies, etc.). They're looking strictly for things which are un-packageable, not commenting on the necessity of said software. And we need people (thanks, Monty) willing to build new tools to find a way to turn that list into something the package maintainers can consume without burdening either side. Make sense? - Gabriel I'd be happy to be in this group. As would I, hence I started the conversation in the first place :) Selenium As I wrote previously, the biggest issue currently for me, is selenium. It is very frustrating that I can't run these unit tests when building package, and potentially loose the opportunity to discover problems. I really would like this to be solved. There's 2 ways of solving it: 1/ Someone works so that the .xpi can be built together with the rest of selenium, and therefore, selenium can leave the non-free repository of Debian and go in main (and also be uploaded in other distros). 2/ We move away from Selenium and decide to use something else like PhantomJS. I think you're confusing a couple of things here. Selenium is a system for Javascript running tests in a browser environment. To do that, it needs a browser. PhantomJS provides a headless browser to do that. The tests end up being faster, less intrusive on a desktop and much simpler to run on servers (no virtual X11 shenanigans). I advocate for using PhantomJS, but also for using a reduced Selenium suite where possible - with an emphasis on unit testing of the angular code directly. Selenium is just so flaky, especially with an interface that's even slightly dynamic. What we care is to find a system that will satisfy both worlds: distributions upstream fast moving development. It is looking like NPM has the best feature and that it would be a winner against Bower and Grunt. Again, just to be clear: npm and bower are *both* packaging systems and have completely separate domains of use. npm is used to package command-line tools and libraries written in the node.js programming language whereas bower is used to package browser application components. It's not npm-or-bower. bower is most likely going to be replaced by xstatic in our environment, assuming we have some simple mechanism for creating xstatic packages from bower components. The distributions are going to have to package up the npm-packaged tools that we need, though that set of packages is looking smaller and smaller, and will probably just include the testing tools (karma, protractor, jasmin, phantomjs). Some of those are already packaged \o/ Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/15/2014 05:03 AM, Matthias Runge wrote: On 14/11/14 16:21, Adam Young wrote: Example: I don't need Grunt to run a web server. I need Apache for that. Grunt does not need to be in the distro, mod_wsgi does. I will need every tool required to run e.g. unit tests or selenium tests to be packaged. Why? Because our builders don't have access to the internet and I want to be sure, the packaged version of horizon still passes tests. Matthias Except that selenium is non-free: it's in the non-free repository of Debian, because it contains a pre-built .xpi plugin for firefox, which itself contains pre-built .so and .dll files. So, we're on the same page, but selenium is a bad tool... :) Chatting with Maxime Vidori on IRC, there's a bunch of alternative solutions that we could use instead of Selenium. PhantomJS for example is already in Debian main. I would very much welcome switching to that instead of Selenium. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 17 November 2014 11:11, Thomas Goirand z...@debian.org wrote: On 11/15/2014 05:03 AM, Matthias Runge wrote: On 14/11/14 16:21, Adam Young wrote: Example: I don't need Grunt to run a web server. I need Apache for that. Grunt does not need to be in the distro, mod_wsgi does. I will need every tool required to run e.g. unit tests or selenium tests to be packaged. Why? Because our builders don't have access to the internet and I want to be sure, the packaged version of horizon still passes tests. Matthias Except that selenium is non-free: it's in the non-free repository of Debian, because it contains a pre-built .xpi plugin for firefox, which itself contains pre-built .so and .dll files. Hasn't this issue already been addressed? Horizon currently uses Selenium-based integration tests. So, we're on the same page, but selenium is a bad tool... :) Chatting with Maxime Vidori on IRC, there's a bunch of alternative solutions that we could use instead of Selenium. PhantomJS for example is already in Debian main. I would very much welcome switching to that instead of Selenium. PhantomJS isn't a testing framework - it's just a headless browser that's scriptable from Javascript. It has a webdriver for Selenium, which is a testing framework. If you ditch Selenium then you need to replace it with a testing framework on top of PhantomJS. There's also CasperJS which is an entirely separate testing framework built over PhantomJS (and SlimerJS, which looks to be similar to PhantomJS except built on Gecko - but not headless). It's very JUnit. Some people like that ;) While Selenium has issues, I'm concerned that there's very few people out there in the wilds doing interface testing without it. Indeed there's people advocating for Selenium over PhantomJS directly. Using PhantomJS directly means we won't have a suite of tests we can then target at IE, Safari, etc to ensure that the interface works there in an automated fashion. PhantomJS might be built on WebKit, but it's different enough from Chrome that you still want to run your tests on Chrome to ensure the interface actually works. SlimerJS does allow the testing to be slightly broader, but we'd still miss automated IE and Safari tests. Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 17/11/14 02:07, Richard Jones wrote: Except that selenium is non-free: it's in the non-free repository of Debian, because it contains a pre-built .xpi plugin for firefox, which itself contains pre-built .so and .dll files. Hasn't this issue already been addressed? Horizon currently uses Selenium-based integration tests. For Fedora, we found, that simply removing bundled .dll and .xpi files still leaves selenium intact. But I agree, I would like not to have that stuff bundled at all. Tests in Horizon are: unit tests and selenium tests, both executed at the gate; selenium tests are just mocked, vs. integration tests require a cloud environment set up. This is not executed at the gate right now. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 15/11/14 03:21, Richard Jones wrote: On 15 November 2014 00:58, Radomir Dopieralski openst...@sheep.art.pl wrote: [...] 4. additions and upgrades of libraries moderated by the packagers, Is there already some mechanism for handling the creation and management of xstatic packages and how they interact with linux packagers? Sorry if this is a noob question. Anybody can at any time create any Xstatic package and put it on PyPi. To put it in StackForge, it has to be approved by the infra team, but they usually don't make problems (also, you don't technically need to keep the source code on StackForge). To put it in the global requirements.txt file, it has to be approved by the people who review that repository, and that includes the packagers. To put it in the Horizon's requirements.txt, it has to be approved by the Horizon core reviewers. I imagine we can have a similar setup for JavaScript dependencies, possibly a bower configuration file, that would be handled in a similar way. [...] -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Gabriel Hurley gabriel.hur...@nebula.com writes: Hi Gabriel, As the former Horizon PTL, I have a great respect for the importance of the contributions the distro maintainers/developers make to Horizon and OpenStack as a whole. From how many bugs the distros manage to find, to their diligence in vetting the software that we as Horizon developers provide, to their overall passion for the work they do. It is interesting to me to see the level to which the distros have not had to address this problem before. It shows a significant disconnect between the Node/JavaScript community and the distros as a whole (not surprising since node.js wasn't packaged on many distros until quite recently). I'm not excited to see the OpenStack community leading the charge on resolving packaging issues that ought to be settled between the JS community and the distros. Yet, if we have to in order to move forward I would urge us to reach out to the NPM maintainers, major library maintainers, etc. to try and make decisions that will benefit everyone for years to come. It's also interesting to see how strongly people take sides in this debate... who is OpenStack for? How crucial are the distros? Obviously if you work for RedHat or Canonical the distros are the end-all; OpenStack has to be packaged. Other companies? Opinions vary. Flexibility on this issue is not consistent, as has been pointed out already. I'm sorry if I came across as being hostile towards packagers and distros. I've been running Debian for 15 years and that is because of the work the Debian developers put into making the system work well together at a whole. When it comes to installing software, I only use apt to touch paths outside my home directory. That is to ensure that the integrity of the system isn't compromised. That means that software not yet packaged for Debian has a low change of being installed by me. However, the chances of me installing it improve significantly if I can install it with pip or npm. Simply because this allows me to do a local installation in a home directory -- I know then that I can easily remove the sofware later. -- Martin Geisler http://google.com/+MartinGeisler pgpXzFJPPlcp8.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
I think that it boils down to whether it'is possible that distributions: 1. package the node-based tools (grunt, karma, protractor, ...) as installable programs, and 2. xstatic-package the bower-based packages that we use (probably a couple of dozen at least). We might even be able to get away without using grunt, though an alternative to its LiveReload facility (and one that doesn't then also depend on another node program like django-livereload does) would be required. I believe tox and django's runserver (and other manage commands) could suffice to replace the other functionality typically provided by grunt. Richard On 14 November 2014 18:51, Radomir Dopieralski openst...@sheep.art.pl wrote: On 13/11/14 23:30, Martin Geisler wrote: [...] While I agree that it's chaotic, I also think you make the problem worse than it really is. First, remember that the user who installs Horizon won't need to use the JavaScript based *developer* tools such as npm, bower, etc. That is, I think Horizon developers will use these tools to produce a release -- a tarball -- and that tarball will be something you unpack on your webserver and then you're done. I base this on what I've seen in the project I've been working. The release tarball you download here don't mention npm, bower, or any of the other tools: https://github.com/zerovm/swift-browser/releases The tools were used to produce the tarball and were used to test it, but they're not part of the released product. Somewhat similar to how GCC isn't included in the tarball if you download a pre-compiled binary. [...] Maybe a difference is that you don't (yet) install a web application like you install a system application. Instead you *deploy* it: you unpack files on a webserver, you configure permissions, you setup cache rules, you configure a database, etc. [...] I think I see where the misunderstanding is in this whole discussion. It seems it revolves around the purpose and role of the distribution. From the naive point of view, the role of a Linux distribution is to just collect all the software from respective upstream developers and put it in a single repository, so that it can be easily installed by the users. That's not the case. The role of a distribution is to provide a working ecosystem of software, that is: a) installed and configured in consistent way, b) tested to work with the specific versions that it ships with, c) audited for security, d) maintained, including security patches, e) documented, including external tutorials and the like, f) supported, either by the community or by the companies that provide support, g) free of licensing issues and legal risks as much as possible, h) managed with the common system management tools. In order to do that, they can't just take a tarball and drop it in a directory. They always produce their own builds, to make sure it's the same thing that the source code specifies. They sometimes have to rearrange configuration files, to make them fit the standards in their system. They provide sane configuration defaults. They track the security reports about all the installed components, and apply fixes, often before the security issue is even announced. Basically, a distribution adds a whole bunch of additional guarantees for the software they ship. Those are often long-term guarantees, as they will be supporting our software long after we have forgotten about it already. You say that web development doesn't work like that. That may be true, but that's mostly because if you develop a new web application in-house, and deploy it on your server, you don't really have such large legal risk, configuration complexity or support problem -- you just have to care about that single application, because the packagers from the distribution that you are using are taking care about all the rest of software on your server. -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 13/11/14 21:11, Matthew Farina wrote: I would like to take a moment to point out that developing system software is different from developing web applications. The way systems software is developed and often deployed is different from web applications. Horizon as it sits today appears to be web application development by systems software developers. This raises the barrier to entry for web application developers. The approach being proposed moves horizon into the realm of web application technologies that web application people use today. The debate as I'm reading it is about taking web application development processes and turning them into systems development processes which are often foreign to web application developers. How is this going to work out? How will web app people be willing to get involved? Why should this be treated the same? Most of OpenStack is a systems problem. This piece is a little different. To make it successful should it get some wiggle room to work well in the space it's in? Note, I'm not saying it should be insecure or anything like that. There are just different approaches. Basically, you're saying, we should lower standards to attract more people? I disagree in your request, to handle horizon different from the rest of OpenStack: why? And it worked quite well in the past. This IMHO that's just wrong. When new folks show up: great! Everybody's welcome. We might need to educate people here. There are so many patterns used, and they have been proven wrong in the past. Technically, it's possible to have several copies of the same library installed. But because it's possible, that doesn't mean, you should do that. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 13/11/14 19:11, Donald Stufft wrote: As far as I’m aware npm supports TLS the same as pip does. That secures the transport between the end users and the repository so you can be assured that there is no man in the middle. Security wise npm (and pip) are about ~95% (mad up numbers, but you can get the gist) of the effectiveness as the OS package managers. Oh, e.g rpm allows packages to be cryptographically signed, and depending on your systems config, that is enforced. This is quite different from just tls'ing a connection. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 14, 2014, at 7:48 AM, Matthias Runge mru...@redhat.com wrote: On 13/11/14 19:11, Donald Stufft wrote: As far as I’m aware npm supports TLS the same as pip does. That secures the transport between the end users and the repository so you can be assured that there is no man in the middle. Security wise npm (and pip) are about ~95% (mad up numbers, but you can get the gist) of the effectiveness as the OS package managers. Oh, e.g rpm allows packages to be cryptographically signed, and depending on your systems config, that is enforced. This is quite different from just tls'ing a connection. You do realize that TLS provides cryptographic proof of authenticity and integrity just like PGP does right? (It also provides the cool benefit of privacy which PGP signing does not). Generally even with PGP signing you still have a number of online keys sitting on servers which are able to sign packages and the tooling will accept their signatures. The essential difference is basically, with TLS you depend on the web server to not be compromised, with PGP signing you depend on the build server to not be compromised. In theory you *can* use PGP signing in a way that all of the signing keys are offline, however this requires having a person manually sign all artifacts that are created (and even then, you'd want them to also generate said artifacts to ensure that they were not compromised). However in the real world, most (if not all) systems involve online keys. All this isn't to say that TLS is 100% as good as using something like PGP for signatures though. PGP does have some good benefits, the major one being that it travels better/easier/at all. For instance a PGP signature can be transfered alongside a package file and hosted on untrusted mirrors while relying on TLS means that you *must* trust the machine from which you're getting the files from. TLS is a fairly decent way of securing a package infrastructure though, it prevents all of the major attacks that PGP signing does in practice but it moves the high value target from the build machines to the web servers and makes mirroring require trusting the mirror. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/14/2014 06:30 AM, Martin Geisler wrote: That is, I think Horizon developers will use these tools to produce a release -- a tarball -- and that tarball will be something you unpack on your webserver and then you're done. I base this on what I've seen in the project I've been working. The release tarball you download here don't mention npm, bower, or any of the other tools: https://github.com/zerovm/swift-browser/releases The tools were used to produce the tarball and were used to test it, but they're not part of the released product. Somewhat similar to how GCC isn't included in the tarball if you download a pre-compiled binary. When doing packages, I don't even use the tarball, but a git clone, which itself produces an orig.tar.xz file. I do that to allow more flexibility and to be able to do upstream code change easily. So to build a Debian package, I will need to have all the tooling, just like I need GCC to build packages. I thought this needed to be cleared out. On 11/14/2014 06:30 AM, Martin Geisler wrote: Maybe a difference is that you don't (yet) install a web application like you install a system application. Instead you *deploy* it: you unpack files on a webserver, you configure permissions, you setup cache rules, you configure a database, etc. I really don't see why a web application should be any different from any other component of OpenStack. No, I wont deploy it, I will just apt-get install it... On 11/14/2014 06:30 AM, Martin Geisler wrote: A web app is something a single user installs on a system (www-data or a similar user) and then this user configures the system web server to serve this web app. The configuration part is the role of the package maintainer's script. At least in Debian, there's the facility to configure apache and https (if you respond positively to the debconf prompts about this), so Horizon is directly useable after you install the package. I don't want this feature to go away. On 11/14/2014 06:30 AM, Martin Geisler wrote: I agree that it would be cool to have web apps be as robust and general purpose as system apps. However, I think that day is a ways off. I'm not sure why you are saying this. Horizon works out of the box in Debian, and so is murano-dashboard and the sahara support. On 11/14/2014 06:30 AM, Martin Geisler wrote: The dependency solver is as good as the community needs it to be. Or put differently, if the JavaScript community is able to produce working software with npm, then they obviously produce it within the bounds of the capabilities of its dependency solver. I'm happy to believe that apt has a top-notch and highly tuned dependency solver. That doesn't really matter since it would be solving problems we don't have. Dependency solving is pure math. It's very hard to get it right. I don't agree that some language may need something weaker, and that it's possible for the maintainers to adapt. It's just that it may, in some case, be possible to go around some defects if they exist, but everyone needs a robust dependency solver. On 11/14/2014 06:30 AM, Martin Geisler wrote: In my view, you're taking on way too much work by going into those details. I don't think I need or want you do to anything more than repack the tarball that npm retrieves -- I don't think you should run tests or generate documentation. Of course, I need to run tests. That's a big part of the QA work, and I will certainly not give-up on that. You will have a hard time convincing anyone within the OpenStack community that it's OK to not run unit tests. As for the doc, well, I do believe it's a big plus. On 11/14/2014 06:30 AM, Martin Geisler wrote: As a user or sysadmin, I would be happy to add a deb line to my sources.list and get Debian packages that wrap the node modules. This means that the packages would *not* be in Debian. Therefore, horizon couldn't be uploaded to Debian (as there would be some not available dependencies). That's absolutely not what I want to do. I want Horizon, just like the rest of OpenStack, to be fully in Debian. Cheers, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 14/11/14 13:02, Richard Jones wrote: On 14 November 2014 18:51, Radomir Dopieralski openst...@sheep.art.pl mailto:openst...@sheep.art.pl wrote: On 13/11/14 23:30, Martin Geisler wrote: [...] Maybe a difference is that you don't (yet) install a web application like you install a system application. Instead you *deploy* it: you unpack files on a webserver, you configure permissions, you setup cache rules, you configure a database, etc. [...] In order to do that, they can't just take a tarball and drop it in a directory. They always produce their own builds, to make sure it's the same thing that the source code specifies. They sometimes have to rearrange configuration files, to make them fit the standards in their system. They provide sane configuration defaults. They track the security reports about all the installed components, and apply fixes, often before the security issue is even announced. [...] I think that it boils down to whether it'is possible that distributions: 1. package the node-based tools (grunt, karma, protractor, ...) as installable programs, and 2. xstatic-package the bower-based packages that we use (probably a couple of dozen at least). We might even be able to get away without using grunt, though an alternative to its LiveReload facility (and one that doesn't then also depend on another node program like django-livereload does) would be required. I believe tox and django's runserver (and other manage commands) could suffice to replace the other functionality typically provided by grunt. We don't really need Xstatic for that. The packagers can as well package the bower-based packages directly. We can use anything, really, as long as we follow a process that makes sure that Horizon can be packaged into the different distributions. That is, we need: 1. All dependencies explicit (with tests failing if a dependency is missing), 2. explicit version ranges, 3. no multiple versions of the same library, 4. additions and upgrades of libraries moderated by the packagers, 5. ability to replace the development environment with packaged libraries from the system, 6. ability to build and test our software with the tools that can be used by all the distributions. As I said, this is more of a process thing than a tool thing -- I believe any tool can be used to follow this process, more or less automatically. -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/11/2014 03:02 PM, Richard Jones wrote: json3 es5-shim angular angular-route angular-cookies angular-animate angular-sanitize angular-smart-table angular-local-storage angular-bootstrap angular-translate font-awesome boot underscore ng-websocket Just FYI, in Debian, the libjs-angularjs already carries: angular-route.js angular-cookies.js angular-animate.js angular-sanitize.js We also already have packaged: font-awesome underscore So, basically, I'd have to package: json3 es5-shim boot angular-smart-table angular-local-storage angular-translate ng-websocket That's a reasonable amount of work. Multiply this by 2 for the xstatic packages (if we keep using that), that's about 14 new packages. By the way, can't we use libjs-sockjs instead of ng-websocket? Last, I'm ok if we add all these, but please, let's do this in the beginning of the Kilo cycle. It was really hard to cope with it at the end of the freeze for Juno. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Thomas Goirand z...@debian.org writes: On 11/14/2014 06:30 AM, Martin Geisler wrote: That is, I think Horizon developers will use these tools to produce a release -- a tarball -- and that tarball will be something you unpack on your webserver and then you're done. I base this on what I've seen in the project I've been working. The release tarball you download here don't mention npm, bower, or any of the other tools: https://github.com/zerovm/swift-browser/releases The tools were used to produce the tarball and were used to test it, but they're not part of the released product. Somewhat similar to how GCC isn't included in the tarball if you download a pre-compiled binary. When doing packages, I don't even use the tarball, but a git clone, which itself produces an orig.tar.xz file. I do that to allow more flexibility and to be able to do upstream code change easily. That seems to be a choice you're making -- I think you could also use the upstream tarball as provided (let's say I also include unminified source files in the tarball). On 11/14/2014 06:30 AM, Martin Geisler wrote: Maybe a difference is that you don't (yet) install a web application like you install a system application. Instead you *deploy* it: you unpack files on a webserver, you configure permissions, you setup cache rules, you configure a database, etc. I really don't see why a web application should be any different from any other component of OpenStack. No, I wont deploy it, I will just apt-get install it... I'm a long time Debian user and web developer. I install system tools (web servers, databases, editors) and I deploy web applications. I believe that's the most common way to handle web applications today. On 11/14/2014 06:30 AM, Martin Geisler wrote: I agree that it would be cool to have web apps be as robust and general purpose as system apps. However, I think that day is a ways off. I'm not sure why you are saying this. Horizon works out of the box in Debian, and so is murano-dashboard and the sahara support. That's cool! On 11/14/2014 06:30 AM, Martin Geisler wrote: The dependency solver is as good as the community needs it to be. Or put differently, if the JavaScript community is able to produce working software with npm, then they obviously produce it within the bounds of the capabilities of its dependency solver. I'm happy to believe that apt has a top-notch and highly tuned dependency solver. That doesn't really matter since it would be solving problems we don't have. Dependency solving is pure math. It's very hard to get it right. I don't agree that some language may need something weaker, and that it's possible for the maintainers to adapt. It's just that it may, in some case, be possible to go around some defects if they exist, but everyone needs a robust dependency solver. I think you're misunderstanding the implication. If apt has a stronger dependency solver than npm, then that's fine. The argument that apt is stronger than npm is not an argument for moving node packages from npm to apt -- the Debian packages will still only use a subset of apt dependency solver, namely the subset they use with npm. On 11/14/2014 06:30 AM, Martin Geisler wrote: In my view, you're taking on way too much work by going into those details. I don't think I need or want you do to anything more than repack the tarball that npm retrieves -- I don't think you should run tests or generate documentation. Of course, I need to run tests. That's a big part of the QA work, and I will certainly not give-up on that. You will have a hard time convincing anyone within the OpenStack community that it's OK to not run unit tests. That's not what I said: the OpenStack developers will continue to tests the software. I personally don't think it's the job of the downstream packagers to do this QA work. (It's of course cool to run the tests on the system installed by your packages -- that test run would then install the needed tools using npm and bower and whatnot if that's how the upstream has setup the test framework.) On 11/14/2014 06:30 AM, Martin Geisler wrote: As a user or sysadmin, I would be happy to add a deb line to my sources.list and get Debian packages that wrap the node modules. This means that the packages would *not* be in Debian. Therefore, horizon couldn't be uploaded to Debian (as there would be some not available dependencies). That's absolutely not what I want to do. I want Horizon, just like the rest of OpenStack, to be fully in Debian. You don't have to convince me -- I'm not going to deploy OpenStack anytime soon (apart from DevStack). So I'm not really the right customer for these packages. All I wanted to say is that if there is a cool OpenStack dashboard written as a web app, then I would be happy to deploy it like I deploy other web apps. If there were a package I could use, that would be cool, but it's by no means a show-stopper for someone like me.
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 2014-11-14 15:10:59 +0100 (+0100), Martin Geisler wrote: [...] That's not what I said: the OpenStack developers will continue to tests the software. I personally don't think it's the job of the downstream packagers to do this QA work. (It's of course cool to run the tests on the system installed by your packages -- that test run would then install the needed tools using npm and bower and whatnot if that's how the upstream has setup the test framework.) [...] Just to quibble on this particular point... distro packagers are also developers. They often (more often than we'd like, and we do try to find ways to help avoid this where possible) need to carry their own patches to tweak the software to fit their deployment and operation model. Being able to rerun tests in-place with packaged versions of everything including their patches helps them confirm that what they distribute still works as intended. Further, the distro users are well within their rights to modify and respin these packages themselves, and will potentially want to be able to run these tests for the same reasons. We distribute our tests as part of our software because our tests *are* part of our software. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 2014-11-14 08:31:37 -0500 (-0500), Donald Stufft wrote: [...] with TLS you depend on the web server to not be compromised [...] Or in some cases, the CDN. ;) -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/14/2014 09:05 AM, Thomas Goirand wrote: That's a reasonable amount of work. Multiply this by 2 for the xstatic packages (if we keep using that), that's about 14 new packages. By the way, can't we use libjs-sockjs instead of ng-websocket? Last, I'm ok if we add all these, but please, let's do this in the beginning of the Kilo cycle. It was really hard to cope with it at the end of the freeze for Juno. Hear hear! And good work by all of the package maintainers. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/14/2014 08:48 PM, Matthias Runge wrote: On 13/11/14 19:11, Donald Stufft wrote: As far as I’m aware npm supports TLS the same as pip does. That secures the transport between the end users and the repository so you can be assured that there is no man in the middle. Security wise npm (and pip) are about ~95% (mad up numbers, but you can get the gist) of the effectiveness as the OS package managers. Oh, e.g rpm allows packages to be cryptographically signed, and depending on your systems config, that is enforced. This is quite different from just tls'ing a connection. Matthias Just like the Debian Release file is signed into a Release.gpg. So, the RPM system signs every package, while in Debian, it's the full repository that is signed. That's 2 different approaches that both works. pip doesn't offer this kind of security, but at the same time, is there any kind of check for things that are uploaded to PyPi? Is there at least a peer review process? You do realize that TLS provides cryptographic proof of authenticity and integrity just like PGP does right? (It also provides the cool benefit of privacy which PGP signing does not). Do you realize that with the TLS system, you have to trust every and all CA, while with PGP, you only need to trust a single fingerprint? All this isn't to say that TLS is 100% as good as using something like PGP for signatures though. I don't agree. I don't trust the CNNIC or the hong-kong post office, though their key is on every browser. I do trust the Debian PGP key generated by the Debian team. TLS is a fairly decent way of securing a package infrastructure though, it prevents all of the major attacks that PGP signing does in practice but it moves the high value target from the build machines to the web servers [...] And ... to a huge list of root CA which you have to trust. Cheers, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/14/2014 09:58 PM, Radomir Dopieralski wrote: On 14/11/14 13:02, Richard Jones wrote: We might even be able to get away without using grunt, though an alternative to its LiveReload facility (and one that doesn't then also depend on another node program like django-livereload does) would be required. I believe tox and django's runserver (and other manage commands) could suffice to replace the other functionality typically provided by grunt. We don't really need Xstatic for that. The packagers can as well package the bower-based packages directly. We can use anything, really, as long as we follow a process that makes sure that Horizon can be packaged into the different distributions. That is, we need: 1. All dependencies explicit (with tests failing if a dependency is missing), 2. explicit version ranges, 3. no multiple versions of the same library, 4. additions and upgrades of libraries moderated by the packagers, 5. ability to replace the development environment with packaged libraries from the system, 6. ability to build and test our software with the tools that can be used by all the distributions. What I liked a lot with the xstatic package thing, is that it was *very* easy for me to be able to manage path. Horizon just imports the xstatic package, and then the BASE_DIR or some symlink do the work. Whatever system we choose, I'd like it to be at least as simple as xstatic in this regard. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/14/2014 10:10 PM, Martin Geisler wrote: Of course, I need to run tests. That's a big part of the QA work, and I will certainly not give-up on that. You will have a hard time convincing anyone within the OpenStack community that it's OK to not run unit tests. That's not what I said: the OpenStack developers will continue to tests the software. I personally don't think it's the job of the downstream packagers to do this QA work. (It's of course cool to run the tests on the system installed by your packages -- that test run would then install the needed tools using npm and bower and whatnot if that's how the upstream has setup the test framework.) What happens is that the environment within the distribution, inevitably, will be different from the one ran on the gate. There's going to be different versions of many components and so on. So it is very important for me to also run these unit tests, to make sure that everything continues to work. Yes, the build-dependencies will pull the same components as pulled by npm/bower, though they may be installed in different path, and maybe using different versions. On 11/14/2014 10:21 PM, Jeremy Stanley wrote: On 2014-11-14 15:10:59 +0100 (+0100), Martin Geisler wrote: [...] Just to quibble on this particular point... distro packagers are also developers. They often (more often than we'd like, and we do try to find ways to help avoid this where possible) need to carry their own patches to tweak the software to fit their deployment and operation model. Being able to rerun tests in-place with packaged versions of everything including their patches helps them confirm that what they distribute still works as intended. Further, the distro users are well within their rights to modify and respin these packages themselves, and will potentially want to be able to run these tests for the same reasons. We distribute our tests as part of our software because our tests *are* part of our software. Exactly! Let me give a few examples... In Jessie, Nova carries patches so that there is support for Ceph. Until a few days, there was still an issue with live migration over NFS. This has just been fixed (thanks to Mehdi!), and running unit tests at build time confirmed that. Another example. Jessie will be released with Icehouse Horizon, but with Django 1.7. The gate didn't test that, but I did. Most patches landed in Juno, though never Icehouse will be tested with Django 1.7 in the gate. Lucky, my package runs these unit tests and I can confirm that it continues to work with Django 1.7 in Jessie. Hoping these are giving you an insight as why it's *really* important to run unit tests at build time for distributions, Cheers, Thomas Goirand (zigo) P.S: I also run tempest tests over a Debian package installation to make sure OpenStack is also functional. But that's another story... :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 14, 2014, at 1:57 PM, Thomas Goirand z...@debian.org wrote: On 11/14/2014 08:48 PM, Matthias Runge wrote: On 13/11/14 19:11, Donald Stufft wrote: As far as I’m aware npm supports TLS the same as pip does. That secures the transport between the end users and the repository so you can be assured that there is no man in the middle. Security wise npm (and pip) are about ~95% (mad up numbers, but you can get the gist) of the effectiveness as the OS package managers. Oh, e.g rpm allows packages to be cryptographically signed, and depending on your systems config, that is enforced. This is quite different from just tls'ing a connection. Matthias Just like the Debian Release file is signed into a Release.gpg. So, the RPM system signs every package, while in Debian, it's the full repository that is signed. That's 2 different approaches that both works. pip doesn't offer this kind of security, but at the same time, is there any kind of check for things that are uploaded to PyPi? Is there at least a peer review process? The entirety of PyPI is signed. It’s not possible to get a copy of our equivalent to Release.gpg that isn’t cryptographically proven to have been sent by a server possessing our RSA private key. No, PyPI is not a curated repository, nor are any of the language repos that I’m aware of. That really has nothing to do with securely fetching a particular package, it only has to do with whether the contents of said package are safe to use. It means that people installing a package from PyPI have to decide if they trust the author of the package prior to installing it, but if they do trust that author then it is roughly as safe to install from PyPI as it is to install from Debian. The Linux distros are curated repositories so you need to decide if you want to trust the gatekeepers of the distro instead of the authors of the software you’re using (or really you probably need to trust both since a malicious author could likely hide back doors that would go unnoticed during packaging as a .deb). You do realize that TLS provides cryptographic proof of authenticity and integrity just like PGP does right? (It also provides the cool benefit of privacy which PGP signing does not). Do you realize that with the TLS system, you have to trust every and all CA, while with PGP, you only need to trust a single fingerprint? You absolutely do not need to trust every single CA, or even any CAs at all. If I recall npm pins which CA they trust. Pip doesn’t (yet) do this because of some historical reasons but it’s on my list of things as well. It’s no harder to limit the set of CAs or even individual certificates that are accepted as valid than it is to limit the set of PGP keys you trust. All this isn't to say that TLS is 100% as good as using something like PGP for signatures though. I don't agree. I don't trust the CNNIC or the hong-kong post office, though their key is on every browser. I do trust the Debian PGP key generated by the Debian team. See above, you’re operating under a misconception that TLS mandates using the same set of CAs as the browsers use. TLS is a fairly decent way of securing a package infrastructure though, it prevents all of the major attacks that PGP signing does in practice but it moves the high value target from the build machines to the web servers [...] And ... to a huge list of root CA which you have to trust. Already discussed above. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 2014-11-15 02:57:15 +0800 (+0800), Thomas Goirand wrote: [...] Do you realize that with the TLS system, you have to trust every and all CA, while with PGP, you only need to trust a single fingerprint? [...] Technically not true *if* the package retrieval tools implement certificate pinning rather than trusting any old CA (after all, they're not Web browsers, so they could in theory do that with minimal impact). Too bad https://github.com/pypa/pip/issues/1168 hasn't gotten much traction. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 14, 2014, at 2:39 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-11-15 02:57:15 +0800 (+0800), Thomas Goirand wrote: [...] Do you realize that with the TLS system, you have to trust every and all CA, while with PGP, you only need to trust a single fingerprint? [...] Technically not true *if* the package retrieval tools implement certificate pinning rather than trusting any old CA (after all, they're not Web browsers, so they could in theory do that with minimal impact). Too bad https://github.com/pypa/pip/issues/1168 hasn't gotten much traction. Yea, primary reason that hasn’t been done is up until recently we (PyPI) were relying on the TLS certificate provided by Fastly and they were unwilling to make a promise to also be using a particular CA for the next N years. We now have dedicated IP addresses with them so we can provide them with whatever certificate we want, now it’s just a matter of selecting CAs and the political process. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 14/11/14 16:21, Adam Young wrote: Example: I don't need Grunt to run a web server. I need Apache for that. Grunt does not need to be in the distro, mod_wsgi does. I will need every tool required to run e.g. unit tests or selenium tests to be packaged. Why? Because our builders don't have access to the internet and I want to be sure, the packaged version of horizon still passes tests. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
As the former Horizon PTL, I have a great respect for the importance of the contributions the distro maintainers/developers make to Horizon and OpenStack as a whole. From how many bugs the distros manage to find, to their diligence in vetting the software that we as Horizon developers provide, to their overall passion for the work they do. It is interesting to me to see the level to which the distros have not had to address this problem before. It shows a significant disconnect between the Node/JavaScript community and the distros as a whole (not surprising since node.js wasn't packaged on many distros until quite recently). I'm not excited to see the OpenStack community leading the charge on resolving packaging issues that ought to be settled between the JS community and the distros. Yet, if we have to in order to move forward I would urge us to reach out to the NPM maintainers, major library maintainers, etc. to try and make decisions that will benefit everyone for years to come. It's also interesting to see how strongly people take sides in this debate... who is OpenStack for? How crucial are the distros? Obviously if you work for RedHat or Canonical the distros are the end-all; OpenStack has to be packaged. Other companies? Opinions vary. Flexibility on this issue is not consistent, as has been pointed out already. Monty and Adam Young have, I think, voiced a good moderate viewpoint, which is that the distros are a core part of OpenStack's success and we have to ensure that they can package our software, but that the distros also cannot dictate the tools we use in order to produce the best possible product. Distros are ultimately middle-men, they provide value to their users and they make sure more people use the software we produce. We can all agree that we want to maximize the number of people we can reach. So, I propose that a group gets together and defines criteria: we need to accept that the Horizon team (and those knowledgeable about web-app development) know best what tools they need, and they need to produce such a list as a starting point. We then need packagers and maintainers to examine that list and evaluate it for problems (non-free software, irresolvable dependencies, etc.). They're looking strictly for things which are un-packageable, not commenting on the necessity of said software. And we need people (thanks, Monty) willing to build new tools to find a way to turn that list into something the package maintainers can consume without burdening either side. Make sense? - Gabriel -Original Message- From: Thomas Goirand [mailto:z...@debian.org] Sent: Friday, November 14, 2014 11:11 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon On 11/14/2014 10:10 PM, Martin Geisler wrote: Of course, I need to run tests. That's a big part of the QA work, and I will certainly not give-up on that. You will have a hard time convincing anyone within the OpenStack community that it's OK to not run unit tests. That's not what I said: the OpenStack developers will continue to tests the software. I personally don't think it's the job of the downstream packagers to do this QA work. (It's of course cool to run the tests on the system installed by your packages -- that test run would then install the needed tools using npm and bower and whatnot if that's how the upstream has setup the test framework.) What happens is that the environment within the distribution, inevitably, will be different from the one ran on the gate. There's going to be different versions of many components and so on. So it is very important for me to also run these unit tests, to make sure that everything continues to work. Yes, the build-dependencies will pull the same components as pulled by npm/bower, though they may be installed in different path, and maybe using different versions. On 11/14/2014 10:21 PM, Jeremy Stanley wrote: On 2014-11-14 15:10:59 +0100 (+0100), Martin Geisler wrote: [...] Just to quibble on this particular point... distro packagers are also developers. They often (more often than we'd like, and we do try to find ways to help avoid this where possible) need to carry their own patches to tweak the software to fit their deployment and operation model. Being able to rerun tests in-place with packaged versions of everything including their patches helps them confirm that what they distribute still works as intended. Further, the distro users are well within their rights to modify and respin these packages themselves, and will potentially want to be able to run these tests for the same reasons. We distribute our tests as part of our software because our tests *are* part of our software. Exactly! Let me give a few examples... In Jessie, Nova carries patches so that there is support for Ceph. Until a few days
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 15 November 2014 00:58, Radomir Dopieralski openst...@sheep.art.pl wrote: On 14/11/14 13:02, Richard Jones wrote: I think that it boils down to whether it'is possible that distributions: 1. package the node-based tools (grunt, karma, protractor, ...) as installable programs, and 2. xstatic-package the bower-based packages that we use (probably a couple of dozen at least). We might even be able to get away without using grunt, though an alternative to its LiveReload facility (and one that doesn't then also depend on another node program like django-livereload does) would be required. I believe tox and django's runserver (and other manage commands) could suffice to replace the other functionality typically provided by grunt. We don't really need Xstatic for that. The packagers can as well package the bower-based packages directly. We can use anything, really, as long as we follow a process that makes sure that Horizon can be packaged into the different distributions. That is, we need: xstatic provides additional meta-data that makes it much easier to integrate the static bundle into an application - specifically it automatically provides the application with file locations and endpoints needed to be inserted into the application HTML. That stuff would have to be done manually without xstatic, and would be a configuration pain. 1. All dependencies explicit (with tests failing if a dependency is missing), xstatic provides us with a dependency mechanism through standard Python setuptools facilities. 2. explicit version ranges, xstatic is done through requirements.txt, so yep :) 3. no multiple versions of the same library, This is not a thing in bower/client-side land. It's really only an issue for npm-based packages, and as I've mentioned, the things we're using there should be packaged as tools by the linux vendor, rather than accessed as packages by our system. Except on non-Linux, of course, where we'll just use npm ;) So I guess the big open question is about how distros are going to deal with npm tool packaging. I can't really answer that question for them (except to say: don't try to replace npm's dependency solution with your own; it simply won't work because you'll probably never find a set of versions that satisfy even just the few tools we're looking at for this project). 4. additions and upgrades of libraries moderated by the packagers, Is there already some mechanism for handling the creation and management of xstatic packages and how they interact with linux packagers? Sorry if this is a noob question. 5. ability to replace the development environment with packaged libraries from the system, I would very much like to still use bower for rapid development and testing, with xstatic coming in once the experimentation is over. But if there was a tool to automatically generate an xstatic package from a bower one, that would be eaiser... 6. ability to build and test our software with the tools that can be used by all the distributions. Yep, I think that just implies that the xstatic stuff is done, and that the distros package the few npm tools we use. On 15 November 2014 01:05, Thomas Goirand z...@debian.org wrote: On 11/11/2014 03:02 PM, Richard Jones wrote: json3 es5-shim angular angular-route angular-cookies angular-animate angular-sanitize angular-smart-table angular-local-storage angular-bootstrap angular-translate font-awesome boot underscore ng-websocket Just FYI, in Debian, the libjs-angularjs already carries: angular-route.js angular-cookies.js angular-animate.js angular-sanitize.js We also already have packaged: font-awesome underscore So, basically, I'd have to package: json3 es5-shim boot angular-smart-table angular-local-storage angular-translate ng-websocket That's a reasonable amount of work. Multiply this by 2 for the xstatic packages (if we keep using that), that's about 14 new packages. The issue is integration with the Django application. How do we know what the file path is to the entry point for that that code, and also how it differs from other packagers? xstatic takes care of that for us (in much the same way that bower does), so is valuable. By the way, can't we use libjs-sockjs instead of ng-websocket? They offer different functionality, as far as I can tell. sockjs is a compatibility layer thing, I think. I'm not sure. The description is a little vague. On the other hand, ng-websocket is all about providing an interface for angular applications over the top of websockets (and a handy mock). Last, I'm ok if we add all these, but please, let's do this in the beginning of the Kilo cycle. It was really hard to cope with it at the end of the freeze for Juno. I'd imagine this should be reasonable, barring any additional dependencies we decide to include along the way.
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Matthias Runge mru...@redhat.com writes:
On Wed, Nov 12, 2014 at 08:35:18AM -0500, Monty Taylor wrote:
Just for the record, I believe that we should chose the tools that
make sense for making our software, as long as it's not physically
impossible for them to be packaged. This means we should absolutely
not use things that require multiple versions of node to be needed.
The nodejs that's in trusty is new enough to work with all of the
modern javascript tool chain things needed for this, so other than
the various javascript tools and libraries not being packaged in the
distros yet, it should be fine.
Agreed. We're in the position to describe or define, what we'd like to
use or to see in the future. That may require us to create required
tools.
You're not concerned about node.js? Most probably, since you're not
distributing it. Looking at the changelog, I'm a bit worried[1]:
[...]
For better or for worse, the JavaScript community is using Node,
Grunt/Gulp, bower, ... as the default infrastructure tools.
Not using them or putting effort into creating alternatives would be
working against that community and I would say it's wasted effort. That
effort could be put to better use in the core OpenStack code.
I haven't cared much about Node itself, it's just a VM that runs my
JavaScript code. If I were to deploy it on a server I would agree that
the security and robustness becomes critical.
I find npm and bower alright -- they do their job just fine. The
semantic versioning craze is strange to me, but you can avoid it by
fully specifying the versions you depend on.
I find Grunt and Gulp to be overrated. My very simple Gruntfile[1] now
has about 170 lines of JSON to configure the simple tasks. For a copy
this to that task, the JSON format is fine, but for more complex tasks
with several steps it feels forced. I mean, I need to copy files in
several tasks, so I end up with
copy: {
some_target: {
...
},
other_target: {
...
}
}
in one part of the Gruntfile and then
task: {
some_target: {
...
}
}
many lines away. There's nothing to connect the two pieces of
configuration than a third task that runs both.
The whole multi-task idea also seems strange to me. It feels like an
idea that felt nice when the system was small and now the entire system
is built around it. As an example, running 'grunt copy' by itself is
useless when the two copy targets are small parts of bigger tasks.
About Gulp... I don't get it and I don't buy it :) The implication that
using streams will make your build fast is at best an
over-simplification. While streaming data is cool, elevating this single
idea to the fundamental building block in a simple task runner seems
contrieved.
It forces you into a pattern and you end up writing code looking like:
gulp.src('./client/templates/*.jade')
.pipe(jade())
.pipe(gulp.dest('./build/templates'))
.pipe(minify())
.pipe(gulp.dest('./build/minified_templates'));
That might look cute for a minimal example like this, but I bet it'll
make things harder than they should be in the future. As in, how can I
inject more files into the stream conditionally? How do I read an
environment variable and inject some extra files? With normal JavaScript
I would know how to do this.
(Here I would apparently use gulp-if together with something called a
lazypipe and I would still need to somehow insert the extra files in the
stream.)
[1]: https://github.com/zerovm/swift-browser/blob/master/Gruntfile.js
--
Martin Geisler
http://google.com/+MartinGeisler
pgp8QxY1uE7Ym.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Jiri Tomasek jtoma...@redhat.com writes: Which tools should we use eventually: Based on the contributions by Maxime, Martin and the others, I think the list of tools should end up as follows: Tooling: npm bower gulp While I find the design of Gulp strange, I'm sure it will do the job. Someone said that the Angular teams is moving to it, so that is a +1 in its favor. Jasmine Karma/Protractor(?)/eslint I've used Protractor for my end-to-end tests and after getting to know it, it works fine. It's my impression that it used to be annying to getup selenium and actually writing this kind of tests -- with Protractor you get up and running very quickly. I don't have anything to compare it with, though, but it's the standard for Angular development and that alone should be a strong hint. -- Martin Geisler http://google.com/+MartinGeisler pgp0giFB1wybi.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 12/11/14 18:23, Jiri Tomasek wrote: I see relation between Nodejs and js libs/tools and Angular app defining it's dependencies using NPM and Bower quite similar as Ruby, Rubygems and Rails application defining it's dependencies in Gemfile.lock. Rubygems are being packaged in distros, so why shouldn't node packages? Some of them are already packaged by distros, and we have even guidelines to do that: https://fedoraproject.org/wiki/Packaging:Node.js But then you'll be using yum/dnf/whatever instead of npm to install it. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/11/14 08:02, Richard Jones wrote: [...] There were some discussions around tooling. We're using xstatic to manage 3rd party components, but there's a lot missing from that environment. I hesitate to add supporting xstatic components on to the already large pile of work we have to do, so would recommend we switch to managing those components with bower instead. For reference the list of 3rd party components I used in angboard* (which is really only a teensy fraction of the total application we'd end up with, so this components list is probably reduced): [...] Just looking at PyPI, it looks like only a few of those are in xstatic, and those are out of date. There is a very good reason why we only have a few external JavaScript libraries, and why they are in those versions. You see, we are not developing Horizon for our own enjoyment, or to install it at our own webserver and be done with it. What we write has to be then packaged for different Linux distributions by the packagers. Those packagers have very little wiggle room with respect to how they can package it all, and what they can include. In particular, libraries should get packaged separately, so that they can upgrade them and apply security patches and so on. Before we used xstatic, they have to go through the sources of Horizon file by file, and replace all of our bundled files with symlinks to what is provided in their distribution. Obviously that was laborious and introduced bugs when the versions of libraries didn't match. So now we have the xstatic system. That means, that the libraries are explicitly listed, with their minimum and maximum version numbers, and it's easy to make a dummy xstatic package that just points at some other location of the static files. This simplifies the work of the packagers. But the real advantage of using the xstatic packages is that in order to add them to Horizon, you need to add them to the global-requirements list, which is being watched and approved by the packagers themselves. That means, that when you try to introduce a new library, or a version of an old library, that is for some reason problematic for any of the distributions (due to licensing issues, due to them needing to remain at an older version, etc.), they get to veto it and you have a chance of resolving the problem early, not dropping it at the last moment on the packagers. Going back to the versions of the xstatic packages that we use, they are so old for a reason. Those are the newest versions that are available with reasonable effort in the distributions for which we make Horizon. If you want to replace this system with anything else, please keep in contact with the packagers to make sure that the resulting process makes sense and is acceptable for them. -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 13/11/14 08:23, Matthias Runge wrote: [...] Since we don't require node.js on the server (yet), but only for the development process: did anyone look at node's competitors? Like CommonJS, Rhino, or SpiderMonkey? When we were struggling with adding jslint to our CI, we did try a number of different alternatives to node.js, like Rhino, SpiderMonkey, V8, phantomjs, etc. The conclusion was that even tools that advertised themselves as working on Rhino dropped their support for it several years ago, and just didn't update the documentation. Node seems to be the only thing that works without having to modify the code of those tools. Of course things might have changed since, or we may have someone with better JavaScript hacking skills who would manage to make it work. But last year we failed. -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 13/11/14 01:32, Richard Jones wrote: [...] We're currently using xstatic and that works with Linux packaging because it was designed to cope with being a global installation. The current Horizon codebase has a django-xstatic plugin which further makes dealing with xstatic components nicer - for example it handles path management and static file compilation (JS minification and concatenation, for example). That's really nice, but poses some problems: - we would need to xstatic-ify (and deb/rpm-ify) all those components Yes. They will need to be deb/rpm/arch/slack/whatever-ified anyways, because that's how the Linux distributions that are going to ship them work. - we could run into global version conflict issues if we run more than one service on a system - is this likely to be an issue in practise though? Yes, this is an issue in practice, and that's why the packagers have a say in what libraries and in what versions you are adding to the global-requirements. We have to use versions that are the least problematic. - as far as I'm aware, the xstatic JS minification is not angular-aware, and will break angular code that has not been written to be dumb-minifier-aware (the angular minifier ngMin is written in node and knows how to do things more correctly); adding dumb-minifier-awareness to angular code makes it ugly and more error-prone :/ You can use any minifier with the django-compress plugin that Horizon uses (django-xstatic has nothing to do with it). You just define the command (or a filter written in Python) to use for every mime type. But I assume that ngMin is written in the Node.js language (which is superficially similar to JavaScript) and therefore if we used it, you would have to convince your fellow system administrators to install node.js on their production servers. Violence may result. [...] -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Radomir Dopieralski openst...@sheep.art.pl writes: On 11/11/14 08:02, Richard Jones wrote: [...] There were some discussions around tooling. We're using xstatic to manage 3rd party components, but there's a lot missing from that environment. I hesitate to add supporting xstatic components on to the already large pile of work we have to do, so would recommend we switch to managing those components with bower instead. For reference the list of 3rd party components I used in angboard* (which is really only a teensy fraction of the total application we'd end up with, so this components list is probably reduced): [...] Just looking at PyPI, it looks like only a few of those are in xstatic, and those are out of date. There is a very good reason why we only have a few external JavaScript libraries, and why they are in those versions. You see, we are not developing Horizon for our own enjoyment, or to install it at our own webserver and be done with it. What we write has to be then packaged for different Linux distributions by the packagers. [...] Maybe a silly question, but why insist on this? Why would you insist on installing a JavaScript based application using your package manager? I'm a huge fan of package managers and typically refuse to install anything globally if it doesn't come as a package. However, the whole JavaScript ecosystem seems to be centered around the idea of doing local installations. That means that you no longer need the package manager to install the software -- you only need a package manager to install the base system (NodeJs and npm for JavaScript). Notice that Python has been moving rapidly in the same direction for years: you only need Python and pip to bootstrap yourself. After getting used to virtualenv, I've mostly stopped installing Python modules globally and that is how the JavaScript world expects you to work too. (Come to think of it, the same applies to some extend to Haskell and Emacs where there also exist nice package managers that'll pull in and manage dependencies for you.) So maybe the Horizon package should be an installer package like the ones that download fonts or Adobe? That package would get the right version of node and which then runs the npm and bower commands to download the rest plus (importantly and much appreciated) puts the files in a sensible location and gives them good permissions. -- Martin Geisler http://google.com/+MartinGeisler pgph36NhgvYqz.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/13/2014 12:13 PM, Richard Jones wrote: the npm stuff is all tool chain; tools that I believe should be packaged as such by packagers. npm is already in Debian: https://packages.debian.org/sid/npm However, just like we can't use CPAN, pear install, pip install and such when building or installing package, we wont be able to use NPM. This means every single dependency that isn't in Debian will need to be packaged. Horizon is an incredibly complex application. Just so we're all on the same page, the components installed by bower for angboard are: angular Because writing an application the size of Horizon without it would be madness :) angular-route Provides structure to the application through URL routing. angular-cookies Provides management of browser cookies in a way that integrates well with angular. angular-sanitize Allows direct embedding of HTML into angular templates, with sanitization. json3 Compatibility for older browsers so JSON works. es5-shim Compatibility for older browsers so Javascript (ECMAScript 5) works. angular-smart-table Table management (population, sorting, filtering, pagination, etc) angular-local-storage Browser local storage with cookie fallback, integrated with angular mechanisms. angular-bootstrap Extensions to angular that leverage bootstrap (modal popups, tabbed displays, ...) font-awesome Additional glyphs to use in the user interface (warning symbol, info symbol, ...) boot Bootstrap for CSS styling (this is the dependency that brings in jquery and requirejs) underscore Javascript utility library providing a ton of features Javascript lacks but Python programmers expect. ng-websocket Angular-friendly interface to using websockets angular-translate Support for localization in angular using message catalogs generated by gettext/transifex. angular-mocks Mocking support for unit testing angular code angular-scenario More support for angular unit tests Additionally, angboard vendors term.js because it was very poorly packaged in the bower ecosystem. +1 for xstatic there I guess :) So those are the components we needed to create the prototype in a few weeks. Not using them would have added months (or possibly years) to the development time. Creating an application of the scale of Horizon without leveraging all that existing work would be like developing OpenStack while barring all use of Python 3rd-party packages. I have no problem with adding dependencies. That's how things work, for sure, I just want to make sure it doesn't become hell, with so many components inter-depending on 100s of them, which would become not manageable. If we define clear boundaries, then fine! The above seems reasonable anyway. Though did you list the dependencies of the above? Also, if the Horizon project starts using something like NPM (which again, is already available in Debian, so it has my preference), will we at least be able to control what version gets in, just like with pip? Because that's a huge concern for me, and this has been very well and carefully addressed during the Juno cycle. I would very much appreciate if the same kind of care was taken again during the Kilo cycle, whatever path we take. How do I use npm by the way? Any pointer? Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/13/2014 08:05 PM, Radomir Dopieralski wrote: On 11/11/14 08:02, Richard Jones wrote: [...] There were some discussions around tooling. We're using xstatic to manage 3rd party components, but there's a lot missing from that environment. I hesitate to add supporting xstatic components on to the already large pile of work we have to do, so would recommend we switch to managing those components with bower instead. For reference the list of 3rd party components I used in angboard* (which is really only a teensy fraction of the total application we'd end up with, so this components list is probably reduced): [...] Just looking at PyPI, it looks like only a few of those are in xstatic, and those are out of date. There is a very good reason why we only have a few external JavaScript libraries, and why they are in those versions. You see, we are not developing Horizon for our own enjoyment, or to install it at our own webserver and be done with it. What we write has to be then packaged for different Linux distributions by the packagers. Those packagers have very little wiggle room with respect to how they can package it all, and what they can include. In particular, libraries should get packaged separately, so that they can upgrade them and apply security patches and so on. Before we used xstatic, they have to go through the sources of Horizon file by file, and replace all of our bundled files with symlinks to what is provided in their distribution. Obviously that was laborious and introduced bugs when the versions of libraries didn't match. So now we have the xstatic system. That means, that the libraries are explicitly listed, with their minimum and maximum version numbers, and it's easy to make a dummy xstatic package that just points at some other location of the static files. This simplifies the work of the packagers. But the real advantage of using the xstatic packages is that in order to add them to Horizon, you need to add them to the global-requirements list, which is being watched and approved by the packagers themselves. That means, that when you try to introduce a new library, or a version of an old library, that is for some reason problematic for any of the distributions (due to licensing issues, due to them needing to remain at an older version, etc.), they get to veto it and you have a chance of resolving the problem early, not dropping it at the last moment on the packagers. Going back to the versions of the xstatic packages that we use, they are so old for a reason. Those are the newest versions that are available with reasonable effort in the distributions for which we make Horizon. If you want to replace this system with anything else, please keep in contact with the packagers to make sure that the resulting process makes sense and is acceptable for them. Thanks a lot for all you wrote above. I 100% agree with it, and you wrote it better than I would have. Also, I'd like to thank you for the work we did together during the Juno cycle. Interactions and communication on IRC were great. I just hope this continues for Kilo, on the line of what you wrote above. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/13/2014 08:32 AM, Richard Jones wrote: I note that the Debian JS guidelines* only recommend that libraries *should* be minified (though I'm unsure why they even recommend that). I'm not sure why. Though what *must* be done, is that source packages, and no point, should ever include a minified version. This should be done either at build time, or at runtime. There's already some issues within the current XStatic packages that I had to deal with (eg: remove these minified versions so it could be uploaded to Debian). Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/13/2014 04:04 PM, Thomas Goirand wrote: On 11/13/2014 12:13 PM, Richard Jones wrote: the npm stuff is all tool chain; tools that I believe should be packaged as such by packagers. npm is already in Debian: https://packages.debian.org/sid/npm However, just like we can't use CPAN, pear install, pip install and such when building or installing package, we wont be able to use NPM. This means every single dependency that isn't in Debian will need to be packaged. Horizon is an incredibly complex application. Just so we're all on the same page, the components installed by bower for angboard are: angular Because writing an application the size of Horizon without it would be madness :) angular-route Provides structure to the application through URL routing. angular-cookies Provides management of browser cookies in a way that integrates well with angular. angular-sanitize Allows direct embedding of HTML into angular templates, with sanitization. json3 Compatibility for older browsers so JSON works. es5-shim Compatibility for older browsers so Javascript (ECMAScript 5) works. angular-smart-table Table management (population, sorting, filtering, pagination, etc) angular-local-storage Browser local storage with cookie fallback, integrated with angular mechanisms. angular-bootstrap Extensions to angular that leverage bootstrap (modal popups, tabbed displays, ...) font-awesome Additional glyphs to use in the user interface (warning symbol, info symbol, ...) boot Bootstrap for CSS styling (this is the dependency that brings in jquery and requirejs) underscore Javascript utility library providing a ton of features Javascript lacks but Python programmers expect. ng-websocket Angular-friendly interface to using websockets angular-translate Support for localization in angular using message catalogs generated by gettext/transifex. angular-mocks Mocking support for unit testing angular code angular-scenario More support for angular unit tests Additionally, angboard vendors term.js because it was very poorly packaged in the bower ecosystem. +1 for xstatic there I guess :) So those are the components we needed to create the prototype in a few weeks. Not using them would have added months (or possibly years) to the development time. Creating an application of the scale of Horizon without leveraging all that existing work would be like developing OpenStack while barring all use of Python 3rd-party packages. I have no problem with adding dependencies. That's how things work, for sure, I just want to make sure it doesn't become hell, with so many components inter-depending on 100s of them, which would become not manageable. If we define clear boundaries, then fine! The above seems reasonable anyway. Though did you list the dependencies of the above? Also, if the Horizon project starts using something like NPM (which again, is already available in Debian, so it has my preference), will we at least be able to control what version gets in, just like with pip? Because that's a huge concern for me, and this has been very well and carefully addressed during the Juno cycle. I would very much appreciate if the same kind of care was taken again during the Kilo cycle, whatever path we take. How do I use npm by the way? Any pointer? NPM and Bower work the similar way as pip, they maintain similar files as requirements.txt that list dependencies and it's versions. I think we should bring up patch that introduces this toolset so we can discuss the real amount of dependencies and the process. It would be also nice to introduce something similar as global-requirements.txt in OpenStack project to make sure we have all deps in one place and get some approval process on versions used. Here is an example of random Angular application's package.json (used by NPM) and bower.json (used by Bower) files: http://fpaste.org/150513/89599214/ I'll try to search for a good article that describes how this ecosystem works. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Jirka ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Thomas Goirand z...@debian.org writes:
Also, if the Horizon project starts using something like NPM (which
again, is already available in Debian, so it has my preference), will we
at least be able to control what version gets in, just like with pip?
Yes, npm similarly to pip in that you can specify the versions you want
to install. You can expect loose versions (like 1.2.x if you're okay
with gettting a random patch version) or you can specify the full
version.
In parallel with that, you can add a shrinkwrap file which lists the
versions to install recursively. This locks down the versions of
indirect dependencies too (one of your dependencies might otherwise
depend on a loose version number).
Because that's a huge concern for me, and this has been very well and
carefully addressed during the Juno cycle. I would very much appreciate
if the same kind of care was taken again during the Kilo cycle, whatever
path we take. How do I use npm by the way? Any pointer?
After installing it, you can try running 'npm install eslint'. That will
create a node_modules folder in your current working directory and
install ESLint inside it. It will also create a cache in ~/.npm.
The ESLint executable is now
node_modules/.bin/eslint
You'll notice that npm creates
node_modules/eslint/node_modules/
and install the ESLint dependencies there. Try removing node_modules,
then install one of the dependencies first followed by ESLint:
rm -r node_modules
npm install object-assign eslint
This will put both object-assign and eslint at the top of node_modules
and object-assign is no longer in node_modules/eslint/node_modules/.
This works because require('object-assign') in NodeJS will search up the
directory tree until it finds the module. So the ESLint code can still
use object-assign.
You can run 'npm dedupe' to move modules up the tree and de-duplicate
the install somewhat.
This nested module system also works the other way: if you run 'npm
install bower' after installing ESLint, you end up with two versions of
object-assign -- check 'npm list object-assign' for a dependency graph.
Surprisingly and unlike, say, Python, executing
require('object-assign')
can give you different modules depending on where the code lives that
execute the statement. This allows different parts of Bower to use
different versions of object-assign. This is seen as a feature in this
world... I fear that it can cause strange problems and bugs when data
travels from one part of the program to another.
So, the philosophy behind this is very different from what we're used to
with system-level package managers (focus on local installs) and even
From what we have in the Python world with pip (multiple versions
installed concurrently).
--
Martin Geisler
http://google.com/+MartinGeisler
pgp6u4TvkQL2G.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Matthias Runge mru...@redhat.com writes: On 13/11/14 15:56, Martin Geisler wrote: Maybe a silly question, but why insist on this? Why would you insist on installing a JavaScript based application using your package manager? I'm a huge fan of package managers and typically refuse to install anything globally if it doesn't come as a package. However, the whole JavaScript ecosystem seems to be centered around the idea of doing local installations. That means that you no longer need the package manager to install the software -- you only need a package manager to install the base system (NodeJs and npm for JavaScript). Yeah, I understand you. Let me just add that this shift has been a very recent change for me. With anything but Python and JavaScript, I use my system-level package manager. But: doing local installs or: installing things aside a package manager means, that software is not maintained, or properly updated any more. I'm a huge fan of not bundling stuff and re-using libraries from a central location. Copying foreign code to your own codebase is quite popular in JavaScript world. That doesn't mean, it's the right thing to do. I agree that you don't want to copy third-party libraries into your code. In some sense, that's not what the JavaScript world is doing, at least not before install time. What I mean is: the ease of use of local package managers has lead to an explosion in the number of tiny packages. So JS projects will no longer copy dependencies into their own project (into their version control system). They will instead depend on them using a package manager such as npm or bower. It seems to me that it should be possible translate the node module into system level packages in a mechanical fashion, assuming that you're willing to have a system package for each version of the node module (you'll need multiple system packages since it's very likely that you'll end up using multiple different versions at the same time -- alternatively, you could let each system package install every published or popular node module version). The guys behind npm has written a little about how that could work here: http://nodejs.org/api/modules.html#modules_addenda_package_manager_tips Has anyone written such wrapper packages? Not the xstatic system which seems to incur a porting effort -- but really a wrapper system that can translate any node module into a system package. -- Martin Geisler http://google.com/+MartinGeisler pgpvFpbyk_SbN.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/13/2014 10:56 PM, Martin Geisler wrote: Maybe a silly question, but why insist on this? Why would you insist on installing a JavaScript based application using your package manager? I'm a huge fan of package managers and typically refuse to install anything globally if it doesn't come as a package. However, the whole JavaScript ecosystem seems to be centered around the idea of doing local installations. That means that you no longer need the package manager to install the software -- you only need a package manager to install the base system (NodeJs and npm for JavaScript). Yeah... Just like for Java, PHP, Perl, Python, you-name-it... In what way Javascript will be any different from all of these languages? Notice that Python has been moving rapidly in the same direction for years: you only need Python and pip to bootstrap yourself. After getting used to virtualenv, I've mostly stopped installing Python modules globally and that is how the JavaScript world expects you to work too. Fine for development. Not for deployments. Not for distributions. Or you just get a huge mess of every library installed 10 times, with 10 different versions, and then a security issue needs to be fixed... So maybe the Horizon package should be an installer package like the ones that download fonts or Adobe? This is a horrible design which will *never* make it to distributions. Please think again. What is it that makes Horizon so special? Answer: nothing. It's just a web app, so it doesn't need any special care. It should be packaged, just like the rest of everything, with .deb/.rpm and so on. That package would get the right version of node and which then runs the npm and bower commands to download the rest plus (importantly and much appreciated) puts the files in a sensible location and gives them good permissions. Fine for your development environment. But that's it. Also, does your $language-specific-package--manager has enough checks so that there's no man in the middle attack possible? Is it secured enough? Can a replay attack be done on it? Does it supports any kind of cryptography checks like yum or apt does? I'm almost sure that's not the case. pip is really horrible in this regard. I haven't checked, but I'm almost sure what we're proposing (eg: npm and such) have the same weakness. And here, I'm only scratching security concerns. There's other concerns, like how good is the dependency solver and such (remember: it took *years* for apt to be as good as it is right now, and it still has some defects). On 11/14/2014 12:59 AM, Martin Geisler wrote: It seems to me that it should be possible translate the node module into system level packages in a mechanical fashion, assuming that you're willing to have a system package for each version of the node module Sure! That's how I do most of my Python modules these days. I don't just create them from scratch, I use my own debpypi script, which generates a template for packaging. But it can't be fully automated. I could almost do it in a fully automated manner for PEAR packages for PHP (see debpear in the Debian archive), but it's harder with Python and pip/PyPi. Stuff like debian/copyright files have to be processed by hand, and each package is different (How to run unit tests? nose, testr, pytest? Does it support python3? Is there a sphinx doc? How good is upstream short and long description?). I guess it's going to be the same for Javascript packages: it will be possible to do automation for some parts, but manual work will always be needed. On 11/14/2014 12:59 AM, Martin Geisler wrote: The guys behind npm has written a little about how that could work here: http://nodejs.org/api/modules.html#modules_addenda_package_manager_tips It's fun to read, but very naive. First thing that is very shocking is that arch independent things gets installed into /usr/lib, where they belong in /usr/share. If that is what the NPM upstream produces, that's scary: he doesn't even know how the FSHS (FileSystem Hierarchy Standard) works. Has anyone written such wrapper packages? Not the xstatic system which seems to incur a porting effort -- but really a wrapper system that can translate any node module into a system package. The xstatic packages are quite painless, from my view point. What's painful is to link an existing xstatic package with an already existing libjs-* package that may have a completely different directory structure. You can then end-up with a forest of symlinks, but there's no way around that. No wrapper can solve that problem either. And more generally, a wrapper that writes a $distribution source package out of a $language-specific package manager will never solve all, it will only reduce the amount of packaging work. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 13, 2014, at 12:38 PM, Thomas Goirand z...@debian.org wrote: On 11/13/2014 10:56 PM, Martin Geisler wrote: Maybe a silly question, but why insist on this? Why would you insist on installing a JavaScript based application using your package manager? I'm a huge fan of package managers and typically refuse to install anything globally if it doesn't come as a package. However, the whole JavaScript ecosystem seems to be centered around the idea of doing local installations. That means that you no longer need the package manager to install the software -- you only need a package manager to install the base system (NodeJs and npm for JavaScript). Yeah... Just like for Java, PHP, Perl, Python, you-name-it... In what way Javascript will be any different from all of these languages? Node.js, and npm in particular tends to solve the dependency hell problem by making it possible to install multiple versions of a particular thing and use them all within the same process. As far as I know the OS tooling doesn’t really handle SxS installs of the same thing in multiple versions very well, I think the closest that you can do is multiple separate packages with version numbers in the package name? In other words it’s entirely possible that a particular set of npm packages can not be resolved to a single version per library. Notice that Python has been moving rapidly in the same direction for years: you only need Python and pip to bootstrap yourself. After getting used to virtualenv, I've mostly stopped installing Python modules globally and that is how the JavaScript world expects you to work too. Fine for development. Not for deployments. Not for distributions. Or you just get a huge mess of every library installed 10 times, with 10 different versions, and then a security issue needs to be fixed… Eh, I wouldn’t say it’s not fine for deployments. Generally I find that the less I tie the things where I care about versions to my OS the happier my life gets. It’s not fine for distributions wanting to offer it though, that is correct. So maybe the Horizon package should be an installer package like the ones that download fonts or Adobe? This is a horrible design which will *never* make it to distributions. Please think again. What is it that makes Horizon so special? Answer: nothing. It's just a web app, so it doesn't need any special care. It should be packaged, just like the rest of everything, with .deb/.rpm and so on. That package would get the right version of node and which then runs the npm and bower commands to download the rest plus (importantly and much appreciated) puts the files in a sensible location and gives them good permissions. Fine for your development environment. But that's it. Also, does your $language-specific-package--manager has enough checks so that there's no man in the middle attack possible? Is it secured enough? Can a replay attack be done on it? Does it supports any kind of cryptography checks like yum or apt does? I'm almost sure that's not the case. pip is really horrible in this regard. I haven't checked, but I'm almost sure what we're proposing (eg: npm and such) have the same weakness. And here, I'm only scratching security concerns. There's other concerns, like how good is the dependency solver and such (remember: it took *years* for apt to be as good as it is right now, and it still has some defects). As far as I’m aware npm supports TLS the same as pip does. That secures the transport between the end users and the repository so you can be assured that there is no man in the middle. Security wise npm (and pip) are about ~95% (mad up numbers, but you can get the gist) of the effectiveness as the OS package managers. On 11/14/2014 12:59 AM, Martin Geisler wrote: It seems to me that it should be possible translate the node module into system level packages in a mechanical fashion, assuming that you're willing to have a system package for each version of the node module Sure! That's how I do most of my Python modules these days. I don't just create them from scratch, I use my own debpypi script, which generates a template for packaging. But it can't be fully automated. I could almost do it in a fully automated manner for PEAR packages for PHP (see debpear in the Debian archive), but it's harder with Python and pip/PyPi. I would be interested to know what makes Python harder in this regard, I would like to fix it. Stuff like debian/copyright files have to be processed by hand, and each package is different (How to run unit tests? nose, testr, pytest? Does it support python3? Is there a sphinx doc? How good is upstream short and long description?). I guess it's going to be the same for Javascript packages: it will be possible to do automation for some parts, but manual work will always be needed. On 11/14/2014 12:59 AM, Martin Geisler wrote: The guys behind npm has written a little about
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
I would like to take a moment to point out that developing system software is different from developing web applications. The way systems software is developed and often deployed is different from web applications. Horizon as it sits today appears to be web application development by systems software developers. This raises the barrier to entry for web application developers. The approach being proposed moves horizon into the realm of web application technologies that web application people use today. The debate as I'm reading it is about taking web application development processes and turning them into systems development processes which are often foreign to web application developers. How is this going to work out? How will web app people be willing to get involved? Why should this be treated the same? Most of OpenStack is a systems problem. This piece is a little different. To make it successful should it get some wiggle room to work well in the space it's in? Note, I'm not saying it should be insecure or anything like that. There are just different approaches. On Thu, Nov 13, 2014 at 1:11 PM, Donald Stufft don...@stufft.io wrote: On Nov 13, 2014, at 12:38 PM, Thomas Goirand z...@debian.org wrote: On 11/13/2014 10:56 PM, Martin Geisler wrote: Maybe a silly question, but why insist on this? Why would you insist on installing a JavaScript based application using your package manager? I'm a huge fan of package managers and typically refuse to install anything globally if it doesn't come as a package. However, the whole JavaScript ecosystem seems to be centered around the idea of doing local installations. That means that you no longer need the package manager to install the software -- you only need a package manager to install the base system (NodeJs and npm for JavaScript). Yeah... Just like for Java, PHP, Perl, Python, you-name-it... In what way Javascript will be any different from all of these languages? Node.js, and npm in particular tends to solve the dependency hell problem by making it possible to install multiple versions of a particular thing and use them all within the same process. As far as I know the OS tooling doesn’t really handle SxS installs of the same thing in multiple versions very well, I think the closest that you can do is multiple separate packages with version numbers in the package name? In other words it’s entirely possible that a particular set of npm packages can not be resolved to a single version per library. Notice that Python has been moving rapidly in the same direction for years: you only need Python and pip to bootstrap yourself. After getting used to virtualenv, I've mostly stopped installing Python modules globally and that is how the JavaScript world expects you to work too. Fine for development. Not for deployments. Not for distributions. Or you just get a huge mess of every library installed 10 times, with 10 different versions, and then a security issue needs to be fixed… Eh, I wouldn’t say it’s not fine for deployments. Generally I find that the less I tie the things where I care about versions to my OS the happier my life gets. It’s not fine for distributions wanting to offer it though, that is correct. So maybe the Horizon package should be an installer package like the ones that download fonts or Adobe? This is a horrible design which will *never* make it to distributions. Please think again. What is it that makes Horizon so special? Answer: nothing. It's just a web app, so it doesn't need any special care. It should be packaged, just like the rest of everything, with .deb/.rpm and so on. That package would get the right version of node and which then runs the npm and bower commands to download the rest plus (importantly and much appreciated) puts the files in a sensible location and gives them good permissions. Fine for your development environment. But that's it. Also, does your $language-specific-package--manager has enough checks so that there's no man in the middle attack possible? Is it secured enough? Can a replay attack be done on it? Does it supports any kind of cryptography checks like yum or apt does? I'm almost sure that's not the case. pip is really horrible in this regard. I haven't checked, but I'm almost sure what we're proposing (eg: npm and such) have the same weakness. And here, I'm only scratching security concerns. There's other concerns, like how good is the dependency solver and such (remember: it took *years* for apt to be as good as it is right now, and it still has some defects). As far as I’m aware npm supports TLS the same as pip does. That secures the transport between the end users and the repository so you can be assured that there is no man in the middle. Security wise npm (and pip) are about ~95% (mad up numbers, but you can get the gist) of the effectiveness as the OS package managers.
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/14/2014 02:11 AM, Donald Stufft wrote: On Nov 13, 2014, at 12:38 PM, Thomas Goirand z...@debian.org wrote: On 11/13/2014 10:56 PM, Martin Geisler wrote: However, the whole JavaScript ecosystem seems to be centered around the idea of doing local installations. That means that you no longer need the package manager to install the software -- you only need a package manager to install the base system (NodeJs and npm for JavaScript). Yeah... Just like for Java, PHP, Perl, Python, you-name-it... In what way Javascript will be any different from all of these languages? Node.js, and npm in particular tends to solve the dependency hell problem by making it possible to install multiple versions of a particular thing and use them all within the same process. As far as I know the OS tooling doesn’t really handle SxS installs of the same thing in multiple versions very well, I think the closest that you can do is multiple separate packages with version numbers in the package name? Yeah, and for a very good reason: having multiple version of the same thing is just really horrible, and should be avoided at all costs. Also, does your $language-specific-package--manager has enough checks so that there's no man in the middle attack possible? Is it secured enough? Can a replay attack be done on it? Does it supports any kind of cryptography checks like yum or apt does? I'm almost sure that's not the case. pip is really horrible in this regard. I haven't checked, but I'm almost sure what we're proposing (eg: npm and such) have the same weakness. And here, I'm only scratching security concerns. There's other concerns, like how good is the dependency solver and such (remember: it took *years* for apt to be as good as it is right now, and it still has some defects). As far as I’m aware npm supports TLS the same as pip does. That secures the transport between the end users and the repository so you can be assured that there is no man in the middle. Security wise npm (and pip) are about ~95% (mad up numbers, but you can get the gist) of the effectiveness as the OS package managers. I don't agree at all with this view. Using TLS is *far* from being enough IMO. But that's not the point. Using anything else than the distribution package manager is a hack (or unfinished work). On 11/14/2014 12:59 AM, Martin Geisler wrote: It seems to me that it should be possible translate the node module into system level packages in a mechanical fashion, assuming that you're willing to have a system package for each version of the node module Sure! That's how I do most of my Python modules these days. I don't just create them from scratch, I use my own debpypi script, which generates a template for packaging. But it can't be fully automated. I could almost do it in a fully automated manner for PEAR packages for PHP (see debpear in the Debian archive), but it's harder with Python and pip/PyPi. I would be interested to know what makes Python harder in this regard, I would like to fix it. The fact that the standard from PyPi is very fuzzy is one of the issue. There's nothing in the format (for example in the DOAP.xml record) that tells if a module supports Python3 for example. Then the short and long descriptions aren't respected, often, you get some changelog entries there. Then there's no real convention for the location of the sphinx doc. There's also the fact that dependencies for Python have to be written by hand on a Debian package. See for example, dependencies on arparse, distribute, ordereddict, which I never put in a Debian package as it's available in Python 2.7. Or the fact that there's no real unique place where dependencies are written on a PyPi package (is it hidden somewhere in setup.py, or is it explicitly written in requirements.txt?). Etc. On the PHP world, everything is much cleaner, in the package.xml, which is very easily parse-able. On 11/14/2014 12:59 AM, Martin Geisler wrote: The guys behind npm has written a little about how that could work here: http://nodejs.org/api/modules.html#modules_addenda_package_manager_tips It's fun to read, but very naive. First thing that is very shocking is that arch independent things gets installed into /usr/lib, where they belong in /usr/share. If that is what the NPM upstream produces, that's scary: he doesn't even know how the FSHS (FileSystem Hierarchy Standard) works. I may be wrong, but doesn’t the FHS state that /usr/share is for arch independent *data* that is read only? No, that's for arch independent *things*. Like for example, javascript. In Debian, these are going in /usr/share/javascript. Python code used to live within /usr/share/pyshared too (but we stopped the symlink forest during the Jessie cycle). I believe it also states that /usr/lib is for object files, libraries, and internal binaries. It's for arch dependent things. As far as I’m aware the things that npm installs are libraries the same
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Thomas Goirand z...@debian.org writes: On 11/13/2014 10:56 PM, Martin Geisler wrote: Maybe a silly question, but why insist on this? Why would you insist on installing a JavaScript based application using your package manager? I'm a huge fan of package managers and typically refuse to install anything globally if it doesn't come as a package. However, the whole JavaScript ecosystem seems to be centered around the idea of doing local installations. That means that you no longer need the package manager to install the software -- you only need a package manager to install the base system (NodeJs and npm for JavaScript). Yeah... Just like for Java, PHP, Perl, Python, you-name-it... In what way Javascript will be any different from all of these languages? Let me again say that I'm fairly new to this modern JavaScript world. I knew almost nothing about node, npm, bower, and grunt six months ago. One answer may be that there isn't an expectation in the JavaScript community that you'll be reusing system libraries the same way that there is in at least the Python and Perl communities. It's my impression that the JavaScript world is used to move *very* quickly. People release versions very rapidly and are happy to break APIs. I think they're okay with it because of semver -- the idea that as long as you increment the right digit in your version number, you don't have to care (much) about the work you put on your users when you ask them to upgrade. I hope that's too harsh a description, but the way the node module system is explicitly designed to allow a single running program to use multiple versions of the *same* module hints that this chaotic situation is both expected and considered normal. When reading about the module system, I came across blog posts that called it superior compared to, say, Python, exactly because of this flexibility. Reasonable people are sure to disagree, but that seems to be the current situation. Notice that Python has been moving rapidly in the same direction for years: you only need Python and pip to bootstrap yourself. After getting used to virtualenv, I've mostly stopped installing Python modules globally and that is how the JavaScript world expects you to work too. Fine for development. Not for deployments. Not for distributions. Or you just get a huge mess of every library installed 10 times, with 10 different versions, and then a security issue needs to be fixed... While I agree that it's chaotic, I also think you make the problem worse than it really is. First, remember that the user who installs Horizon won't need to use the JavaScript based *developer* tools such as npm, bower, etc. That is, I think Horizon developers will use these tools to produce a release -- a tarball -- and that tarball will be something you unpack on your webserver and then you're done. I base this on what I've seen in the project I've been working. The release tarball you download here don't mention npm, bower, or any of the other tools: https://github.com/zerovm/swift-browser/releases The tools were used to produce the tarball and were used to test it, but they're not part of the released product. Somewhat similar to how GCC isn't included in the tarball if you download a pre-compiled binary. So maybe the Horizon package should be an installer package like the ones that download fonts or Adobe? This is a horrible design which will *never* make it to distributions. Please think again. What is it that makes Horizon so special? Answer: nothing. It's just a web app, so it doesn't need any special care. It should be packaged, just like the rest of everything, with .deb/.rpm and so on. Maybe a difference is that you don't (yet) install a web application like you install a system application. Instead you *deploy* it: you unpack files on a webserver, you configure permissions, you setup cache rules, you configure a database, etc. I find this to be quite different from, say, installing Emacs. Emacs is something you install once on a system and this single installation can be done in a right way so that it's useable for several users on the system. A web app is something a single user installs on a system (www-data or a similar user) and then this user configures the system web server to serve this web app. I agree that it would be cool to have web apps be as robust and general purpose as system apps. However, I think that day is a ways off. That package would get the right version of node and which then runs the npm and bower commands to download the rest plus (importantly and much appreciated) puts the files in a sensible location and gives them good permissions. Fine for your development environment. But that's it. Also, does your $language-specific-package--manager has enough checks so that there's no man in the middle attack possible? Is it secured enough? Can a replay attack be done on it? Does it supports any kind of cryptography checks like
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 13, 2014, at 5:23 PM, Thomas Goirand z...@debian.org wrote: On 11/14/2014 02:11 AM, Donald Stufft wrote: On Nov 13, 2014, at 12:38 PM, Thomas Goirand z...@debian.org wrote: On 11/13/2014 10:56 PM, Martin Geisler wrote: However, the whole JavaScript ecosystem seems to be centered around the idea of doing local installations. That means that you no longer need the package manager to install the software -- you only need a package manager to install the base system (NodeJs and npm for JavaScript). Yeah... Just like for Java, PHP, Perl, Python, you-name-it... In what way Javascript will be any different from all of these languages? Node.js, and npm in particular tends to solve the dependency hell problem by making it possible to install multiple versions of a particular thing and use them all within the same process. As far as I know the OS tooling doesn’t really handle SxS installs of the same thing in multiple versions very well, I think the closest that you can do is multiple separate packages with version numbers in the package name? Yeah, and for a very good reason: having multiple version of the same thing is just really horrible, and should be avoided at all costs. I don’t disagree with you that I don’t particularly like that situation, just saying that node.js/npm *is* special in this regard because it’s entirely possible that you can’t resolve things to a single version per dependency and their tooling will just work for that. Also, does your $language-specific-package--manager has enough checks so that there's no man in the middle attack possible? Is it secured enough? Can a replay attack be done on it? Does it supports any kind of cryptography checks like yum or apt does? I'm almost sure that's not the case. pip is really horrible in this regard. I haven't checked, but I'm almost sure what we're proposing (eg: npm and such) have the same weakness. And here, I'm only scratching security concerns. There's other concerns, like how good is the dependency solver and such (remember: it took *years* for apt to be as good as it is right now, and it still has some defects). As far as I’m aware npm supports TLS the same as pip does. That secures the transport between the end users and the repository so you can be assured that there is no man in the middle. Security wise npm (and pip) are about ~95% (mad up numbers, but you can get the gist) of the effectiveness as the OS package managers. I don't agree at all with this view. Using TLS is *far* from being enough IMO. But that's not the point. Using anything else than the distribution package manager is a hack (or unfinished work). This is an argument that I don’t think either of us will convince the other of, so I’ll just say agree to disagree. On 11/14/2014 12:59 AM, Martin Geisler wrote: It seems to me that it should be possible translate the node module into system level packages in a mechanical fashion, assuming that you're willing to have a system package for each version of the node module Sure! That's how I do most of my Python modules these days. I don't just create them from scratch, I use my own debpypi script, which generates a template for packaging. But it can't be fully automated. I could almost do it in a fully automated manner for PEAR packages for PHP (see debpear in the Debian archive), but it's harder with Python and pip/PyPi. I would be interested to know what makes Python harder in this regard, I would like to fix it. The fact that the standard from PyPi is very fuzzy is one of the issue. There's nothing in the format (for example in the DOAP.xml record) that tells if a module supports Python3 for example. Then the short and long descriptions aren't respected, often, you get some changelog entries there. Then there's no real convention for the location of the sphinx doc. There's also the fact that dependencies for Python have to be written by hand on a Debian package. See for example, dependencies on arparse, distribute, ordereddict, which I never put in a Debian package as it's available in Python 2.7. Or the fact that there's no real unique place where dependencies are written on a PyPi package (is it hidden somewhere in setup.py, or is it explicitly written in requirements.txt?). Etc. On the PHP world, everything is much cleaner, in the package.xml, which is very easily parse-able. (This is fairly off topic, so if you want to reply to this in private that’s fine): Nothing that says if it supports py3: Yea, this is a problem, you can somewhat estimate it using the Python 3 classifier though. Short and Long descriptions aren’t respected: I’m not sure what you mean by isn’t respected? No real convention for the location of the sphinx docs: Ok, I’ll add this to the list of things that needs work. Have to write dependencies by hand: Not sure what you mean by not depending on argparse, distribute, ordereddict, etc?
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 14 November 2014 02:04, Thomas Goirand z...@debian.org wrote: On 11/13/2014 12:13 PM, Richard Jones wrote: the npm stuff is all tool chain; tools that I believe should be packaged as such by packagers. npm is already in Debian: https://packages.debian.org/sid/npm However, just like we can't use CPAN, pear install, pip install and such when building or installing package, we wont be able to use NPM. This means every single dependency that isn't in Debian will need to be packaged. Just to be clearer, when I wrote the npm stuff I meant npm and the tools installed by it, so grunt, bower, karma, phantomjs, etc. Not the stuff managed by bower, just the stuff installed by npm. Those npm-based things should be packaged by the distros as tools, just like other programs the distros package. Horizon is an incredibly complex application. Just so we're all on the same page, the components installed by bower for angboard are: angular Because writing an application the size of Horizon without it would be madness :) angular-route Provides structure to the application through URL routing. angular-cookies Provides management of browser cookies in a way that integrates well with angular. angular-sanitize Allows direct embedding of HTML into angular templates, with sanitization. json3 Compatibility for older browsers so JSON works. es5-shim Compatibility for older browsers so Javascript (ECMAScript 5) works. angular-smart-table Table management (population, sorting, filtering, pagination, etc) angular-local-storage Browser local storage with cookie fallback, integrated with angular mechanisms. angular-bootstrap Extensions to angular that leverage bootstrap (modal popups, tabbed displays, ...) font-awesome Additional glyphs to use in the user interface (warning symbol, info symbol, ...) boot Bootstrap for CSS styling (this is the dependency that brings in jquery and requirejs) underscore Javascript utility library providing a ton of features Javascript lacks but Python programmers expect. ng-websocket Angular-friendly interface to using websockets angular-translate Support for localization in angular using message catalogs generated by gettext/transifex. angular-mocks Mocking support for unit testing angular code angular-scenario More support for angular unit tests Additionally, angboard vendors term.js because it was very poorly packaged in the bower ecosystem. +1 for xstatic there I guess :) So those are the components we needed to create the prototype in a few weeks. Not using them would have added months (or possibly years) to the development time. Creating an application of the scale of Horizon without leveraging all that existing work would be like developing OpenStack while barring all use of Python 3rd-party packages. I have no problem with adding dependencies. That's how things work, for sure, I just want to make sure it doesn't become hell, with so many components inter-depending on 100s of them, which would become not manageable. If we define clear boundaries, then fine! The above seems reasonable anyway. Though did you list the dependencies of the above? Again, just so we're clear, yes, the above is *all* the components installed by bower, including dependencies (jquery and requirejs being the *only* dependencies not directly installed). As I mentioned, the dependency trees in bower are significantly simpler than npm packages tend to be (most bower packages have zero or one dependency). The 100s of dependencies are in npm packages - but as Martin Gleiser has pointed out, npm solves that problem with its local install and node_modules directory structures. Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 13/11/14 23:30, Martin Geisler wrote: [...] While I agree that it's chaotic, I also think you make the problem worse than it really is. First, remember that the user who installs Horizon won't need to use the JavaScript based *developer* tools such as npm, bower, etc. That is, I think Horizon developers will use these tools to produce a release -- a tarball -- and that tarball will be something you unpack on your webserver and then you're done. I base this on what I've seen in the project I've been working. The release tarball you download here don't mention npm, bower, or any of the other tools: https://github.com/zerovm/swift-browser/releases The tools were used to produce the tarball and were used to test it, but they're not part of the released product. Somewhat similar to how GCC isn't included in the tarball if you download a pre-compiled binary. [...] Maybe a difference is that you don't (yet) install a web application like you install a system application. Instead you *deploy* it: you unpack files on a webserver, you configure permissions, you setup cache rules, you configure a database, etc. [...] I think I see where the misunderstanding is in this whole discussion. It seems it revolves around the purpose and role of the distribution. From the naive point of view, the role of a Linux distribution is to just collect all the software from respective upstream developers and put it in a single repository, so that it can be easily installed by the users. That's not the case. The role of a distribution is to provide a working ecosystem of software, that is: a) installed and configured in consistent way, b) tested to work with the specific versions that it ships with, c) audited for security, d) maintained, including security patches, e) documented, including external tutorials and the like, f) supported, either by the community or by the companies that provide support, g) free of licensing issues and legal risks as much as possible, h) managed with the common system management tools. In order to do that, they can't just take a tarball and drop it in a directory. They always produce their own builds, to make sure it's the same thing that the source code specifies. They sometimes have to rearrange configuration files, to make them fit the standards in their system. They provide sane configuration defaults. They track the security reports about all the installed components, and apply fixes, often before the security issue is even announced. Basically, a distribution adds a whole bunch of additional guarantees for the software they ship. Those are often long-term guarantees, as they will be supporting our software long after we have forgotten about it already. You say that web development doesn't work like that. That may be true, but that's mostly because if you develop a new web application in-house, and deploy it on your server, you don't really have such large legal risk, configuration complexity or support problem -- you just have to care about that single application, because the packagers from the distribution that you are using are taking care about all the rest of software on your server. -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 12/11/14 08:40, Richard Jones wrote: I believe the nodeenv method of installing node solves this, as it's entirely local to the development environment. See below, this touches package build as well. I will have to go through all dependencies and do a review, if those are acceptable for inclusion e.g in Fedora. The same is true for Thomas Goirand for inclusion in Debian. Petr Belanyi has added optional jshint install for js linting into Horizon and it installs nodejs as it depends on it. Could this approach work for our need of js tooling too? [1] Sigh, this nonsense doesn't go away? This is the third time the same issue comes up. jshint is NOT free software. https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 They're trying to resolve that https://github.com/jshint/jshint/issues/1234 But regardless, jshint doesn't have to be installed from a Linux repository; it's usually installed using npm alongside the other node tools. Thanks for the pointer, this is good news! Regarding package managers, my POV is completely different. From a distributor perspective, where customers expect everything provided from a single source, I'm not using npm, pip etc. I'm packaging that stuff properly before using it. There are companies out there, where security policy does not allow to install software from a third party repository. pypi etc. are considered as a third party here in this context. I would prefer to have the complete tool chain available as (RPM) packages. I am executing unit tests etc. during package build. Our builders don't have access to the internet, downloading any other stuff from the internet is no option. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/11/14 08:02, Richard Jones wrote: There were some discussions around tooling. We're using xstatic to manage 3rd party components, but there's a lot missing from that environment. I hesitate to add supporting xstatic components on to the already large pile of work we have to do, so would recommend we switch to managing those components with bower instead. For reference the list of 3rd party components I used in angboard* (which is really only a teensy fraction of the total application we'd end up with, so this components list is probably reduced): json3 es5-shim angular angular-route angular-cookies angular-animate angular-sanitize angular-smart-table angular-local-storage angular-bootstrap angular-translate font-awesome boot underscore ng-websocket Just looking at PyPI, it looks like only a few of those are in xstatic, and those are out of date. Richard, thank you for writing this up! bower is (not yet) available e.g in Fedora, Debian, Ubuntu. I'm quite a bit skeptical why the need for 3 additional package managers (npm, grunt, bower) for simply installing javascript files. Looking at es5-shim, it pulls in additional 28 dependent packages, json3 has 12 dependencies (including a circular dependency, one circular depencency in dependencies), Not counting dependencies in dependent packages, I can only suspect, this will bring us at least 100 new packages required. Once this is finalized, we'll need people to walk through all deps and put packages up for review, to have those available when kilo is ready. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 12/11/14 09:28, Matthias Runge wrote: Looking at es5-shim, it pulls in additional 28 dependent packages, json3 has 12 dependencies (including a circular dependency, one circular depencency in dependencies), Please scratch that. I'll need to look at that a bit deeper (after another coffee) Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Richard Jones r1chardj0...@gmail.com writes: On 12 November 2014 18:17, Matthias Runge mru...@redhat.com wrote: Sigh, this nonsense doesn't go away? This is the third time the same issue comes up. jshint is NOT free software. https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 They're trying to resolve that https://github.com/jshint/jshint/issues/1234 I've switched to ESLint for my own projects: http://eslint.org/ I find it to be better documented and easier to extend than JSHint. Plus it has had a sane license from the beginning :) -- Martin Geisler http://google.com/+MartinGeisler pgpwAiqU4ITyx.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Hello all, I will write here some things I have noticed in the mailing list and on which I can contribute. Npm and Bower: bower and npm are here for two separate things, the first one is here to handle the dependencies of the webapp (angular, es5shim, d3.js and so on), meanwhile npm is here to handle the development environment (karma, grunt, jasmine, etc...). There is a possibility to avoid the use of bower by using npm alone, but I never tested that so a POC can be required. Grunt vs Gulp: Both are equivalent but I have a preference for the second one. It is more powerful than grunt, more customizable, easier to understand for new comers. All the plugins for an angular development exists and are quite stable. Last but not least the angular team is now pushing to replace its infra from grunt to gulp. But like I said both are equivalent and can be switched easily, so if you are more comfortable with grunt it is not a big deal. Tastipy vs Djangular: Like it has already been said we will use a restful API to handle the client server exchanges, so Tastipy seems to be preferable over Djangular. Jasmine vs Mocha + Chai: I am not really aware of Mocha + Chai features, but I have already used Jasmine for all my angular testing and it feeds all my needs, please use the version 2.0 of it for its powerful features on asynchronous testing and the clean handle of variables in the testings. Last it is only one library which already have plugins for the karma test runner. I do not know if I addressed all the questions but here is a starting. Cheers Max - Original Message - From: Matthias Runge mru...@redhat.com To: openstack-dev@lists.openstack.org Sent: Wednesday, November 12, 2014 9:39:29 AM Subject: Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon On 12/11/14 09:28, Matthias Runge wrote: Looking at es5-shim, it pulls in additional 28 dependent packages, json3 has 12 dependencies (including a circular dependency, one circular depencency in dependencies), Please scratch that. I'll need to look at that a bit deeper (after another coffee) Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Richard Jones r1chardj0...@gmail.com writes: Hi all, At the summit last week, we developed a plan for moving forward with modernising Horizon's UI using AngularJS. If you weren't at that meeting and are interested in helping out with this effort please let me know! I have been working on an interface for Swift written in Angular. You'll find a ready-to-deploy tarball and zip file here: https://github.com/zerovm/swift-browser/releases/ This will eventually be an interface for starting ZeroVM instances on a Swift cluster, but for now it's a pure Swift interface. I don't know if this is something that can be used in Horizon, but maybe there are useful bits. One thing interesting thing in Swift Browser is its testing: I wrote a Swift simulator that is injected into the application for end-to-end testing with Protractor. Is is basically an in-memory (and in-browser) Swift that responds to queries made against the Swift REST API. I've found that very useful in the development: https://github.com/zerovm/swift-browser/blob/master/app/js/test/swift-simulator.js I don't know the other OpenStack components and their APIs very well, but I guess that this kind of mocking might be too much for a full OpenStack dashboard. -- Martin Geisler http://google.com/+MartinGeisler pgpHlDlDL2w3E.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Thanks Martin, I remember that eslint was mentioned at some point to replace jshint. Maybe, it can be interesting to look at this, the extension possibility is a major point for a switch. - Original Message - From: Martin Geisler mar...@geisler.net To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Wednesday, November 12, 2014 10:37:15 AM Subject: Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon Richard Jones r1chardj0...@gmail.com writes: Hi all, At the summit last week, we developed a plan for moving forward with modernising Horizon's UI using AngularJS. If you weren't at that meeting and are interested in helping out with this effort please let me know! I have been working on an interface for Swift written in Angular. You'll find a ready-to-deploy tarball and zip file here: https://github.com/zerovm/swift-browser/releases/ This will eventually be an interface for starting ZeroVM instances on a Swift cluster, but for now it's a pure Swift interface. I don't know if this is something that can be used in Horizon, but maybe there are useful bits. One thing interesting thing in Swift Browser is its testing: I wrote a Swift simulator that is injected into the application for end-to-end testing with Protractor. Is is basically an in-memory (and in-browser) Swift that responds to queries made against the Swift REST API. I've found that very useful in the development: https://github.com/zerovm/swift-browser/blob/master/app/js/test/swift-simulator.js I don't know the other OpenStack components and their APIs very well, but I guess that this kind of mocking might be too much for a full OpenStack dashboard. -- Martin Geisler http://google.com/+MartinGeisler ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/11/2014 03:02 PM, Richard Jones wrote: Hi all, At the summit last week, we developed a plan for moving forward with modernising Horizon's UI using AngularJS. If you weren't at that meeting and are interested in helping out with this effort please let me know! The relevant etherpad from the meeting: https://etherpad.openstack.org/p/kilo-horizon-contributors-meetup TL;DR: piece by piece we will replace Django views in Horizon with angular views, and we're going to start with Identity First up, I'd like to ask the UX folk who raised their hands in that meeting to indicate which of the Identity panes we should start with. I believe a wizard was mentioned, as a way to exercise the new wizard code from Maxime. At the same time, I'm looking at updating the AngularJS recommendations in the wiki. I believe other aspects of the current approach to angular code should also be revisited, if we're to scale up to the full angular front-end envisaged. I'd appreciate if those interested in this aspect in particular could contact me so we can sort this out as a team! I'd like to start the design work for the new REST API layer we'll be exposing to the angular application code, but that is also part of the broader discussion about the structure of the angular code in the Horizon application as mentioned above. Should it be a new blueprint/spec? There were some discussions around tooling. We're using xstatic to manage 3rd party components, but there's a lot missing from that environment. I hesitate to add supporting xstatic components on to the already large pile of work we have to do, so would recommend we switch to managing those components with bower instead. For reference the list of 3rd party components I used in angboard* (which is really only a teensy fraction of the total application we'd end up with, so this components list is probably reduced): json3 es5-shim angular angular-route angular-cookies angular-animate angular-sanitize angular-smart-table angular-local-storage angular-bootstrap angular-translate font-awesome boot underscore ng-websocket Hi, Let's please continue to use xstatic packages, which are very good for distributions. I will do the effort on the Debian side of things to make sure that we have all packaged correctly. Using some magic that do the same kind of horrible stuff as pip install or pear install or CPAN, or... you name it, will not fix things on my side. Quite the opposite, it will add issues after issues. The XStatic things, on the other hand, is very easy for me to maintain, even though that's a lot of work. On 11/12/2014 04:03 PM, Matthias Runge wrote: There are companies out there, where security policy does not allow to install software from a third party repository. pypi etc. are considered as a third party here in this context. It's also not Debian policy compliant to download from these sources in a package. So that'd be a no-go for Debian, and anyway, these dependencies would have to be packaged in some way. Again, let's please stick on the xstatic way of doing things. It worked for Juno, let's make it work for Kilo as well. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/12/2014 02:17 AM, Matthias Runge wrote: On 11/11/14 10:53, Jiri Tomasek wrote: Hey, Thanks for writing this up! The Storyboard project has successfully integrated these tools into the OpenStack CI environment. OpenStack CI and distributors are different, because OpenStack CI does not distribute software. Using javascript tooling (yeoman, grunt, bower, etc.) has this issue of being dependent on nodejs which if I recall correctly is causing problems for packagers as some versions of these tools require different nodejs versions - please Mathias correct me if I am wrong. I know this discussion has been here before, but using these tools is necessary for effective development. So we need to resolve the problem asap. Storyboard does not have this issue as it is infra thing. As far as I know, those tools don't require different nodejs versions. But: we can not have different node.js versions installed at the same time. I assume, this is true for all distributions. Creating and maintaining parallel installable versions just sucks and causes many issues. In the past, customers wanted the complete tool-chain to modify their version of dashboard. I understand, this is not an issue, for all companies not distributing software, but directly providing services based on that software. I will have to go through all dependencies and do a review, if those are acceptable for inclusion e.g in Fedora. The same is true for Thomas Goirand for inclusion in Debian. Petr Belanyi has added optional jshint install for js linting into Horizon and it installs nodejs as it depends on it. Could this approach work for our need of js tooling too? [1] Sigh, this nonsense doesn't go away? This is the third time the same issue comes up. jshint is NOT free software. https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 Reasonable people disagree on this point. I, for one, am one of them. In my opnion: A usage clause in a license header is bonghits and unenforcable, as copyright terms cover the legality of copying things, not how you use them. For those terms to be enforcable, it would need to be a EULA, which jshint does not have. However, other people disagree with me, which is their prerogative. I'm mainly suggesting that it's probably not nonsense, it's a disagreement, and it's also probably not going to go away, since jshint is far and away the most common javascript linting tool in existence. Maybe some of the people who are vehemently opposed to the clause in the license header should figure out an alternative. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/12/2014 02:40 AM, Richard Jones wrote: On 12 November 2014 18:17, Matthias Runge mru...@redhat.com wrote: On 11/11/14 10:53, Jiri Tomasek wrote: Hey, Thanks for writing this up! The Storyboard project has successfully integrated these tools into the OpenStack CI environment. OpenStack CI and distributors are different, because OpenStack CI does not distribute software. Ah, I wasn't clear; my concern was whether the tools chosen would be compatible with the CI environment. I'm hoping that distribution of the tools isn't our concern (see below). u Using javascript tooling (yeoman, grunt, bower, etc.) has this issue of being dependent on nodejs which if I recall correctly is causing problems for packagers as some versions of these tools require different nodejs versions - please Mathias correct me if I am wrong. I know this discussion has been here before, but using these tools is necessary for effective development. So we need to resolve the problem asap. Storyboard does not have this issue as it is infra thing. As far as I know, those tools don't require different nodejs versions. But: we can not have different node.js versions installed at the same time. I assume, this is true for all distributions. Creating and maintaining parallel installable versions just sucks and causes many issues. I believe the nodeenv method of installing node solves this, as it's entirely local to the development environment. Just for the record, I believe that we should chose the tools that make sense for making our software, as long as it's not physically impossible for them to be packaged. This means we should absolutely not use things that require multiple versions of node to be needed. The nodejs that's in trusty is new enough to work with all of the modern javascript tool chain things needed for this, so other than the various javascript tools and libraries not being packaged in the distros yet, it should be fine. That a bunch of javascript libraries will need to be distro pacakged should not be a blocker (although I don't think that anyone is saying it is) That is, after all, the important work the distros do. At this point, given the popularity of javascript and javascript tooling, I'm pretty sure the problem is going to have to be solved at some point. I will have to go through all dependencies and do a review, if those are acceptable for inclusion e.g in Fedora. The same is true for Thomas Goirand for inclusion in Debian. Petr Belanyi has added optional jshint install for js linting into Horizon and it installs nodejs as it depends on it. Could this approach work for our need of js tooling too? [1] Sigh, this nonsense doesn't go away? This is the third time the same issue comes up. jshint is NOT free software. https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 They're trying to resolve that https://github.com/jshint/jshint/issues/1234 But regardless, jshint doesn't have to be installed from a Linux repository; it's usually installed using npm alongside the other node tools. Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/12/2014 02:35 PM, Monty Taylor wrote: On 11/12/2014 02:40 AM, Richard Jones wrote: On 12 November 2014 18:17, Matthias Runge mru...@redhat.com wrote: On 11/11/14 10:53, Jiri Tomasek wrote: Hey, Thanks for writing this up! The Storyboard project has successfully integrated these tools into the OpenStack CI environment. OpenStack CI and distributors are different, because OpenStack CI does not distribute software. Ah, I wasn't clear; my concern was whether the tools chosen would be compatible with the CI environment. I'm hoping that distribution of the tools isn't our concern (see below). u Using javascript tooling (yeoman, grunt, bower, etc.) has this issue of being dependent on nodejs which if I recall correctly is causing problems for packagers as some versions of these tools require different nodejs versions - please Mathias correct me if I am wrong. I know this discussion has been here before, but using these tools is necessary for effective development. So we need to resolve the problem asap. Storyboard does not have this issue as it is infra thing. As far as I know, those tools don't require different nodejs versions. But: we can not have different node.js versions installed at the same time. I assume, this is true for all distributions. Creating and maintaining parallel installable versions just sucks and causes many issues. I believe the nodeenv method of installing node solves this, as it's entirely local to the development environment. Just for the record, I believe that we should chose the tools that make sense for making our software, as long as it's not physically impossible for them to be packaged. This means we should absolutely not use things that require multiple versions of node to be needed. The nodejs that's in trusty is new enough to work with all of the modern javascript tool chain things needed for this, so other than the various javascript tools and libraries not being packaged in the distros yet, it should be fine. That a bunch of javascript libraries will need to be distro pacakged should not be a blocker (although I don't think that anyone is saying it is) That is, after all, the important work the distros do. At this point, given the popularity of javascript and javascript tooling, I'm pretty sure the problem is going to have to be solved at some point. +1, I am really glad this has been said. I will have to go through all dependencies and do a review, if those are acceptable for inclusion e.g in Fedora. The same is true for Thomas Goirand for inclusion in Debian. Petr Belanyi has added optional jshint install for js linting into Horizon and it installs nodejs as it depends on it. Could this approach work for our need of js tooling too? [1] Sigh, this nonsense doesn't go away? This is the third time the same issue comes up. jshint is NOT free software. https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 They're trying to resolve that https://github.com/jshint/jshint/issues/1234 But regardless, jshint doesn't have to be installed from a Linux repository; it's usually installed using npm alongside the other node tools. Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Jiri ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/11/2014 08:02 AM, Richard Jones wrote: Hi all, At the summit last week, we developed a plan for moving forward with modernising Horizon's UI using AngularJS. If you weren't at that meeting and are interested in helping out with this effort please let me know! The relevant etherpad from the meeting: https://etherpad.openstack.org/p/kilo-horizon-contributors-meetup TL;DR: piece by piece we will replace Django views in Horizon with angular views, and we're going to start with Identity First up, I'd like to ask the UX folk who raised their hands in that meeting to indicate which of the Identity panes we should start with. I believe a wizard was mentioned, as a way to exercise the new wizard code from Maxime. At the same time, I'm looking at updating the AngularJS recommendations in the wiki. I believe other aspects of the current approach to angular code should also be revisited, if we're to scale up to the full angular front-end envisaged. I'd appreciate if those interested in this aspect in particular could contact me so we can sort this out as a team! I'd like to start the design work for the new REST API layer we'll be exposing to the angular application code, but that is also part of the broader discussion about the structure of the angular code in the Horizon application as mentioned above. Should it be a new blueprint/spec? There were some discussions around tooling. We're using xstatic to manage 3rd party components, but there's a lot missing from that environment. I hesitate to add supporting xstatic components on to the already large pile of work we have to do, so would recommend we switch to managing those components with bower instead. For reference the list of 3rd party components I used in angboard* (which is really only a teensy fraction of the total application we'd end up with, so this components list is probably reduced): json3 es5-shim angular angular-route angular-cookies angular-animate angular-sanitize angular-smart-table angular-local-storage angular-bootstrap angular-translate font-awesome boot underscore ng-websocket Just looking at PyPI, it looks like only a few of those are in xstatic, and those are out of date. grunt provides a lot of features for developing an angular interface. In particular LiveReload accelerates development significantly. There's a django-livereload but it uses tiny-lr under the hood, so we're still using a node application for LiveReload support... so it might make sense to just use grunt. grunt provides many other features as well (wiredep integration with bower, build facilities with ngMin, test monitoring and reload etc). There seemed to be agreement to move to jasmine (from qunit) for writing the tests. It's not noted in the etherpad, but I recall karma was accepted as a given for the test runner. For those not in the meeting, angboard uses mocha+chai for test writing, but I agreed that jasmine is acceptable, and is already used by Storyboard (see below). Also, phantomjs so we don't have to fire up a browser for exercising (what should hopefully be an extensive) unit test suite. The Storyboard project has successfully integrated these tools into the OpenStack CI environment. Richard * https://github.com/r1chardj0n3s/angboard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I am going to try to conclude what has been said in emails across this thread. As Monty Taylor said, nodejs itself is not a blocker as multiple versions of it should not be needed by our tools. (That's also what npm and bower are taking care of, right?) Only thing that is required is that all tools/js libs we want to use would eventually have to be packaged. This is just a bunch of work for packagers. Approach on using Xstatic packages vs Js tooling: As only problem with using js tooling should be just actual packaging of it, I think it makes sense to use these tools and make development simpler then going other way around and using Xstatic packages - which means devs would have to care about getting stuff packaged as xstatic and added to the code, while maintaining proper versions and making sure that they work ok together. NPM and Bower do this for us. Common sense tells me packagers should take care of packaging. Packaging of these tools will have to get resolved somehow anyway, as there will be rise in requirements of using them not just from Horizon... Which tools should we use eventually: Based on the contributions by Maxime, Martin and the others, I think the list of tools should end up as follows: Tooling: npm bower gulp Jasmine Karma/Protractor(?)/eslint ...? Bower and Gulp seems to get along well (https://github.com/yeoman/generator-gulp-webapp) Tastypie on the Django side Angular
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 12/11/14 15:12, Jiri Tomasek wrote: Approach on using Xstatic packages vs Js tooling: As only problem with using js tooling should be just actual packaging of it, I think it makes sense to use these tools and make development simpler then going other way around and using Xstatic packages - which means devs would have to care about getting stuff packaged as xstatic and added to the code, while maintaining proper versions and making sure that they work ok together. NPM and Bower do this for us. Common sense tells me packagers should take care of packaging. Packaging of these tools will have to get resolved somehow anyway, as there will be rise in requirements of using them not just from Horizon... I can't speak for the rest but that part doesn't seem correct to me. The XStatic packages are Python packages (as in, dependencies) that the Horizon team is responsible for (when they don't already exist) and maintains on stackforge, so we do have to create them and make sure they all work well together. The later packaging as rpm or deb or others is left to the distro packagers of course. There are instructions already on how to create xstatic packages [1], it's not very complicated and just takes some review time. Thanks, Julie [1] http://docs.openstack.org/developer/horizon/contributing.html#javascript-and-css-libraries ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/12/2014 05:18 PM, Julie Pichon wrote: On 12/11/14 15:12, Jiri Tomasek wrote: Approach on using Xstatic packages vs Js tooling: As only problem with using js tooling should be just actual packaging of it, I think it makes sense to use these tools and make development simpler then going other way around and using Xstatic packages - which means devs would have to care about getting stuff packaged as xstatic and added to the code, while maintaining proper versions and making sure that they work ok together. NPM and Bower do this for us. Common sense tells me packagers should take care of packaging. Packaging of these tools will have to get resolved somehow anyway, as there will be rise in requirements of using them not just from Horizon... I can't speak for the rest but that part doesn't seem correct to me. The XStatic packages are Python packages (as in, dependencies) that the Horizon team is responsible for (when they don't already exist) and maintains on stackforge, so we do have to create them and make sure they all work well together. The later packaging as rpm or deb or others is left to the distro packagers of course. There are instructions already on how to create xstatic packages [1], it's not very complicated and just takes some review time. Thanks, Julie [1] http://docs.openstack.org/developer/horizon/contributing.html#javascript-and-css-libraries ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I might have expressed myself wrong about XStatic packages. But as you say as well, to use XStatic packages, we need to most often create them and maintain the correct versions we require in Horizon and they don't help to packagers either. I makes sense to use them in Django application as they can be included in requirements.txt and we don't have to carry them directly in the code. So I am definitely ok to use them for Django dependencies we have. Similar thing is npm and bower doing on the Angular side except for we don't have to create these libraries as they already exist. NPM and Bower are taking care of including the right versions of js libs our dev env and our application needs. They use similar description files as requirements.txt in Django. It makes no sense not to use them and try to include js libraries using XStatic packages and listing them in requirements.txt because this way we don't know which version of js lib to use etc. NPM and bower are doing this for us. In both approaches dependencies need to have packages in the end regadles of being it XStatic package or js library or Angular module. It is about using the right tools for the job. I see relation between Nodejs and js libs/tools and Angular app defining it's dependencies using NPM and Bower quite similar as Ruby, Rubygems and Rails application defining it's dependencies in Gemfile.lock. Rubygems are being packaged in distros, so why shouldn't node packages? Jiri ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/12/2014 03:03 AM, Matthias Runge wrote: On 12/11/14 08:40, Richard Jones wrote: I believe the nodeenv method of installing node solves this, as it's entirely local to the development environment. See below, this touches package build as well. I will have to go through all dependencies and do a review, if those are acceptable for inclusion e.g in Fedora. The same is true for Thomas Goirand for inclusion in Debian. Petr Belanyi has added optional jshint install for js linting into Horizon and it installs nodejs as it depends on it. Could this approach work for our need of js tooling too? [1] Sigh, this nonsense doesn't go away? This is the third time the same issue comes up. jshint is NOT free software. https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 They're trying to resolve that https://github.com/jshint/jshint/issues/1234 But regardless, jshint doesn't have to be installed from a Linux repository; it's usually installed using npm alongside the other node tools. Thanks for the pointer, this is good news! Regarding package managers, my POV is completely different. From a distributor perspective, where customers expect everything provided from a single source, I'm not using npm, pip etc. I'm packaging that stuff properly before using it. There are companies out there, where security policy does not allow to install software from a third party repository. pypi etc. are considered as a third party here in this context. I would prefer to have the complete tool chain available as (RPM) packages. I am executing unit tests etc. during package build. Our builders don't have access to the internet, downloading any other stuff from the internet is no option. And this is really not-negotiable, too. Debian has the same requirements, it is not Red Hat/Fedora speciufic. It is no different than the Python Code. We dodn't pip install for RHOSP for the Python packages, and we can't use any of the language specific package managers. But, we are used to it, and dealing with package dependencies is what we do. Having these in Fedora, while painful, will be ultimately very valuable. Matthias ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 13 November 2014 09:35, Adam Young ayo...@redhat.com wrote: On 11/12/2014 03:03 AM, Matthias Runge wrote: On 12/11/14 08:40, Richard Jones wrote: I believe the nodeenv method of installing node solves this, as it's entirely local to the development environment. See below, this touches package build as well. I will have to go through all dependencies and do a review, if those are acceptable for inclusion e.g in Fedora. The same is true for Thomas Goirand for inclusion in Debian. Petr Belanyi has added optional jshint install for js linting into Horizon and it installs nodejs as it depends on it. Could this approach work for our need of js tooling too? [1] Sigh, this nonsense doesn't go away? This is the third time the same issue comes up. jshint is NOT free software. https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 They're trying to resolve that https://github.com/jshint/ jshint/issues/1234 But regardless, jshint doesn't have to be installed from a Linux repository; it's usually installed using npm alongside the other node tools. Thanks for the pointer, this is good news! Regarding package managers, my POV is completely different. From a distributor perspective, where customers expect everything provided from a single source, I'm not using npm, pip etc. I'm packaging that stuff properly before using it. There are companies out there, where security policy does not allow to install software from a third party repository. pypi etc. are considered as a third party here in this context. I would prefer to have the complete tool chain available as (RPM) packages. I am executing unit tests etc. during package build. Our builders don't have access to the internet, downloading any other stuff from the internet is no option. And this is really not-negotiable, too. Debian has the same requirements, it is not Red Hat/Fedora speciufic. It is no different than the Python Code. We dodn't pip install for RHOSP for the Python packages, and we can't use any of the language specific package managers. But, we are used to it, and dealing with package dependencies is what we do. Having these in Fedora, while painful, will be ultimately very valuable. I guess a point of difference here is that the Linux packaging systems target global installation, whereas bower and npm (in our case) target a local installation. It's effectively vendoring the code, but you still have to pull it down from a repository for each installation. Don't worry, I'm not going to suggest we actually move back to vendoring :) We're currently using xstatic and that works with Linux packaging because it was designed to cope with being a global installation. The current Horizon codebase has a django-xstatic plugin which further makes dealing with xstatic components nicer - for example it handles path management and static file compilation (JS minification and concatenation, for example). That's really nice, but poses some problems: - we would need to xstatic-ify (and deb/rpm-ify) all those components - we could run into global version conflict issues if we run more than one service on a system - is this likely to be an issue in practise though? - as far as I'm aware, the xstatic JS minification is not angular-aware, and will break angular code that has not been written to be dumb-minifier-aware (the angular minifier ngMin is written in node and knows how to do things more correctly); adding dumb-minifier-awareness to angular code makes it ugly and more error-prone :/ On the minification issue though: I talked to a few people at the summit and we agreed that minification should just be avoided; gzip at the transport layer buys significantly better compression, and has the added bonus that the resultant code is readable. I note that the Debian JS guidelines* only recommend that libraries *should* be minified (though I'm unsure why they even recommend that). One further note on xstatic vs. bower: during development it's very useful to be able to install random components from bower to try them out; during angboard development I would sometimes install and remove half a dozen components just in one day. I would hope that whatever solution we come up with has at least some of this flexibility for experimentation. Richard * https://wiki.debian.org/Javascript/Policy ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On 11/12/2014 11:12 PM, Jiri Tomasek wrote: As Monty Taylor said, nodejs itself is not a blocker as multiple versions of it should not be needed by our tools. (That's also what npm and bower are taking care of, right?) Only thing that is required is that all tools/js libs we want to use would eventually have to be packaged. This is just a bunch of work for packagers. Approach on using Xstatic packages vs Js tooling: As only problem with using js tooling should be just actual packaging of it, I think it makes sense to use these tools and make development simpler then going other way around and using Xstatic packages - which means devs would have to care about getting stuff packaged as xstatic and added to the code, while maintaining proper versions and making sure that they work ok together. NPM and Bower do this for us. Common sense tells me packagers should take care of packaging. Packaging of these tools will have to get resolved somehow anyway, as there will be rise in requirements of using them not just from Horizon... I see an obvious issue with your approach. Just like with pip and PyPi, it's going to be just one line of patch for Horizon people to add yet-another-javascript-dependency. It's already a dependency hell. I have created 21 new python-xstatic packages in Debian, and at least, for half a dozen of them (maybe more), I also packaged the libjs-* packages. It took a long time to have them in order, and it just got stabilized for Juno (there's still something to fix in Jessie, but I don't really care as it's Icehouse there...). Now, your motivation to go into the direction of using tooling seems to be motivated so that it's super easy to add more. And more... And more again. This leads me to believe that it's going to be even more crazy. When is this going to stop? We're definitively enlarging the potential surface of attack and potential security breaches. And OpenStack at large is not at all doing great in terms of security. [1] I just don't understand why all of this is needed. I did some JS programming myself back in the days (10 years ago, using PHP...). I could do Ajax by hand, without using a single library. What is it that you're doing in Horizon that needs so many libs? When I see Horizon in Juno, I don't even get it. Or is it because you just want to have fancy animation? Frankly, I don't care these. I care a lot more that we're not adding new potential security problems. On 11/13/2014 12:18 AM, Julie Pichon wrote: There are instructions already on how to create xstatic packages [1], it's not very complicated and just takes some review time. Exactly!!! And if someone can't make the effort to package something as an XStatic lib, then he doesn't deserve the rights to add a new libjs depends on the project. :) Also, why do we need to: 1/ Change our procedure starting on each new cycle 2/ Constantly reinvent the wheel 3/ Make my life miserable... :) On 11/13/2014 01:23 AM, Jiri Tomasek wrote: I might have expressed myself wrong about XStatic packages. But as you say as well, to use XStatic packages, we need to most often create them and maintain the correct versions we require in Horizon and they don't help to packagers either. They do! Well, except Ubuntu guys who found it funny to re-embed all the Javascript back in Horizon, but for all sane people doing packaging in this world, it's very helpful!!! Similar thing is npm and bower doing on the Angular side except for we don't have to create these libraries as they already exist. NPM and Bower are taking care of including the right versions of js libs our dev env and our application needs. They use similar description files as requirements.txt in Django. How are you going to express the changes in the global-requirements.txt? Is this just going to live within Horizon? How about projects like tuskar-ui, or murano-dashboard? There simply wont be a single unique place for me to look at to see what work I need to process. And no unique place either to have an overview and see how bad we're doing in terms of dependency (ie: we have a way too many...). It makes no sense not to use them and try to include js libraries using XStatic packages and listing them in requirements.txt because this way we don't know which version of js lib to use etc. Not correct. XStatic package versions are following the one of upstream libjs packages. For example, python-xstatic-jsencrypt in Debian has version 2.0.0.2, and depends on version 2.0.0 of jsencrypt. If there's evolution of the dependency, then the XStatic package version will also change. I think it's very easy to follow, and a very good idea. The thing is, using all these NodeJS stuff, we'll end-up depending on Node so much, that it's going to pull about a 100 more dependencies (or even more). Maybe it's going to be only build-depends, but that's the same from my point of view: it's going to be packages that we'll need to take care of in the distributions,
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
On Nov 12, 2014, at 8:29 PM, Thomas Goirand z...@debian.org wrote: On 11/12/2014 09:31 PM, Monty Taylor wrote: jshint is NOT free software. https://github.com/jshint/jshint/blob/master/src/jshint.js#L19 Reasonable people disagree on this point. Feel free to have this debate with the entire Debian community. When you're done, then come back to us, and we can use jshint. In the mean while, let's not use it. (and by the way, read again the Debian Free Software Guideline and especially point number 5 and 6, I'm not saying that this is *my* point of argumentation, but it has been so within some threads in Debian). The terrible line in the jshint license is going to go away in the future, you can read https://github.com/jshint/jshint/issues/1234#issuecomment-56875247 but the tl;dr is that Doug Crockford gave Eclipse a license to his software without that line, and eclipse is sharing it so jshint is going to rebase off of the “Crockford” version and onto the Eclipse version. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Two things of note, having been doing heavy javascript development over the past couple years: 1) NPM actually resolves conflicts in a dependency tree. Unlike Python, it can ensure that if different packages declare conflicting versions, each one gets the version it requested. And conflicting dependencies are very common in JS packages. That's gonna be a nightmare for more traditional packaging systems if you try to shoehorn them together. 2) The OpenStack community is overwhelmingly fond of reinventing wheels. The JavaScript community has the opposite habit, and tends to create a package for every last function (that's why some of these seemingly single-purpose tools have 50+ dependencies). I caution against putting in place a system that penalizes developers for trying to use established tools or to use packages that do the job and do it well (such as forcing them to deal with xstatic packaging). While I respect the licensing and packaging concerns involved in large dependency trees, we need to stop encouraging people to reinvent wheels. I think most folks at this point agree that the future of Horizon is to become a JavaScript-driven web app. It's the way the industry is going this point, and it's the way to satisfy what users want and expect. Provided we accept that future, we should strive to embrace the best practices of that development community, not to tell them we don't like it or it doesn't fit how OpenStack does things. On a historical note, OpenStack had a very rocky relationship with the broader Python community in its early days. Since then, thanks to great efforts on both sides, that relationship has gotten much better. Let's try not to replay history by doing the same thing with the JavaScript/Node community. We have to attract great developers so we can produce the best possible OpenStack. I remind people to think about how we can do *that* first and how we can deal with distro requirements to support the process second. - Gabriel -Original Message- From: Thomas Goirand [mailto:z...@debian.org] Sent: Wednesday, November 12, 2014 5:30 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon On 11/12/2014 11:12 PM, Jiri Tomasek wrote: As Monty Taylor said, nodejs itself is not a blocker as multiple versions of it should not be needed by our tools. (That's also what npm and bower are taking care of, right?) Only thing that is required is that all tools/js libs we want to use would eventually have to be packaged. This is just a bunch of work for packagers. Approach on using Xstatic packages vs Js tooling: As only problem with using js tooling should be just actual packaging of it, I think it makes sense to use these tools and make development simpler then going other way around and using Xstatic packages - which means devs would have to care about getting stuff packaged as xstatic and added to the code, while maintaining proper versions and making sure that they work ok together. NPM and Bower do this for us. Common sense tells me packagers should take care of packaging. Packaging of these tools will have to get resolved somehow anyway, as there will be rise in requirements of using them not just from Horizon... I see an obvious issue with your approach. Just like with pip and PyPi, it's going to be just one line of patch for Horizon people to add yet-another-javascript-dependency. It's already a dependency hell. I have created 21 new python-xstatic packages in Debian, and at least, for half a dozen of them (maybe more), I also packaged the libjs-* packages. It took a long time to have them in order, and it just got stabilized for Juno (there's still something to fix in Jessie, but I don't really care as it's Icehouse there...). Now, your motivation to go into the direction of using tooling seems to be motivated so that it's super easy to add more. And more... And more again. This leads me to believe that it's going to be even more crazy. When is this going to stop? We're definitively enlarging the potential surface of attack and potential security breaches. And OpenStack at large is not at all doing great in terms of security. [1] I just don't understand why all of this is needed. I did some JS programming myself back in the days (10 years ago, using PHP...). I could do Ajax by hand, without using a single library. What is it that you're doing in Horizon that needs so many libs? When I see Horizon in Juno, I don't even get it. Or is it because you just want to have fancy animation? Frankly, I don't care these. I care a lot more that we're not adding new potential security problems. On 11/13/2014 12:18 AM, Julie Pichon wrote: There are instructions already on how to create xstatic packages [1], it's not very complicated and just takes some review time. Exactly!!! And if someone can't make
Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon
Gabriel has responded saying very much what I would have said, so I won't repeat that. I would like to note though that bower and npm are two separate beasts entirely. The dependency trees in bower are very limited indeed (only two additional components are installed beyond the list below) which is in stark contrast to npm. The xstatic question only applies to the bower components - the npm stuff is all tool chain; tools that I believe should be packaged as such by packagers. I would like to address one specific point Thomas raised though: On 13 November 2014 12:29, Thomas Goirand z...@debian.org wrote: I just don't understand why all of this is needed. I did some JS programming myself back in the days (10 years ago, using PHP...). I could do Ajax by hand, without using a single library. What is it that you're doing in Horizon that needs so many libs? When I see Horizon in Juno, I don't even get it. Or is it because you just want to have fancy animation? Frankly, I don't care these. I care a lot more that we're not adding new potential security problems. Horizon is an incredibly complex application. Just so we're all on the same page, the components installed by bower for angboard are: angular Because writing an application the size of Horizon without it would be madness :) angular-route Provides structure to the application through URL routing. angular-cookies Provides management of browser cookies in a way that integrates well with angular. angular-sanitize Allows direct embedding of HTML into angular templates, with sanitization. json3 Compatibility for older browsers so JSON works. es5-shim Compatibility for older browsers so Javascript (ECMAScript 5) works. angular-smart-table Table management (population, sorting, filtering, pagination, etc) angular-local-storage Browser local storage with cookie fallback, integrated with angular mechanisms. angular-bootstrap Extensions to angular that leverage bootstrap (modal popups, tabbed displays, ...) font-awesome Additional glyphs to use in the user interface (warning symbol, info symbol, ...) boot Bootstrap for CSS styling (this is the dependency that brings in jquery and requirejs) underscore Javascript utility library providing a ton of features Javascript lacks but Python programmers expect. ng-websocket Angular-friendly interface to using websockets angular-translate Support for localization in angular using message catalogs generated by gettext/transifex. angular-mocks Mocking support for unit testing angular code angular-scenario More support for angular unit tests Additionally, angboard vendors term.js because it was very poorly packaged in the bower ecosystem. +1 for xstatic there I guess :) So those are the components we needed to create the prototype in a few weeks. Not using them would have added months (or possibly years) to the development time. Creating an application of the scale of Horizon without leveraging all that existing work would be like developing OpenStack while barring all use of Python 3rd-party packages. Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
