Re: [openstack-dev] [TripleO] fix poor tarball support in source-repositories
Excerpts from Clint Byrum's message of 2014-08-16 14:33:20 +: > That is a separate bug, but I think the answer to that is to use rsync > instead of mv and globs. So this: > > mv $tmp/./* $destdir > > becomes this: > > rsync --remove-source-files $tmp/. $destdir > +1 on this approach. It's straightforward to explain and fairly easy to reason about. I saw mention of checking the in-tarball contents to determine what to do - I would prefer us to be explicit about the behavior to take rather than depend on some internal structure of an external package. While we have to depend on the structure of the tarball to a certain extent, IMO we should minimize this if possible. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] fix poor tarball support in source-repositories
On 16 August 2014 03:11, Brownell, Jonathan C (Corvallis) wrote: > The current DIB element support for downloading tarballs via > "source-repository" allows an entry in the following form: > > tar > > Today, this feature is currently used only by the mysql DIB element. You can > see how it's used here: > https://github.com/openstack/tripleo-image-elements/blob/master/elements/mysql/source-repository-mysql > > However, the underlying diskimage-builder implementation of tarball handling > is rather odd and inflexible. After downloading the file (or retrieving from > cache) and unpacking into a tmp directory, it performs: > > mv $tmp/*/* $targetdir > > This does work as long as the tarball follows a structure where all its > files/directories are contained within a single directory, but it fails if > the tarball contains no subdirectories. (Even worse is when it contains some > files and some subdirectories, in which case the files are lost and the > contents of all subdirs get lumped together in the output folder.) > > Since this tarball support is only used today by the mysql DIB element, I > would love to fix this in both diskimage-builder and tripleo-image-element by > changing to simply: > > mv $tmp/* $targetdir > > And then manually tweaking the directory structure of $targetdir from a new > install.d script in the mysql element to restore the desired layout. As Derekh says, this would tie tarball use to the element code, which is overly tight coupling. > However, it's important to note that this will break backwards compatibility > if tarball support is used in its current fashion by users with private DIB > elements. And we're committing to not doing that. > Personally, I consider the current behavior so egregious that it really needs > to be fixed across the board rather than preserving backwards compatibility. I presume the current behaviour is breaking something for you? > Do others agree? If not, do you have suggestions as to how to improve this > mechanism cleanly without sacrificing backwards compatibility? We could make a heuristic (there is a patch up already to do that for dib) that looks at the unpacked content to decide what to do. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] fix poor tarball support in source-repositories
On 15/08/14 20:57, Clint Byrum wrote: > Excerpts from Brownell, Jonathan C (Corvallis)'s message of 2014-08-15 > 08:11:18 -0700: >> The current DIB element support for downloading tarballs via >> "source-repository" allows an entry in the following form: >> >> tar >> >> Today, this feature is currently used only by the mysql DIB element. You can >> see how it's used here: >> https://github.com/openstack/tripleo-image-elements/blob/master/elements/mysql/source-repository-mysql >> >> However, the underlying diskimage-builder implementation of tarball handling >> is rather odd and inflexible. After downloading the file (or retrieving from >> cache) and unpacking into a tmp directory, it performs: >> >> mv $tmp/*/* $targetdir iirc the reason this was done was so that tarball support could be used as a replacement for git if needed, eg. if I wanted to install nova from tarball instead of git I could set DIB_REPOTYPE_nova=tar DIB_REPOLOCATION_nova=http://tarballs.openstack.org/nova/nova-2014.1.2.tar.gz The end result should be the same as if it was git clone so that the nova element could continue to work as is. >> >> This does work as long as the tarball follows a structure where all its >> files/directories are contained within a single directory, but it fails if >> the tarball contains no subdirectories. (Even worse is when it contains some >> files and some subdirectories, in which case the files are lost and the >> contents of all subdirs get lumped together in the output folder.) >> >> Since this tarball support is only used today by the mysql DIB element, I >> would love to fix this in both diskimage-builder and tripleo-image-element >> by changing to simply: >> >> mv $tmp/* $targetdir >> >> And then manually tweaking the directory structure of $targetdir from a new >> install.d script in the mysql element to restore the desired layout. >> >> However, it's important to note that this will break backwards compatibility >> if tarball support is used in its current fashion by users with private DIB >> elements. >> >> Personally, I consider the current behavior so egregious that it really >> needs to be fixed across the board rather than preserving backwards >> compatibility. >> >> Do others agree? If not, do you have suggestions as to how to improve this >> mechanism cleanly without sacrificing backwards compatibility? >> > > How about we make a glob to use, so like this: > > mysql tar /usr/local/mysql http://someplace/mysql.tar.gz mysql-5.* > > That would result in > > mv $tmp/mysql-5.*/* $targetdir > > And then we would warn that assuming the glob will be '*' is deprecated, > to be changed in a later release. > > Users who want your proposed behavior would use . until the default > changes. That would result in > > mv $tmp/./* $targetdir this works for me, once the default changes users who use the example above(If there are any) would need to include DIB_REPOREF_nova=nova-2014.1.2 > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] fix poor tarball support in source-repositories
Excerpts from Jyoti Ranjan's message of 2014-08-16 00:57:52 -0700: > We will have to be little bit cautious in using glob because of its > inherent usage pattern. For e.g. the file starting with . will not get > matched. > That is a separate bug, but I think the answer to that is to use rsync instead of mv and globs. So this: mv $tmp/./* $destdir becomes this: rsync --remove-source-files $tmp/. $destdir ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] fix poor tarball support in source-repositories
We will have to be little bit cautious in using glob because of its inherent usage pattern. For e.g. the file starting with . will not get matched. $ touch .a .ab .abc .abcde .abcdef $ echo * * $ echo ? ? $ echo ?? ?? $ echo .? .. .a $ echo .?? .ab $ echo .* . .. .a .ab .abc .abcde .abcdef $ echo .??* .ab .abc .abcde .abcdef $ echo [.]* [.]* On Sat, Aug 16, 2014 at 4:43 AM, Brownell, Jonathan C (Corvallis) < brown...@hp.com> wrote: > Looks like a good solution to me. If there are no philosophical objections > to it, I'll prepare a patch next week to make this happen. > > -JB > > -Original Message- > From: Clint Byrum [mailto:cl...@fewbar.com] > Sent: Friday, August 15, 2014 12:58 PM > To: openstack-dev > Subject: Re: [openstack-dev] [TripleO] fix poor tarball support in > source-repositories > > Excerpts from Brownell, Jonathan C (Corvallis)'s message of 2014-08-15 > 08:11:18 -0700: > > The current DIB element support for downloading tarballs via > "source-repository" allows an entry in the following form: > > > > tar > > > > Today, this feature is currently used only by the mysql DIB element. You > can see how it's used here: > > https://github.com/openstack/tripleo-image-elements/blob/master/elemen > > ts/mysql/source-repository-mysql > > > > However, the underlying diskimage-builder implementation of tarball > handling is rather odd and inflexible. After downloading the file (or > retrieving from cache) and unpacking into a tmp directory, it performs: > > > > mv $tmp/*/* $targetdir > > > > This does work as long as the tarball follows a structure where all > > its files/directories are contained within a single directory, but it > > fails if the tarball contains no subdirectories. (Even worse is when > > it contains some files and some subdirectories, in which case the > > files are lost and the contents of all subdirs get lumped together in > > the output folder.) > > > > Since this tarball support is only used today by the mysql DIB element, > I would love to fix this in both diskimage-builder and > tripleo-image-element by changing to simply: > > > > mv $tmp/* $targetdir > > > > And then manually tweaking the directory structure of $targetdir from a > new install.d script in the mysql element to restore the desired layout. > > > > However, it's important to note that this will break backwards > compatibility if tarball support is used in its current fashion by users > with private DIB elements. > > > > Personally, I consider the current behavior so egregious that it really > needs to be fixed across the board rather than preserving backwards > compatibility. > > > > Do others agree? If not, do you have suggestions as to how to improve > this mechanism cleanly without sacrificing backwards compatibility? > > > > How about we make a glob to use, so like this: > > mysql tar /usr/local/mysql http://someplace/mysql.tar.gz mysql-5.* > > That would result in > > mv $tmp/mysql-5.*/* $targetdir > > And then we would warn that assuming the glob will be '*' is deprecated, > to be changed in a later release. > > Users who want your proposed behavior would use . until the default > changes. That would result in > > mv $tmp/./* $targetdir > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] fix poor tarball support in source-repositories
Looks like a good solution to me. If there are no philosophical objections to it, I'll prepare a patch next week to make this happen. -JB -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: Friday, August 15, 2014 12:58 PM To: openstack-dev Subject: Re: [openstack-dev] [TripleO] fix poor tarball support in source-repositories Excerpts from Brownell, Jonathan C (Corvallis)'s message of 2014-08-15 08:11:18 -0700: > The current DIB element support for downloading tarballs via > "source-repository" allows an entry in the following form: > > tar > > Today, this feature is currently used only by the mysql DIB element. You can > see how it's used here: > https://github.com/openstack/tripleo-image-elements/blob/master/elemen > ts/mysql/source-repository-mysql > > However, the underlying diskimage-builder implementation of tarball handling > is rather odd and inflexible. After downloading the file (or retrieving from > cache) and unpacking into a tmp directory, it performs: > > mv $tmp/*/* $targetdir > > This does work as long as the tarball follows a structure where all > its files/directories are contained within a single directory, but it > fails if the tarball contains no subdirectories. (Even worse is when > it contains some files and some subdirectories, in which case the > files are lost and the contents of all subdirs get lumped together in > the output folder.) > > Since this tarball support is only used today by the mysql DIB element, I > would love to fix this in both diskimage-builder and tripleo-image-element by > changing to simply: > > mv $tmp/* $targetdir > > And then manually tweaking the directory structure of $targetdir from a new > install.d script in the mysql element to restore the desired layout. > > However, it's important to note that this will break backwards compatibility > if tarball support is used in its current fashion by users with private DIB > elements. > > Personally, I consider the current behavior so egregious that it really needs > to be fixed across the board rather than preserving backwards compatibility. > > Do others agree? If not, do you have suggestions as to how to improve this > mechanism cleanly without sacrificing backwards compatibility? > How about we make a glob to use, so like this: mysql tar /usr/local/mysql http://someplace/mysql.tar.gz mysql-5.* That would result in mv $tmp/mysql-5.*/* $targetdir And then we would warn that assuming the glob will be '*' is deprecated, to be changed in a later release. Users who want your proposed behavior would use . until the default changes. That would result in mv $tmp/./* $targetdir ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] fix poor tarball support in source-repositories
Excerpts from Brownell, Jonathan C (Corvallis)'s message of 2014-08-15 08:11:18 -0700: > The current DIB element support for downloading tarballs via > "source-repository" allows an entry in the following form: > > tar > > Today, this feature is currently used only by the mysql DIB element. You can > see how it's used here: > https://github.com/openstack/tripleo-image-elements/blob/master/elements/mysql/source-repository-mysql > > However, the underlying diskimage-builder implementation of tarball handling > is rather odd and inflexible. After downloading the file (or retrieving from > cache) and unpacking into a tmp directory, it performs: > > mv $tmp/*/* $targetdir > > This does work as long as the tarball follows a structure where all its > files/directories are contained within a single directory, but it fails if > the tarball contains no subdirectories. (Even worse is when it contains some > files and some subdirectories, in which case the files are lost and the > contents of all subdirs get lumped together in the output folder.) > > Since this tarball support is only used today by the mysql DIB element, I > would love to fix this in both diskimage-builder and tripleo-image-element by > changing to simply: > > mv $tmp/* $targetdir > > And then manually tweaking the directory structure of $targetdir from a new > install.d script in the mysql element to restore the desired layout. > > However, it's important to note that this will break backwards compatibility > if tarball support is used in its current fashion by users with private DIB > elements. > > Personally, I consider the current behavior so egregious that it really needs > to be fixed across the board rather than preserving backwards compatibility. > > Do others agree? If not, do you have suggestions as to how to improve this > mechanism cleanly without sacrificing backwards compatibility? > How about we make a glob to use, so like this: mysql tar /usr/local/mysql http://someplace/mysql.tar.gz mysql-5.* That would result in mv $tmp/mysql-5.*/* $targetdir And then we would warn that assuming the glob will be '*' is deprecated, to be changed in a later release. Users who want your proposed behavior would use . until the default changes. That would result in mv $tmp/./* $targetdir ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [TripleO] fix poor tarball support in source-repositories
The current DIB element support for downloading tarballs via "source-repository" allows an entry in the following form: tar Today, this feature is currently used only by the mysql DIB element. You can see how it's used here: https://github.com/openstack/tripleo-image-elements/blob/master/elements/mysql/source-repository-mysql However, the underlying diskimage-builder implementation of tarball handling is rather odd and inflexible. After downloading the file (or retrieving from cache) and unpacking into a tmp directory, it performs: mv $tmp/*/* $targetdir This does work as long as the tarball follows a structure where all its files/directories are contained within a single directory, but it fails if the tarball contains no subdirectories. (Even worse is when it contains some files and some subdirectories, in which case the files are lost and the contents of all subdirs get lumped together in the output folder.) Since this tarball support is only used today by the mysql DIB element, I would love to fix this in both diskimage-builder and tripleo-image-element by changing to simply: mv $tmp/* $targetdir And then manually tweaking the directory structure of $targetdir from a new install.d script in the mysql element to restore the desired layout. However, it's important to note that this will break backwards compatibility if tarball support is used in its current fashion by users with private DIB elements. Personally, I consider the current behavior so egregious that it really needs to be fixed across the board rather than preserving backwards compatibility. Do others agree? If not, do you have suggestions as to how to improve this mechanism cleanly without sacrificing backwards compatibility? Thanks, Jonathan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
