Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
Hi Rob, That is slightly different: from logical point of view that are different schemas indeed, however they all map to the same DB schema, so we do not have any issues with upgrades. There are some limitations on the schema modifications as well, and being able to work with multiple versions of the same artifact type is supported from the beginning and works quite well. -- Regards, Alexander Tivelkov On Mon, Feb 16, 2015 at 9:50 PM, Robert Collins robe...@robertcollins.net wrote: On 17 February 2015 at 03:31, Alexander Tivelkov ativel...@mirantis.com wrote: Hi Client, Thanks for your input. We actually support the scenarios you speak about, yet in a slightly different way. The authors of the Artifact Type (the plugin developers) may define their own custom field (or set of fields) to store their sequence information or any other type-specific version-related metadata. So, they may use generic version field (which is defined in the base artifact type) to store their numeric version - and use their type-specific field for local client-side processing. That sounds scarily like what Neutron did, leading to a different schema for every configuration. The reason Clint brought up Debian version numbers is that to sort them in a database you need a custom field type - e.g. http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/database/schema/launchpad-2209-00-0.sql#L25 . And thats quite a burden :) We've had fairly poor results with the Neutron variation in schemas, as it tightly couples things, making upgrades that change plugins super tricky, as well as making it very hard to concurrently support multiple plugins. I hope you don't mean you're doing the same thing :) -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
On 17 February 2015 at 03:31, Alexander Tivelkov ativel...@mirantis.com wrote: Hi Client, Thanks for your input. We actually support the scenarios you speak about, yet in a slightly different way. The authors of the Artifact Type (the plugin developers) may define their own custom field (or set of fields) to store their sequence information or any other type-specific version-related metadata. So, they may use generic version field (which is defined in the base artifact type) to store their numeric version - and use their type-specific field for local client-side processing. That sounds scarily like what Neutron did, leading to a different schema for every configuration. The reason Clint brought up Debian version numbers is that to sort them in a database you need a custom field type - e.g. http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/database/schema/launchpad-2209-00-0.sql#L25 . And thats quite a burden :) We've had fairly poor results with the Neutron variation in schemas, as it tightly couples things, making upgrades that change plugins super tricky, as well as making it very hard to concurrently support multiple plugins. I hope you don't mean you're doing the same thing :) -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
Hi Ian, On Tue, Feb 10, 2015 at 11:17 PM, Ian Cordasco ian.corda...@rackspace.com wrote: I think the fundamental disconnect is that not every column in a database needs offer sorting to the user. Imposing that restriction here causes a cascade of further restrictions that will fundamentally cause this to be unusable by a large number of people. I didn't say that every column needs to offer sorting. However, ability to differentiate between different artifacts and different versions of the same artifact, as well as ability to sort and filter these different versions according to some criteria is one of the core features we were looking for when designing the whole artifact proposal. In fact, the initial request for this feature was not even made by me: I was asked about versions at the first design summit where we initially suggested artifacts (Icehouse mid-cycle) and in subsequent email and IRC discussions. The first design proposal which was presented in Atlanta already included this concept and so it was approved there, so this feature was always considered as a natural and important concept. I don't understand why it causes so much confusion now, when the implementation is almost complete. Except for the fact that versions do typically mean more than the values SemVer attaches to them. SemVer is further incompatible with any versioning scheme using epochs and is so relatively recent compared to versioning practices as a whole that I don’t see how we can justify restricting what should be a very generic system to something so specific to recent history and favored almost entirely by *developers*. I believe any person in software industry may invent their own versioning scheme - and most of them will be incompatible with each other. If you attempt to build something which is compatible with all of them at once, you will eventually end up having plain strings, without any semantic and precedence rules. This definitely does not satisfy our initial goal (see above). Instead, I suggest to choose the scheme which is strict, unambiguous and satisfies our initial goal, but has maximum possible adoption among the developers, so most of the adopters will be able to use it. Semver seems to be the best candidate here, but any other proposals are welcome as well. Meanwhile this does not prevent adopters from having their own schemas if they want it: Artifact Type developers may add their own type-specific string metadata field, put some regexp of code-based constraints on it - and use it to store their own version. Yet they will not be able to utilize the version-related features which are built-in into the Glance. Because up until now the only use case you’ve been referencing is CD software. There is some disconnect here: I believe the message which started this thread was the first time where I mentioned Artifact Repository in the context of Continuous Delivery solutions; and I was saying about CD system built on top of it, not about the Artifact Repostiory being a CD-system on its own. All other use-cases and scenarios did not mention CDs at all: Artifact Repostiory is definitely a generic catalog, not a CD solution. -- Regards, Alexander Tivelkov __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
Donald, Thanks for your comments, really useful! I think I need to clarify a bit: I am not speaking about the actual semantic: placing the meaning into the actual values is still up to the end-users (or the developers of Artifact Types, if they build some custom logic which processes version info somehow). So, this thread is really about preferred syntax scheme - and the rules to determine precedence. I understand that pep440 has richer syntax with more capabilities (epochs, unlimited number of version segments, development releases etc). My only concern is that being a python-only standard it is less generic (in term of adoption) that the syntax of semver. The same goes to Monty's pbr semver: it is openstack-only and thus may be confusing. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 11:32 PM, Donald Stufft don...@stufft.io wrote: On Feb 10, 2015, at 3:17 PM, Ian Cordasco ian.corda...@rackspace.com wrote: And of course, the chosen solution should be mappable to database, so we may do sorting and filtering on the DB-side. So, having it as a simple string and letting the user to decide what it means is not an option. Except for the fact that versions do typically mean more than the values SemVer attaches to them. SemVer is further incompatible with any versioning scheme using epochs and is so relatively recent compared to versioning practices as a whole that I don’t see how we can justify restricting what should be a very generic system to something so specific to recent history and favored almost entirely by *developers*. Semver vs PEP 440 is largely a syntax question since PEP 440 purposely does not have much of an opinion on how something like 2.0.0 and 2.1.0 are related other than for sorting. We do have operators in PEP 440 that support treating these versions in a semver like way, and some that support treating them in other ways. The primary purpose of PEP 440 was to define a standard way to parse and sort and specify versions across several hundred thouse versions that currently exist in PyPI. This means that it is more complicated to implement but it is much more powerful than semver eve could be. One example, as Ian mentioned is the lack of the ability to do an Epoch, another example is that PEP 440 has explicit support for someone taking version 1.0 adding some unofficial patches to it, and then releasing that in their own distribution channels. The primary purpose of Semver was to be extremely opinionated in what meaning you place on the *content* of the version parts and the syntax is really a secondary concern which exists just to make it easier to parse. This means that if you know ahead of time that something is Semver you can guess a lot more information about the relationship of two versions. It was our intention that PEP 440 would (is?) aimed primarily at people implementing tools that work with versions, and the additional PEPs or other documentations would be written on top of PEP 440 to add opinions on what a version looks like within the framework that PEP 440 sets up. A great example is the pbr semver document that Monty linked. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
Hi Client, Thanks for your input. We actually support the scenarios you speak about, yet in a slightly different way. The authors of the Artifact Type (the plugin developers) may define their own custom field (or set of fields) to store their sequence information or any other type-specific version-related metadata. So, they may use generic version field (which is defined in the base artifact type) to store their numeric version - and use their type-specific field for local client-side processing. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 11:37 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Alexander Tivelkov's message of 2015-02-10 07:28:55 -0800: Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision, the Artifact Repository should be as generic as possible in terms of potential adoption. The artifacts were never supposed to be python packages only, and even the projects which will create and use these artifacts are not mandatory limited to be pythonic, the developers of that projects may not be python developers! So, I'd really wanted to avoid any python-specific notations, such as PEP-440 for artifacts. I've put this vision into a spec [3] which also contains a proposal on how to convert the semver-compatible version strings into the comparable values which may be mapped to database types, so a database table may be queried, ordered and filtered by the object version. So, we need some feedback on this topic. Would you prefer artifacts to be versioned with SemVer or with PEP-440 notation? Are you interested in having some generic utility which will map versions (in either format) to database columns? If so, which version format would you prefer? We are on a tight schedule here, as we want to begin landing artifact-related code soon. So, I would appreciate your feedback during this week: here in the ML or in the comments to [3] review. Hi. This is really interesting work and I'm glad Glance is growing into an artifact catalog as I think it will assist cloud users and UI development at the same time. It seems to me that there are really only two reasons to care about the content of the versions: sorting, and filtering. You want to make sure if people upload artifacts named myapp like this: myapp:1.0 myapp:2.0 myapp:1.1 That when they say show me the newest myapp they get 2.0, not 1.1. And if they say show me the newest myapp in the 1.x series they get 1.1. I am a little worried this is not something that can or should be made generic in a micro service. Here's a thought: You could just have the version, series, and sequence, and let users manage the sequencing themselves on the client side. This way if users want to use the _extremely_ difficult to program for Debian packaging version, you don't have to figure out how to make 1.0~special less than 1.0 and more than 0.9. To start with, you can have a default strategy of a single series, and max(sequence)+1000 if unspecified. Then teach the clients the various semvers/pep440's/etc. etc. and let them choose their own sequencing and series strategy. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not
[openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision, the Artifact Repository should be as generic as possible in terms of potential adoption. The artifacts were never supposed to be python packages only, and even the projects which will create and use these artifacts are not mandatory limited to be pythonic, the developers of that projects may not be python developers! So, I'd really wanted to avoid any python-specific notations, such as PEP-440 for artifacts. I've put this vision into a spec [3] which also contains a proposal on how to convert the semver-compatible version strings into the comparable values which may be mapped to database types, so a database table may be queried, ordered and filtered by the object version. So, we need some feedback on this topic. Would you prefer artifacts to be versioned with SemVer or with PEP-440 notation? Are you interested in having some generic utility which will map versions (in either format) to database columns? If so, which version format would you prefer? We are on a tight schedule here, as we want to begin landing artifact-related code soon. So, I would appreciate your feedback during this week: here in the ML or in the comments to [3] review. Thanks! [1] www.semver.org [2] www.python.org/dev/peps/pep-0440/ [3] https://review.openstack.org/#/c/139595/ -- Regards, Alexander Tivelkov __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
On 02/10/2015 10:28 AM, Alexander Tivelkov wrote: Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision, the Artifact Repository should be as generic as possible in terms of potential adoption. The artifacts were never supposed to be python packages only, and even the projects which will create and use these artifacts are not mandatory limited to be pythonic, the developers of that projects may not be python developers! So, I'd really wanted to avoid any python-specific notations, such as PEP-440 for artifacts. I've put this vision into a spec [3] which also contains a proposal on how to convert the semver-compatible version strings into the comparable values which may be mapped to database types, so a database table may be queried, ordered and filtered by the object version. So, we need some feedback on this topic. Would you prefer artifacts to be versioned with SemVer or with PEP-440 notation? Are you interested in having some generic utility which will map versions (in either format) to database columns? If so, which version format would you prefer? We are on a tight schedule here, as we want to begin landing artifact-related code soon. So, I would appreciate your feedback during this week: here in the ML or in the comments to [3] review. Because you should have more things to look at: http://docs.openstack.org/developer/pbr/semver.html We've already done some work to try to reconcile the world of semver with the world of PEP440 over in PBR land. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
Hi Ian, Automatic version generation is not the only and not the primary reason for the version concept. In fact, the implementation which is planned to land in this cycle does not contain this feature at all: currently we also leave the version assignment up to uploader (version is a regular immutable generic artifact property). Auto-increments as part of clone-and-modify scenarios are postponed for the next cycle. However, even now we do need to have some sorting order - so, we need rules to determine precedence. That's the reason for having some notation defined: if we leave the notation up to the end-user we won't be able to compare artifacts having versions in different notations. And we even can't leave it up to the Artifact Type developer, since this is a generic property, thus common for all the artifact types. And of course, the chosen solution should be mappable to database, so we may do sorting and filtering on the DB-side. So, having it as a simple string and letting the user to decide what it means is not an option. Speaking about Artifactory - that's entirely different thing. It is indeed a continuous delivery solution, composed around build machines, deployment solutions and CI systems. That's definitely not what Glance Artifact Repository is. Even the concepts of Artifact are entirely different. So, while Artifact Repository may be used to build some CD solutions on top of it (or to be integrated with the existing ones) it is not a storage solution for build outputs and thus I can barely see how we may compare them. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 8:15 PM, Ian Cordasco ian.corda...@rackspace.com wrote: On 2/10/15, 10:35, Alexander Tivelkov ativel...@mirantis.com wrote: Thanks Monty! Yup, probably I've missed that. I was looking at pbr and its version implementation, but didn't realize that this is actually a fusion of semver and pep440. So, we have this as an extra alternative to choose from. It would be an obvious choice if we were just looking for some common solution to version objects within openstack. However, I am a bit concerned about applying it to Artifact Repository. As I wrote before, we are trying to make the Repository to be language- and platform-agnostic tool for other developers, including the ones originating from non-python and non-openstack worlds. Having a versioning notation which is non-standard for everybody but openstack developers does not look like a good idea to me. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com wrote: On 02/10/2015 10:28 AM, Alexander Tivelkov wrote: Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision, the Artifact Repository should be as generic as possible in terms of potential adoption. The artifacts were never supposed to be python packages only, and even the projects which will create and use these artifacts are not mandatory limited to be pythonic, the developers of that projects may not be python developers! So, I'd really wanted to avoid any python-specific notations, such as PEP-440 for artifacts. I've put this vision into a spec [3] which also contains a proposal on how to convert the semver-compatible version strings into the comparable values which may be mapped to database types, so a database table may be queried, ordered and filtered by the object version. So, we need some feedback on this topic. Would you prefer artifacts to be versioned with SemVer or with PEP-440 notation? Are
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
Thanks Monty! Yup, probably I've missed that. I was looking at pbr and its version implementation, but didn't realize that this is actually a fusion of semver and pep440. So, we have this as an extra alternative to choose from. It would be an obvious choice if we were just looking for some common solution to version objects within openstack. However, I am a bit concerned about applying it to Artifact Repository. As I wrote before, we are trying to make the Repository to be language- and platform-agnostic tool for other developers, including the ones originating from non-python and non-openstack worlds. Having a versioning notation which is non-standard for everybody but openstack developers does not look like a good idea to me. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com wrote: On 02/10/2015 10:28 AM, Alexander Tivelkov wrote: Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision, the Artifact Repository should be as generic as possible in terms of potential adoption. The artifacts were never supposed to be python packages only, and even the projects which will create and use these artifacts are not mandatory limited to be pythonic, the developers of that projects may not be python developers! So, I'd really wanted to avoid any python-specific notations, such as PEP-440 for artifacts. I've put this vision into a spec [3] which also contains a proposal on how to convert the semver-compatible version strings into the comparable values which may be mapped to database types, so a database table may be queried, ordered and filtered by the object version. So, we need some feedback on this topic. Would you prefer artifacts to be versioned with SemVer or with PEP-440 notation? Are you interested in having some generic utility which will map versions (in either format) to database columns? If so, which version format would you prefer? We are on a tight schedule here, as we want to begin landing artifact-related code soon. So, I would appreciate your feedback during this week: here in the ML or in the comments to [3] review. Because you should have more things to look at: http://docs.openstack.org/developer/pbr/semver.html We've already done some work to try to reconcile the world of semver with the world of PEP440 over in PBR land. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
On 2/10/15, 10:35, Alexander Tivelkov ativel...@mirantis.com wrote: Thanks Monty! Yup, probably I've missed that. I was looking at pbr and its version implementation, but didn't realize that this is actually a fusion of semver and pep440. So, we have this as an extra alternative to choose from. It would be an obvious choice if we were just looking for some common solution to version objects within openstack. However, I am a bit concerned about applying it to Artifact Repository. As I wrote before, we are trying to make the Repository to be language- and platform-agnostic tool for other developers, including the ones originating from non-python and non-openstack worlds. Having a versioning notation which is non-standard for everybody but openstack developers does not look like a good idea to me. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com wrote: On 02/10/2015 10:28 AM, Alexander Tivelkov wrote: Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision, the Artifact Repository should be as generic as possible in terms of potential adoption. The artifacts were never supposed to be python packages only, and even the projects which will create and use these artifacts are not mandatory limited to be pythonic, the developers of that projects may not be python developers! So, I'd really wanted to avoid any python-specific notations, such as PEP-440 for artifacts. I've put this vision into a spec [3] which also contains a proposal on how to convert the semver-compatible version strings into the comparable values which may be mapped to database types, so a database table may be queried, ordered and filtered by the object version. So, we need some feedback on this topic. Would you prefer artifacts to be versioned with SemVer or with PEP-440 notation? Are you interested in having some generic utility which will map versions (in either format) to database columns? If so, which version format would you prefer? We are on a tight schedule here, as we want to begin landing artifact-related code soon. So, I would appreciate your feedback during this week: here in the ML or in the comments to [3] review. Because you should have more things to look at: http://docs.openstack.org/developer/pbr/semver.html We've already done some work to try to reconcile the world of semver with the world of PEP440 over in PBR land. _ _ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev So Semantic Versioning, as I’ve already mentioned in the past, isn’t really a de facto standard in any language community but it is a language agnostic proposal. That said, just because it’s language agnostic does not mean it won’t conflict with other language’s versioning semantics. Since we’re effectively reinventing an existing open source solution here, I think we should look to how Artifactory [1] handles this. I haven’t used artifactory very much but a cursory look makes it apparent that it is strongly decoupling the logic of
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
On Feb 10, 2015, at 3:17 PM, Ian Cordasco ian.corda...@rackspace.com wrote: And of course, the chosen solution should be mappable to database, so we may do sorting and filtering on the DB-side. So, having it as a simple string and letting the user to decide what it means is not an option. Except for the fact that versions do typically mean more than the values SemVer attaches to them. SemVer is further incompatible with any versioning scheme using epochs and is so relatively recent compared to versioning practices as a whole that I don’t see how we can justify restricting what should be a very generic system to something so specific to recent history and favored almost entirely by *developers*. Semver vs PEP 440 is largely a syntax question since PEP 440 purposely does not have much of an opinion on how something like 2.0.0 and 2.1.0 are related other than for sorting. We do have operators in PEP 440 that support treating these versions in a semver like way, and some that support treating them in other ways. The primary purpose of PEP 440 was to define a standard way to parse and sort and specify versions across several hundred thouse versions that currently exist in PyPI. This means that it is more complicated to implement but it is much more powerful than semver eve could be. One example, as Ian mentioned is the lack of the ability to do an Epoch, another example is that PEP 440 has explicit support for someone taking version 1.0 adding some unofficial patches to it, and then releasing that in their own distribution channels. The primary purpose of Semver was to be extremely opinionated in what meaning you place on the *content* of the version parts and the syntax is really a secondary concern which exists just to make it easier to parse. This means that if you know ahead of time that something is Semver you can guess a lot more information about the relationship of two versions. It was our intention that PEP 440 would (is?) aimed primarily at people implementing tools that work with versions, and the additional PEPs or other documentations would be written on top of PEP 440 to add opinions on what a version looks like within the framework that PEP 440 sets up. A great example is the pbr semver document that Monty linked. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
I agree with Alexander on this. We should certainly learn what we can from existing software. That said, the Solum team really wants this feature in Glance so we can leverage that instead of having our own repository for Heat templates we generate when building apps. We want to keep our requirements list small, and Glance is already there. Adrian On Feb 10, 2015, at 10:01 AM, Alexander Tivelkov ativel...@mirantis.com wrote: Hi Ian, Automatic version generation is not the only and not the primary reason for the version concept. In fact, the implementation which is planned to land in this cycle does not contain this feature at all: currently we also leave the version assignment up to uploader (version is a regular immutable generic artifact property). Auto-increments as part of clone-and-modify scenarios are postponed for the next cycle. However, even now we do need to have some sorting order - so, we need rules to determine precedence. That's the reason for having some notation defined: if we leave the notation up to the end-user we won't be able to compare artifacts having versions in different notations. And we even can't leave it up to the Artifact Type developer, since this is a generic property, thus common for all the artifact types. And of course, the chosen solution should be mappable to database, so we may do sorting and filtering on the DB-side. So, having it as a simple string and letting the user to decide what it means is not an option. Speaking about Artifactory - that's entirely different thing. It is indeed a continuous delivery solution, composed around build machines, deployment solutions and CI systems. That's definitely not what Glance Artifact Repository is. Even the concepts of Artifact are entirely different. So, while Artifact Repository may be used to build some CD solutions on top of it (or to be integrated with the existing ones) it is not a storage solution for build outputs and thus I can barely see how we may compare them. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 8:15 PM, Ian Cordasco ian.corda...@rackspace.com wrote: On 2/10/15, 10:35, Alexander Tivelkov ativel...@mirantis.com wrote: Thanks Monty! Yup, probably I've missed that. I was looking at pbr and its version implementation, but didn't realize that this is actually a fusion of semver and pep440. So, we have this as an extra alternative to choose from. It would be an obvious choice if we were just looking for some common solution to version objects within openstack. However, I am a bit concerned about applying it to Artifact Repository. As I wrote before, we are trying to make the Repository to be language- and platform-agnostic tool for other developers, including the ones originating from non-python and non-openstack worlds. Having a versioning notation which is non-standard for everybody but openstack developers does not look like a good idea to me. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com wrote: On 02/10/2015 10:28 AM, Alexander Tivelkov wrote: Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision, the Artifact Repository should be as generic as possible in terms of potential adoption. The artifacts were never supposed to be python packages only, and even the projects which will create and use these artifacts are not mandatory limited to be pythonic, the developers of that projects may not be python
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
On 2/10/15, 12:01, Alexander Tivelkov ativel...@mirantis.com wrote: Hi Ian, Automatic version generation is not the only and not the primary reason for the version concept. In fact, the implementation which is planned to land in this cycle does not contain this feature at all: currently we also leave the version assignment up to uploader (version is a regular immutable generic artifact property). Auto-increments as part of clone-and-modify scenarios are postponed for the next cycle. However, even now we do need to have some sorting order - so, we need rules to determine precedence. That's the reason for having some notation defined: if we leave the notation up to the end-user we won't be able to compare artifacts having versions in different notations. And we even can't leave it up to the Artifact Type developer, since this is a generic property, thus common for all the artifact types. I think the fundamental disconnect is that not every column in a database needs offer sorting to the user. Imposing that restriction here causes a cascade of further restrictions that will fundamentally cause this to be unusable by a large number of people. And of course, the chosen solution should be mappable to database, so we may do sorting and filtering on the DB-side. So, having it as a simple string and letting the user to decide what it means is not an option. Except for the fact that versions do typically mean more than the values SemVer attaches to them. SemVer is further incompatible with any versioning scheme using epochs and is so relatively recent compared to versioning practices as a whole that I don’t see how we can justify restricting what should be a very generic system to something so specific to recent history and favored almost entirely by *developers*. Speaking about Artifactory - that's entirely different thing. It is indeed a continuous delivery solution, composed around build machines, deployment solutions and CI systems. That's definitely not what Glance Artifact Repository is. Even the concepts of Artifact are entirely different. So, while Artifact Repository may be used to build some CD solutions on top of it (or to be integrated with the existing ones) it is not a storage solution for build outputs and thus I can barely see how we may compare them. Because up until now the only use case you’ve been referencing is CD software. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 8:15 PM, Ian Cordasco ian.corda...@rackspace.com wrote: On 2/10/15, 10:35, Alexander Tivelkov ativel...@mirantis.com wrote: Thanks Monty! Yup, probably I've missed that. I was looking at pbr and its version implementation, but didn't realize that this is actually a fusion of semver and pep440. So, we have this as an extra alternative to choose from. It would be an obvious choice if we were just looking for some common solution to version objects within openstack. However, I am a bit concerned about applying it to Artifact Repository. As I wrote before, we are trying to make the Repository to be language- and platform-agnostic tool for other developers, including the ones originating from non-python and non-openstack worlds. Having a versioning notation which is non-standard for everybody but openstack developers does not look like a good idea to me. -- Regards, Alexander Tivelkov On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com wrote: On 02/10/2015 10:28 AM, Alexander Tivelkov wrote: Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision,
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
On 2/10/15, 13:55, Jay Pipes jaypi...@gmail.com wrote: On 02/10/2015 12:15 PM, Ian Cordasco wrote: So Semantic Versioning, as I’ve already mentioned in the past, isn’t really a de facto standard in any language community but it is a language agnostic proposal. That said, just because it’s language agnostic does not mean it won’t conflict with other language’s versioning semantics. Since we’re effectively reinventing an existing open source solution here, I think we should look to how Artifactory [1] handles this. Reinventing an existing open source solution is a bit off-the-mark IMO. Artifactory is more of a SaaS solution through bintray.com -- paying some lip-service to open source more than anything else. Except the base service is entirely free and open and thus can be deployed anywhere. And people looking to rebuild CD services in OpenStack should probably refer to existing implementations (of which I only pointed out the one I had already heard about). I haven’t used artifactory very much but a cursory look makes it apparent that it is strongly decoupling the logic of version management with artifact management (which this set of changes isn’t doing in Glance). Alex's spec is merely proposing to standardize on a single way of managing versioning. It isn't coupling artifact *management* with anything whatsoever. In fact, the idea of Glance being an artifact repository has nothing to do with the management of said artifacts -- which things like Heat, Murano or Solum handle in different ways. Except that Alex is proposing that this would include a number of things, some of which can and will be versioned in a way that will not work with the proposed solution. The idea behind the Glance artifact repository was to store a discoverable *schema* for various objects, along with the object metadata itself. I can see Nova and Cinder flavors, Glance, Docker or AppC image metadata, Cinder snapshot and differential metadata, Murano application manifests, Heat templates, and Solum deployment components all being described (schemas) and stored in the Glance artifact repository. What I have not heard from anyone is the idea to marry the management of the things that this artifact metadata with Glance itself. Right. That’s the other concern I have with Artifacts. Images /can/ be /eventually/ represented as artifacts, but for now we’re grafting what will functionally be an entirely separate project (for at least the next cycle, if not longer) onto Glance while that new project is unstable and at best experimental. The primary argument (as I understand it) for using SemVer with Artifacts in glance is to have a way to automatically have the service create the version number for the uploader. Artifactory (and it’s “Pro” version) seen to leave that to the build tooling. In other words, it is purely storage. It seems to sort versions alphabetically (which everyone can agree is not only suboptimal but wrong) but it also provides the user a way to alter how it performs sorting. I don't know where you get the idea that a service (Glance, you mean?) would automatically create the version number for some artifact. The spec talks only about the ability of the artifact registry to sort a set of artifacts by incrementing version number, and do so in a reasonably short time frame -- i.e. have a database column indexed for that purpose. Several discussions in IRC have happened around Artifacts and the SemVer spec in which this is a feature that Alex wants to add as a consequence of this. Is there a reason (beyond not being an OpenStack project) that Artifactory isn’t being considered by those pushing for Artifacts to provide a CD service? Because we're not looking to create a CD service? :) AFAIK, the artifact repository is a generic storage system for schemas and metadata, nothing more. Except the common explanation for why we /need/ Artifacts is the example of people wanting to set up a CD system and the discussions that have been held elsewhere point to this being so much more than just that in terms of responsibility of managing versions and other (specifically, build) artifacts. Best, -jay Judging by the support it seems to roughly have for other services (via a “Pro” version) it would appear to be able to suit any this need well with little work by the deployer who needs to serve more than one use case. [1] http://www.jfrog.com/open-source/#os-arti _ _ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440
Excerpts from Alexander Tivelkov's message of 2015-02-10 07:28:55 -0800: Hi folks, One of the key features that we are adding to Glance with the introduction of Artifacts is the ability to have multiple versions of the same object in the repository: this gives us the possibility to query for the latest version of something, keep track on the changes history, and build various continuous delivery solutions on top of Artifact Repository. We need to determine the format and rules we will use to define, increment and compare versions of artifacts in the repository. There are two alternatives we have to choose from, and we are seeking advice on this choice. First, there is Semantic Versioning specification, available at [1]. It is a very generic spec, widely used and adopted in many areas of software development. It is quite straightforward: 3 mandatory numeric components for version number, plus optional string labels for pre-release versions and build metadata. And then there is PEP-440 spec, which is a recommended approach to identifying versions and specifying dependencies when distributing Python. It is a pythonic way to set versions of python packages, including PIP version strings. Conceptually PEP-440 and Semantic Versioning are similar in purpose, but slightly different in syntax. Notably, the count of version number components and rules of version precedence resolution differ between PEP-440 and SemVer. Unfortunately, the two version string formats are not compatible, so we have to choose one or the other. According to my initial vision, the Artifact Repository should be as generic as possible in terms of potential adoption. The artifacts were never supposed to be python packages only, and even the projects which will create and use these artifacts are not mandatory limited to be pythonic, the developers of that projects may not be python developers! So, I'd really wanted to avoid any python-specific notations, such as PEP-440 for artifacts. I've put this vision into a spec [3] which also contains a proposal on how to convert the semver-compatible version strings into the comparable values which may be mapped to database types, so a database table may be queried, ordered and filtered by the object version. So, we need some feedback on this topic. Would you prefer artifacts to be versioned with SemVer or with PEP-440 notation? Are you interested in having some generic utility which will map versions (in either format) to database columns? If so, which version format would you prefer? We are on a tight schedule here, as we want to begin landing artifact-related code soon. So, I would appreciate your feedback during this week: here in the ML or in the comments to [3] review. Hi. This is really interesting work and I'm glad Glance is growing into an artifact catalog as I think it will assist cloud users and UI development at the same time. It seems to me that there are really only two reasons to care about the content of the versions: sorting, and filtering. You want to make sure if people upload artifacts named myapp like this: myapp:1.0 myapp:2.0 myapp:1.1 That when they say show me the newest myapp they get 2.0, not 1.1. And if they say show me the newest myapp in the 1.x series they get 1.1. I am a little worried this is not something that can or should be made generic in a micro service. Here's a thought: You could just have the version, series, and sequence, and let users manage the sequencing themselves on the client side. This way if users want to use the _extremely_ difficult to program for Debian packaging version, you don't have to figure out how to make 1.0~special less than 1.0 and more than 0.9. To start with, you can have a default strategy of a single series, and max(sequence)+1000 if unspecified. Then teach the clients the various semvers/pep440's/etc. etc. and let them choose their own sequencing and series strategy. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
