Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-17 Thread Alexander Tivelkov
Hi Rob,

That is slightly different: from logical point of view that are
different schemas indeed, however they all map to the same DB schema,
so we do not have any issues with upgrades. There are some limitations
on the schema modifications as well, and being able to work with
multiple versions of the same artifact type is supported from the
beginning and works quite well.
--
Regards,
Alexander Tivelkov


On Mon, Feb 16, 2015 at 9:50 PM, Robert Collins
robe...@robertcollins.net wrote:
 On 17 February 2015 at 03:31, Alexander Tivelkov ativel...@mirantis.com 
 wrote:
 Hi Client,

 Thanks for your input.

 We actually support the scenarios you speak about, yet in a slightly
 different way.  The authors of the Artifact Type (the plugin
 developers) may define their own custom field (or set of fields) to
 store their sequence information or any other type-specific
 version-related metadata. So, they may use generic version field
 (which is defined in the base artifact type) to store their numeric
 version - and use their type-specific field for local client-side
 processing.

 That sounds scarily like what Neutron did, leading to a different
 schema for every configuration. The reason Clint brought up Debian
 version numbers is that to sort them in a database you need a custom
 field type - e.g.
 http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/database/schema/launchpad-2209-00-0.sql#L25
 . And thats quite a burden :)

 We've had fairly poor results with the Neutron variation in schemas,
 as it tightly couples things, making upgrades that change plugins
 super tricky, as well as making it very hard to concurrently support
 multiple plugins. I hope you don't mean you're doing the same thing :)

 -Rob

 --
 Robert Collins rbtcoll...@hp.com
 Distinguished Technologist
 HP Converged Cloud

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-16 Thread Robert Collins
On 17 February 2015 at 03:31, Alexander Tivelkov ativel...@mirantis.com wrote:
 Hi Client,

 Thanks for your input.

 We actually support the scenarios you speak about, yet in a slightly
 different way.  The authors of the Artifact Type (the plugin
 developers) may define their own custom field (or set of fields) to
 store their sequence information or any other type-specific
 version-related metadata. So, they may use generic version field
 (which is defined in the base artifact type) to store their numeric
 version - and use their type-specific field for local client-side
 processing.

That sounds scarily like what Neutron did, leading to a different
schema for every configuration. The reason Clint brought up Debian
version numbers is that to sort them in a database you need a custom
field type - e.g.
http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/database/schema/launchpad-2209-00-0.sql#L25
. And thats quite a burden :)

We've had fairly poor results with the Neutron variation in schemas,
as it tightly couples things, making upgrades that change plugins
super tricky, as well as making it very hard to concurrently support
multiple plugins. I hope you don't mean you're doing the same thing :)

-Rob

-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-16 Thread Alexander Tivelkov
Hi Ian,

On Tue, Feb 10, 2015 at 11:17 PM, Ian Cordasco
ian.corda...@rackspace.com wrote:
 I think the fundamental disconnect is that not every column in a database
 needs offer sorting to the user. Imposing that restriction here causes a
 cascade of further restrictions that will fundamentally cause this to be
 unusable by a large number of people.

I didn't say that every column needs to offer sorting.
However, ability to differentiate between different artifacts and
different versions of the same artifact, as well as ability to sort
and filter these different versions according to some criteria is one
of the core features we were looking for when designing the whole
artifact proposal. In fact, the initial request for this feature was
not even made by me:  I was asked about versions at the first design
summit where we initially suggested artifacts (Icehouse mid-cycle) and
in subsequent email and IRC discussions. The first design proposal
which was presented in Atlanta already included this concept and so it
was approved there, so this feature was always considered as a natural
and important concept. I don't understand why it causes so much
confusion now, when the implementation is almost complete.

 Except for the fact that versions do typically mean more than the values
 SemVer attaches to them. SemVer is further incompatible with any
 versioning scheme using epochs and is so relatively recent compared to
 versioning practices as a whole that I don’t see how we can justify
 restricting what should be a very generic system to something so specific
 to recent history and favored almost entirely by *developers*.

I believe any person in software industry may invent their own
versioning scheme - and most of them will be incompatible with each
other. If you attempt to build something which is compatible with all
of them at once, you will eventually end up having plain strings,
without any semantic and precedence rules. This definitely does not
satisfy our initial goal (see above).
Instead, I suggest to choose the scheme which is strict, unambiguous
and satisfies our initial goal, but has maximum possible adoption
among the developers, so most of the adopters will be able to use it.
Semver seems to be the best candidate here, but any other proposals
are welcome as well.

Meanwhile this does not prevent adopters from having their own schemas
if they want it: Artifact Type developers may add their own
type-specific string metadata field, put some regexp of code-based
constraints on it - and use it to store their own version. Yet they
will not be able to utilize the version-related features which are
built-in into the Glance.

 Because up until now the only use case you’ve been referencing is CD
 software.

There is some disconnect here: I believe the message which started
this thread was the first time where I mentioned Artifact Repository
in the context of Continuous Delivery solutions; and I was saying
about CD system built on top of it, not about the Artifact Repostiory
being a CD-system on its own.  All other use-cases and scenarios did
not mention CDs at all: Artifact Repostiory is definitely a generic
catalog, not a CD solution.

--
Regards,
Alexander Tivelkov

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-16 Thread Alexander Tivelkov
Donald,

Thanks for your comments, really useful!

I think I need to clarify a bit: I am not speaking about the actual
semantic: placing the meaning into the actual values is still up to
the end-users (or the developers of Artifact Types, if they build some
custom logic which processes version info somehow).

So, this thread is really about preferred syntax scheme - and the
rules to determine precedence.
I understand that pep440 has richer syntax with more capabilities
(epochs, unlimited number of version segments, development releases
etc). My only concern is that being a python-only standard it is less
generic (in term of adoption) that the syntax of semver. The same goes
to Monty's pbr semver: it is openstack-only and thus may be confusing.
--
Regards,
Alexander Tivelkov


On Tue, Feb 10, 2015 at 11:32 PM, Donald Stufft don...@stufft.io wrote:

 On Feb 10, 2015, at 3:17 PM, Ian Cordasco ian.corda...@rackspace.com wrote:


 And of course, the chosen solution should be mappable to database, so
 we may do sorting and filtering on the DB-side.
 So, having it as a simple string and letting the user to decide what
 it means is not an option.

 Except for the fact that versions do typically mean more than the values
 SemVer attaches to them. SemVer is further incompatible with any
 versioning scheme using epochs and is so relatively recent compared to
 versioning practices as a whole that I don’t see how we can justify
 restricting what should be a very generic system to something so specific
 to recent history and favored almost entirely by *developers*.

 Semver vs PEP 440 is largely a syntax question since PEP 440 purposely does 
 not
 have much of an opinion on how something like 2.0.0 and 2.1.0 are related 
 other
 than for sorting. We do have operators in PEP 440 that support treating these
 versions in a semver like way, and some that support treating them in other
 ways.

 The primary purpose of PEP 440 was to define a standard way to parse and sort
 and specify versions across several hundred thouse versions that currently
 exist in PyPI. This means that it is more complicated to implement but it is
 much more powerful than semver eve could be. One example, as Ian mentioned is
 the lack of the ability to do an Epoch, another example is that PEP 440 has
 explicit support for someone taking version 1.0 adding some unofficial patches
 to it, and then releasing that in their own distribution channels.

 The primary purpose of Semver was to be extremely opinionated in what meaning
 you place on the *content* of the version parts and the syntax is really a
 secondary concern which exists just to make it easier to parse. This means 
 that
 if you know ahead of time that something is Semver you can guess a lot more
 information about the relationship of two versions.

 It was our intention that PEP 440 would (is?) aimed primarily at people
 implementing tools that work with versions, and the additional PEPs or other
 documentations would be written on top of PEP 440 to add opinions on what a
 version looks like within the framework that PEP 440 sets up. A great example
 is the pbr semver document that Monty linked.

 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-16 Thread Alexander Tivelkov
Hi Client,

Thanks for your input.

We actually support the scenarios you speak about, yet in a slightly
different way.  The authors of the Artifact Type (the plugin
developers) may define their own custom field (or set of fields) to
store their sequence information or any other type-specific
version-related metadata. So, they may use generic version field
(which is defined in the base artifact type) to store their numeric
version - and use their type-specific field for local client-side
processing.

--
Regards,
Alexander Tivelkov


On Tue, Feb 10, 2015 at 11:37 PM, Clint Byrum cl...@fewbar.com wrote:
 Excerpts from Alexander Tivelkov's message of 2015-02-10 07:28:55 -0800:
 Hi folks,

 One of the key features that we are adding to Glance with the
 introduction of Artifacts is the ability to have multiple versions of
 the same object in the repository: this gives us the possibility to
 query for the latest version of something, keep track on the changes
 history, and build various continuous delivery solutions on top of
 Artifact Repository.

 We need to determine the format and rules we will use to define,
 increment and compare versions of artifacts in the repository. There
 are two alternatives we have to choose from, and we are seeking advice
 on this choice.

 First, there is Semantic Versioning specification, available at [1].
 It is a very generic spec, widely used and adopted in many areas of
 software development. It is quite straightforward: 3 mandatory numeric
 components for version number, plus optional string labels for
 pre-release versions and build metadata.

 And then there is PEP-440 spec, which is a recommended approach to
 identifying versions and specifying dependencies when distributing
 Python. It is a pythonic way to set versions of python packages,
 including PIP version strings.

 Conceptually PEP-440 and Semantic Versioning are similar in purpose,
 but slightly different in syntax. Notably, the count of version number
 components and rules of version precedence resolution differ between
 PEP-440 and SemVer. Unfortunately, the two version string formats are
 not compatible, so we have to choose one or the other.

 According to my initial vision, the Artifact Repository should be as
 generic as possible in terms of potential adoption. The artifacts were
 never supposed to be python packages only, and even the projects which
 will create and use these artifacts are not mandatory limited to be
 pythonic, the developers of that projects may not be python
 developers! So, I'd really wanted to avoid any python-specific
 notations, such as PEP-440 for artifacts.

 I've put this vision into a spec [3] which also contains a proposal on
 how to convert the semver-compatible version strings into the
 comparable values which may be mapped to database types, so a database
 table may be queried, ordered and filtered by the object version.

 So, we need some feedback on this topic. Would you prefer artifacts to
 be versioned with SemVer or with PEP-440 notation? Are you interested
 in having some generic utility which will map versions (in either
 format) to database columns? If so, which version format would you
 prefer?

 We are on a tight schedule here, as we want to begin landing
 artifact-related code soon. So, I would appreciate your feedback
 during this week: here in the ML or in the comments to [3] review.


 Hi. This is really interesting work and I'm glad Glance is growing into
 an artifact catalog as I think it will assist cloud users and UI
 development at the same time.

 It seems to me that there are really only two reasons to care about the
 content of the versions: sorting, and filtering. You want to make sure
 if people upload artifacts named myapp like this:

 myapp:1.0 myapp:2.0 myapp:1.1

 That when they say show me the newest myapp they get 2.0, not 1.1.

 And if they say show me the newest myapp in the 1.x series they get 1.1.

 I am a little worried this is not something that can or should be made
 generic in a micro service.

 Here's a thought: You could just have the version, series, and sequence,
 and let users manage the sequencing themselves on the client side. This
 way if users want to use the _extremely_ difficult to program for Debian
 packaging version, you don't have to figure out how to make 1.0~special
 less than 1.0 and more than 0.9.

 To start with, you can have a default strategy of a single series, and
 max(sequence)+1000 if unspecified. Then teach the clients the various
 semvers/pep440's/etc. etc. and let them choose their own sequencing and
 series strategy.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not 

Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Monty Taylor
On 02/10/2015 10:28 AM, Alexander Tivelkov wrote:
 Hi folks,
 
 One of the key features that we are adding to Glance with the
 introduction of Artifacts is the ability to have multiple versions of
 the same object in the repository: this gives us the possibility to
 query for the latest version of something, keep track on the changes
 history, and build various continuous delivery solutions on top of
 Artifact Repository.
 
 We need to determine the format and rules we will use to define,
 increment and compare versions of artifacts in the repository. There
 are two alternatives we have to choose from, and we are seeking advice
 on this choice.
 
 First, there is Semantic Versioning specification, available at [1].
 It is a very generic spec, widely used and adopted in many areas of
 software development. It is quite straightforward: 3 mandatory numeric
 components for version number, plus optional string labels for
 pre-release versions and build metadata.
 
 And then there is PEP-440 spec, which is a recommended approach to
 identifying versions and specifying dependencies when distributing
 Python. It is a pythonic way to set versions of python packages,
 including PIP version strings.
 
 Conceptually PEP-440 and Semantic Versioning are similar in purpose,
 but slightly different in syntax. Notably, the count of version number
 components and rules of version precedence resolution differ between
 PEP-440 and SemVer. Unfortunately, the two version string formats are
 not compatible, so we have to choose one or the other.
 
 According to my initial vision, the Artifact Repository should be as
 generic as possible in terms of potential adoption. The artifacts were
 never supposed to be python packages only, and even the projects which
 will create and use these artifacts are not mandatory limited to be
 pythonic, the developers of that projects may not be python
 developers! So, I'd really wanted to avoid any python-specific
 notations, such as PEP-440 for artifacts.
 
 I've put this vision into a spec [3] which also contains a proposal on
 how to convert the semver-compatible version strings into the
 comparable values which may be mapped to database types, so a database
 table may be queried, ordered and filtered by the object version.
 
 So, we need some feedback on this topic. Would you prefer artifacts to
 be versioned with SemVer or with PEP-440 notation? Are you interested
 in having some generic utility which will map versions (in either
 format) to database columns? If so, which version format would you
 prefer?
 
 We are on a tight schedule here, as we want to begin landing
 artifact-related code soon. So, I would appreciate your feedback
 during this week: here in the ML or in the comments to [3] review.

Because you should have more things to look at:

http://docs.openstack.org/developer/pbr/semver.html

We've already done some work to try to reconcile the world of semver
with the world of PEP440 over in PBR land.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Alexander Tivelkov
Hi Ian,

Automatic version generation is not the only and not the primary
reason for the version concept. In fact, the implementation which is
planned to land in this cycle does not contain this feature at all:
currently we also leave the version assignment up to uploader (version
is a regular immutable generic artifact property). Auto-increments as
part of clone-and-modify scenarios are postponed for the next cycle.

However, even now we do need to have some sorting order - so, we need
rules to determine precedence. That's the reason for having some
notation defined: if we leave the notation up to the end-user we won't
be able to compare artifacts having versions in different notations.
And we even can't leave it up to the Artifact Type developer, since
this is a generic property, thus common for all the artifact types.
And of course, the chosen solution should be mappable to database, so
we may do sorting and filtering on the DB-side.
So, having it as a simple string and letting the user to decide what
it means is not an option.

Speaking about Artifactory - that's entirely different thing. It is
indeed a continuous delivery solution, composed around build machines,
deployment solutions and CI systems. That's definitely not what Glance
Artifact Repository is. Even the concepts of Artifact are entirely
different.  So, while Artifact Repository may be used to build some CD
solutions on top of it (or to be integrated with the existing ones) it
is not a storage solution for build outputs and thus I can barely see
how we may compare them.

--
Regards,
Alexander Tivelkov


On Tue, Feb 10, 2015 at 8:15 PM, Ian Cordasco
ian.corda...@rackspace.com wrote:


 On 2/10/15, 10:35, Alexander Tivelkov ativel...@mirantis.com wrote:

Thanks Monty!

Yup, probably I've missed that. I was looking at pbr and its version
implementation, but didn't realize that this is actually a fusion of
semver and pep440.

So, we have this as an extra alternative to choose from.

It would be an obvious choice if we were just looking for some common
solution to version objects within openstack. However, I am a bit
concerned about applying it to Artifact Repository. As I wrote before,
we are trying to make the Repository to be language- and
platform-agnostic tool for other developers, including the ones
originating from non-python and non-openstack worlds. Having a
versioning notation which is non-standard for everybody but openstack
developers does not look like a good idea to me.
--
Regards,
Alexander Tivelkov


On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com
wrote:
 On 02/10/2015 10:28 AM, Alexander Tivelkov wrote:
 Hi folks,

 One of the key features that we are adding to Glance with the
 introduction of Artifacts is the ability to have multiple versions of
 the same object in the repository: this gives us the possibility to
 query for the latest version of something, keep track on the changes
 history, and build various continuous delivery solutions on top of
 Artifact Repository.

 We need to determine the format and rules we will use to define,
 increment and compare versions of artifacts in the repository. There
 are two alternatives we have to choose from, and we are seeking advice
 on this choice.

 First, there is Semantic Versioning specification, available at [1].
 It is a very generic spec, widely used and adopted in many areas of
 software development. It is quite straightforward: 3 mandatory numeric
 components for version number, plus optional string labels for
 pre-release versions and build metadata.

 And then there is PEP-440 spec, which is a recommended approach to
 identifying versions and specifying dependencies when distributing
 Python. It is a pythonic way to set versions of python packages,
 including PIP version strings.

 Conceptually PEP-440 and Semantic Versioning are similar in purpose,
 but slightly different in syntax. Notably, the count of version number
 components and rules of version precedence resolution differ between
 PEP-440 and SemVer. Unfortunately, the two version string formats are
 not compatible, so we have to choose one or the other.

 According to my initial vision, the Artifact Repository should be as
 generic as possible in terms of potential adoption. The artifacts were
 never supposed to be python packages only, and even the projects which
 will create and use these artifacts are not mandatory limited to be
 pythonic, the developers of that projects may not be python
 developers! So, I'd really wanted to avoid any python-specific
 notations, such as PEP-440 for artifacts.

 I've put this vision into a spec [3] which also contains a proposal on
 how to convert the semver-compatible version strings into the
 comparable values which may be mapped to database types, so a database
 table may be queried, ordered and filtered by the object version.

 So, we need some feedback on this topic. Would you prefer artifacts to
 be versioned with SemVer or with PEP-440 notation? Are 

Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Alexander Tivelkov
Thanks Monty!

Yup, probably I've missed that. I was looking at pbr and its version
implementation, but didn't realize that this is actually a fusion of
semver and pep440.

So, we have this as an extra alternative to choose from.

It would be an obvious choice if we were just looking for some common
solution to version objects within openstack. However, I am a bit
concerned about applying it to Artifact Repository. As I wrote before,
we are trying to make the Repository to be language- and
platform-agnostic tool for other developers, including the ones
originating from non-python and non-openstack worlds. Having a
versioning notation which is non-standard for everybody but openstack
developers does not look like a good idea to me.
--
Regards,
Alexander Tivelkov


On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com wrote:
 On 02/10/2015 10:28 AM, Alexander Tivelkov wrote:
 Hi folks,

 One of the key features that we are adding to Glance with the
 introduction of Artifacts is the ability to have multiple versions of
 the same object in the repository: this gives us the possibility to
 query for the latest version of something, keep track on the changes
 history, and build various continuous delivery solutions on top of
 Artifact Repository.

 We need to determine the format and rules we will use to define,
 increment and compare versions of artifacts in the repository. There
 are two alternatives we have to choose from, and we are seeking advice
 on this choice.

 First, there is Semantic Versioning specification, available at [1].
 It is a very generic spec, widely used and adopted in many areas of
 software development. It is quite straightforward: 3 mandatory numeric
 components for version number, plus optional string labels for
 pre-release versions and build metadata.

 And then there is PEP-440 spec, which is a recommended approach to
 identifying versions and specifying dependencies when distributing
 Python. It is a pythonic way to set versions of python packages,
 including PIP version strings.

 Conceptually PEP-440 and Semantic Versioning are similar in purpose,
 but slightly different in syntax. Notably, the count of version number
 components and rules of version precedence resolution differ between
 PEP-440 and SemVer. Unfortunately, the two version string formats are
 not compatible, so we have to choose one or the other.

 According to my initial vision, the Artifact Repository should be as
 generic as possible in terms of potential adoption. The artifacts were
 never supposed to be python packages only, and even the projects which
 will create and use these artifacts are not mandatory limited to be
 pythonic, the developers of that projects may not be python
 developers! So, I'd really wanted to avoid any python-specific
 notations, such as PEP-440 for artifacts.

 I've put this vision into a spec [3] which also contains a proposal on
 how to convert the semver-compatible version strings into the
 comparable values which may be mapped to database types, so a database
 table may be queried, ordered and filtered by the object version.

 So, we need some feedback on this topic. Would you prefer artifacts to
 be versioned with SemVer or with PEP-440 notation? Are you interested
 in having some generic utility which will map versions (in either
 format) to database columns? If so, which version format would you
 prefer?

 We are on a tight schedule here, as we want to begin landing
 artifact-related code soon. So, I would appreciate your feedback
 during this week: here in the ML or in the comments to [3] review.

 Because you should have more things to look at:

 http://docs.openstack.org/developer/pbr/semver.html

 We've already done some work to try to reconcile the world of semver
 with the world of PEP440 over in PBR land.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Ian Cordasco


On 2/10/15, 10:35, Alexander Tivelkov ativel...@mirantis.com wrote:

Thanks Monty!

Yup, probably I've missed that. I was looking at pbr and its version
implementation, but didn't realize that this is actually a fusion of
semver and pep440.

So, we have this as an extra alternative to choose from.

It would be an obvious choice if we were just looking for some common
solution to version objects within openstack. However, I am a bit
concerned about applying it to Artifact Repository. As I wrote before,
we are trying to make the Repository to be language- and
platform-agnostic tool for other developers, including the ones
originating from non-python and non-openstack worlds. Having a
versioning notation which is non-standard for everybody but openstack
developers does not look like a good idea to me.
--
Regards,
Alexander Tivelkov


On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com
wrote:
 On 02/10/2015 10:28 AM, Alexander Tivelkov wrote:
 Hi folks,

 One of the key features that we are adding to Glance with the
 introduction of Artifacts is the ability to have multiple versions of
 the same object in the repository: this gives us the possibility to
 query for the latest version of something, keep track on the changes
 history, and build various continuous delivery solutions on top of
 Artifact Repository.

 We need to determine the format and rules we will use to define,
 increment and compare versions of artifacts in the repository. There
 are two alternatives we have to choose from, and we are seeking advice
 on this choice.

 First, there is Semantic Versioning specification, available at [1].
 It is a very generic spec, widely used and adopted in many areas of
 software development. It is quite straightforward: 3 mandatory numeric
 components for version number, plus optional string labels for
 pre-release versions and build metadata.

 And then there is PEP-440 spec, which is a recommended approach to
 identifying versions and specifying dependencies when distributing
 Python. It is a pythonic way to set versions of python packages,
 including PIP version strings.

 Conceptually PEP-440 and Semantic Versioning are similar in purpose,
 but slightly different in syntax. Notably, the count of version number
 components and rules of version precedence resolution differ between
 PEP-440 and SemVer. Unfortunately, the two version string formats are
 not compatible, so we have to choose one or the other.

 According to my initial vision, the Artifact Repository should be as
 generic as possible in terms of potential adoption. The artifacts were
 never supposed to be python packages only, and even the projects which
 will create and use these artifacts are not mandatory limited to be
 pythonic, the developers of that projects may not be python
 developers! So, I'd really wanted to avoid any python-specific
 notations, such as PEP-440 for artifacts.

 I've put this vision into a spec [3] which also contains a proposal on
 how to convert the semver-compatible version strings into the
 comparable values which may be mapped to database types, so a database
 table may be queried, ordered and filtered by the object version.

 So, we need some feedback on this topic. Would you prefer artifacts to
 be versioned with SemVer or with PEP-440 notation? Are you interested
 in having some generic utility which will map versions (in either
 format) to database columns? If so, which version format would you
 prefer?

 We are on a tight schedule here, as we want to begin landing
 artifact-related code soon. So, I would appreciate your feedback
 during this week: here in the ML or in the comments to [3] review.

 Because you should have more things to look at:

 http://docs.openstack.org/developer/pbr/semver.html

 We've already done some work to try to reconcile the world of semver
 with the world of PEP440 over in PBR land.

 
_
_
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

So Semantic Versioning, as I’ve already mentioned in the past, isn’t
really a de facto standard in any language community but it is a language
agnostic proposal. That said, just because it’s language agnostic does not
mean it won’t conflict with other language’s versioning semantics. Since
we’re effectively reinventing an existing open source solution here, I
think we should look to how Artifactory [1] handles this.

I haven’t used artifactory very much but a cursory look makes it apparent
that it is strongly decoupling the logic of 

Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Donald Stufft

 On Feb 10, 2015, at 3:17 PM, Ian Cordasco ian.corda...@rackspace.com wrote:
 
 
 And of course, the chosen solution should be mappable to database, so
 we may do sorting and filtering on the DB-side.
 So, having it as a simple string and letting the user to decide what
 it means is not an option.
 
 Except for the fact that versions do typically mean more than the values
 SemVer attaches to them. SemVer is further incompatible with any
 versioning scheme using epochs and is so relatively recent compared to
 versioning practices as a whole that I don’t see how we can justify
 restricting what should be a very generic system to something so specific
 to recent history and favored almost entirely by *developers*.

Semver vs PEP 440 is largely a syntax question since PEP 440 purposely does not
have much of an opinion on how something like 2.0.0 and 2.1.0 are related other
than for sorting. We do have operators in PEP 440 that support treating these
versions in a semver like way, and some that support treating them in other
ways.

The primary purpose of PEP 440 was to define a standard way to parse and sort
and specify versions across several hundred thouse versions that currently
exist in PyPI. This means that it is more complicated to implement but it is
much more powerful than semver eve could be. One example, as Ian mentioned is
the lack of the ability to do an Epoch, another example is that PEP 440 has
explicit support for someone taking version 1.0 adding some unofficial patches
to it, and then releasing that in their own distribution channels.

The primary purpose of Semver was to be extremely opinionated in what meaning
you place on the *content* of the version parts and the syntax is really a
secondary concern which exists just to make it easier to parse. This means that
if you know ahead of time that something is Semver you can guess a lot more
information about the relationship of two versions.

It was our intention that PEP 440 would (is?) aimed primarily at people
implementing tools that work with versions, and the additional PEPs or other
documentations would be written on top of PEP 440 to add opinions on what a
version looks like within the framework that PEP 440 sets up. A great example
is the pbr semver document that Monty linked.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Adrian Otto
I agree with Alexander on this. We should certainly learn what we can from 
existing software. That said, the Solum team really wants this feature in 
Glance so we can leverage that instead of having our own repository for Heat 
templates we generate when building apps. We want to keep our requirements list 
small, and Glance is already there.

Adrian

On Feb 10, 2015, at 10:01 AM, Alexander Tivelkov ativel...@mirantis.com wrote:

 Hi Ian,
 
 Automatic version generation is not the only and not the primary
 reason for the version concept. In fact, the implementation which is
 planned to land in this cycle does not contain this feature at all:
 currently we also leave the version assignment up to uploader (version
 is a regular immutable generic artifact property). Auto-increments as
 part of clone-and-modify scenarios are postponed for the next cycle.
 
 However, even now we do need to have some sorting order - so, we need
 rules to determine precedence. That's the reason for having some
 notation defined: if we leave the notation up to the end-user we won't
 be able to compare artifacts having versions in different notations.
 And we even can't leave it up to the Artifact Type developer, since
 this is a generic property, thus common for all the artifact types.
 And of course, the chosen solution should be mappable to database, so
 we may do sorting and filtering on the DB-side.
 So, having it as a simple string and letting the user to decide what
 it means is not an option.
 
 Speaking about Artifactory - that's entirely different thing. It is
 indeed a continuous delivery solution, composed around build machines,
 deployment solutions and CI systems. That's definitely not what Glance
 Artifact Repository is. Even the concepts of Artifact are entirely
 different.  So, while Artifact Repository may be used to build some CD
 solutions on top of it (or to be integrated with the existing ones) it
 is not a storage solution for build outputs and thus I can barely see
 how we may compare them.
 
 --
 Regards,
 Alexander Tivelkov
 
 
 On Tue, Feb 10, 2015 at 8:15 PM, Ian Cordasco
 ian.corda...@rackspace.com wrote:
 
 
 On 2/10/15, 10:35, Alexander Tivelkov ativel...@mirantis.com wrote:
 
 Thanks Monty!
 
 Yup, probably I've missed that. I was looking at pbr and its version
 implementation, but didn't realize that this is actually a fusion of
 semver and pep440.
 
 So, we have this as an extra alternative to choose from.
 
 It would be an obvious choice if we were just looking for some common
 solution to version objects within openstack. However, I am a bit
 concerned about applying it to Artifact Repository. As I wrote before,
 we are trying to make the Repository to be language- and
 platform-agnostic tool for other developers, including the ones
 originating from non-python and non-openstack worlds. Having a
 versioning notation which is non-standard for everybody but openstack
 developers does not look like a good idea to me.
 --
 Regards,
 Alexander Tivelkov
 
 
 On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com
 wrote:
 On 02/10/2015 10:28 AM, Alexander Tivelkov wrote:
 Hi folks,
 
 One of the key features that we are adding to Glance with the
 introduction of Artifacts is the ability to have multiple versions of
 the same object in the repository: this gives us the possibility to
 query for the latest version of something, keep track on the changes
 history, and build various continuous delivery solutions on top of
 Artifact Repository.
 
 We need to determine the format and rules we will use to define,
 increment and compare versions of artifacts in the repository. There
 are two alternatives we have to choose from, and we are seeking advice
 on this choice.
 
 First, there is Semantic Versioning specification, available at [1].
 It is a very generic spec, widely used and adopted in many areas of
 software development. It is quite straightforward: 3 mandatory numeric
 components for version number, plus optional string labels for
 pre-release versions and build metadata.
 
 And then there is PEP-440 spec, which is a recommended approach to
 identifying versions and specifying dependencies when distributing
 Python. It is a pythonic way to set versions of python packages,
 including PIP version strings.
 
 Conceptually PEP-440 and Semantic Versioning are similar in purpose,
 but slightly different in syntax. Notably, the count of version number
 components and rules of version precedence resolution differ between
 PEP-440 and SemVer. Unfortunately, the two version string formats are
 not compatible, so we have to choose one or the other.
 
 According to my initial vision, the Artifact Repository should be as
 generic as possible in terms of potential adoption. The artifacts were
 never supposed to be python packages only, and even the projects which
 will create and use these artifacts are not mandatory limited to be
 pythonic, the developers of that projects may not be python
 

Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Ian Cordasco
On 2/10/15, 12:01, Alexander Tivelkov ativel...@mirantis.com wrote:

Hi Ian,

Automatic version generation is not the only and not the primary
reason for the version concept. In fact, the implementation which is
planned to land in this cycle does not contain this feature at all:
currently we also leave the version assignment up to uploader (version
is a regular immutable generic artifact property). Auto-increments as
part of clone-and-modify scenarios are postponed for the next cycle.

However, even now we do need to have some sorting order - so, we need
rules to determine precedence. That's the reason for having some
notation defined: if we leave the notation up to the end-user we won't
be able to compare artifacts having versions in different notations.
And we even can't leave it up to the Artifact Type developer, since
this is a generic property, thus common for all the artifact types.

I think the fundamental disconnect is that not every column in a database
needs offer sorting to the user. Imposing that restriction here causes a
cascade of further restrictions that will fundamentally cause this to be
unusable by a large number of people.

And of course, the chosen solution should be mappable to database, so
we may do sorting and filtering on the DB-side.
So, having it as a simple string and letting the user to decide what
it means is not an option.

Except for the fact that versions do typically mean more than the values
SemVer attaches to them. SemVer is further incompatible with any
versioning scheme using epochs and is so relatively recent compared to
versioning practices as a whole that I don’t see how we can justify
restricting what should be a very generic system to something so specific
to recent history and favored almost entirely by *developers*.

Speaking about Artifactory - that's entirely different thing. It is
indeed a continuous delivery solution, composed around build machines,
deployment solutions and CI systems. That's definitely not what Glance
Artifact Repository is. Even the concepts of Artifact are entirely
different.  So, while Artifact Repository may be used to build some CD
solutions on top of it (or to be integrated with the existing ones) it
is not a storage solution for build outputs and thus I can barely see
how we may compare them.

Because up until now the only use case you’ve been referencing is CD
software.

--
Regards,
Alexander Tivelkov


On Tue, Feb 10, 2015 at 8:15 PM, Ian Cordasco
ian.corda...@rackspace.com wrote:


 On 2/10/15, 10:35, Alexander Tivelkov ativel...@mirantis.com wrote:

Thanks Monty!

Yup, probably I've missed that. I was looking at pbr and its version
implementation, but didn't realize that this is actually a fusion of
semver and pep440.

So, we have this as an extra alternative to choose from.

It would be an obvious choice if we were just looking for some common
solution to version objects within openstack. However, I am a bit
concerned about applying it to Artifact Repository. As I wrote before,
we are trying to make the Repository to be language- and
platform-agnostic tool for other developers, including the ones
originating from non-python and non-openstack worlds. Having a
versioning notation which is non-standard for everybody but openstack
developers does not look like a good idea to me.
--
Regards,
Alexander Tivelkov


On Tue, Feb 10, 2015 at 6:55 PM, Monty Taylor mord...@inaugust.com
wrote:
 On 02/10/2015 10:28 AM, Alexander Tivelkov wrote:
 Hi folks,

 One of the key features that we are adding to Glance with the
 introduction of Artifacts is the ability to have multiple versions of
 the same object in the repository: this gives us the possibility to
 query for the latest version of something, keep track on the changes
 history, and build various continuous delivery solutions on top of
 Artifact Repository.

 We need to determine the format and rules we will use to define,
 increment and compare versions of artifacts in the repository. There
 are two alternatives we have to choose from, and we are seeking
advice
 on this choice.

 First, there is Semantic Versioning specification, available at [1].
 It is a very generic spec, widely used and adopted in many areas of
 software development. It is quite straightforward: 3 mandatory
numeric
 components for version number, plus optional string labels for
 pre-release versions and build metadata.

 And then there is PEP-440 spec, which is a recommended approach to
 identifying versions and specifying dependencies when distributing
 Python. It is a pythonic way to set versions of python packages,
 including PIP version strings.

 Conceptually PEP-440 and Semantic Versioning are similar in purpose,
 but slightly different in syntax. Notably, the count of version
number
 components and rules of version precedence resolution differ between
 PEP-440 and SemVer. Unfortunately, the two version string formats are
 not compatible, so we have to choose one or the other.

 According to my initial vision, 

Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Ian Cordasco
On 2/10/15, 13:55, Jay Pipes jaypi...@gmail.com wrote:

On 02/10/2015 12:15 PM, Ian Cordasco wrote:
 So Semantic Versioning, as I’ve already mentioned in the past, isn’t
 really a de facto standard in any language community but it is a
language
 agnostic proposal. That said, just because it’s language agnostic does
not
 mean it won’t conflict with other language’s versioning semantics. Since
 we’re effectively reinventing an existing open source solution here, I
 think we should look to how Artifactory [1] handles this.

Reinventing an existing open source solution is a bit off-the-mark
IMO. Artifactory is more of a SaaS solution through bintray.com --
paying some lip-service to open source more than anything else.

Except the base service is entirely free and open and thus can be deployed
anywhere. And people looking to rebuild CD services in OpenStack should
probably refer to existing implementations (of which I only pointed out
the one I had already heard about).


 I haven’t used artifactory very much but a cursory look makes it
apparent
 that it is strongly decoupling the logic of version management with
 artifact management (which this set of changes isn’t doing in Glance).

Alex's spec is merely proposing to standardize on a single way of
managing versioning. It isn't coupling artifact *management* with
anything whatsoever. In fact, the idea of Glance being an artifact
repository has nothing to do with the management of said artifacts --
which things like Heat, Murano or Solum handle in different ways.

Except that Alex is proposing that this would include a number of things,
some of which can and will be versioned in a way that will not work with
the proposed solution.

The idea behind the Glance artifact repository was to store a
discoverable *schema* for various objects, along with the object
metadata itself. I can see Nova and Cinder flavors, Glance, Docker or
AppC image metadata, Cinder snapshot and differential metadata, Murano
application manifests, Heat templates, and Solum deployment components
all being described (schemas) and stored in the Glance artifact
repository.

What I have not heard from anyone is the idea to marry the management of
the things that this artifact metadata with Glance itself.

Right. That’s the other concern I have with Artifacts. Images /can/ be
/eventually/ represented as artifacts, but for now we’re grafting what
will functionally be an entirely separate project (for at least the next
cycle, if not longer) onto Glance while that new project is unstable and
at best experimental.

 The primary argument (as I understand it) for using SemVer with
Artifacts
 in glance is to have a way to automatically have the service create the
 version number for the uploader. Artifactory (and it’s “Pro” version)
seen
 to leave that to the build tooling. In other words, it is purely
storage.
 It seems to sort versions alphabetically (which everyone can agree is
not
 only suboptimal but wrong) but it also provides the user a way to alter
 how it performs sorting.

I don't know where you get the idea that a service (Glance, you mean?)
would automatically create the version number for some artifact. The
spec talks only about the ability of the artifact registry to sort a set
of artifacts by incrementing version number, and do so in a reasonably
short time frame -- i.e. have a database column indexed for that purpose.

Several discussions in IRC have happened around Artifacts and the SemVer
spec in which this is a feature that Alex wants to add as a consequence of
this. 


 Is there a reason (beyond not being an OpenStack project) that
Artifactory
 isn’t being considered by those pushing for Artifacts to provide a CD
 service?

Because we're not looking to create a CD service? :) AFAIK, the artifact
repository is a generic storage system for schemas and metadata, nothing
more.

Except the common explanation for why we /need/ Artifacts is the example
of people wanting to set up a CD system and the discussions that have been
held elsewhere point to this being so much more than just that in terms of
responsibility of managing versions and other (specifically, build)
artifacts.


Best,
-jay

  Judging by the support it seems to roughly have for other
 services (via a “Pro” version) it would appear to be able to suit any
this
 need well with little work by the deployer who needs to serve more than
 one use case.

 [1] http://www.jfrog.com/open-source/#os-arti

 
_
_
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [Glance][Artifacts] Object Version format: SemVer vs pep440

2015-02-10 Thread Clint Byrum
Excerpts from Alexander Tivelkov's message of 2015-02-10 07:28:55 -0800:
 Hi folks,
 
 One of the key features that we are adding to Glance with the
 introduction of Artifacts is the ability to have multiple versions of
 the same object in the repository: this gives us the possibility to
 query for the latest version of something, keep track on the changes
 history, and build various continuous delivery solutions on top of
 Artifact Repository.
 
 We need to determine the format and rules we will use to define,
 increment and compare versions of artifacts in the repository. There
 are two alternatives we have to choose from, and we are seeking advice
 on this choice.
 
 First, there is Semantic Versioning specification, available at [1].
 It is a very generic spec, widely used and adopted in many areas of
 software development. It is quite straightforward: 3 mandatory numeric
 components for version number, plus optional string labels for
 pre-release versions and build metadata.
 
 And then there is PEP-440 spec, which is a recommended approach to
 identifying versions and specifying dependencies when distributing
 Python. It is a pythonic way to set versions of python packages,
 including PIP version strings.
 
 Conceptually PEP-440 and Semantic Versioning are similar in purpose,
 but slightly different in syntax. Notably, the count of version number
 components and rules of version precedence resolution differ between
 PEP-440 and SemVer. Unfortunately, the two version string formats are
 not compatible, so we have to choose one or the other.
 
 According to my initial vision, the Artifact Repository should be as
 generic as possible in terms of potential adoption. The artifacts were
 never supposed to be python packages only, and even the projects which
 will create and use these artifacts are not mandatory limited to be
 pythonic, the developers of that projects may not be python
 developers! So, I'd really wanted to avoid any python-specific
 notations, such as PEP-440 for artifacts.
 
 I've put this vision into a spec [3] which also contains a proposal on
 how to convert the semver-compatible version strings into the
 comparable values which may be mapped to database types, so a database
 table may be queried, ordered and filtered by the object version.
 
 So, we need some feedback on this topic. Would you prefer artifacts to
 be versioned with SemVer or with PEP-440 notation? Are you interested
 in having some generic utility which will map versions (in either
 format) to database columns? If so, which version format would you
 prefer?
 
 We are on a tight schedule here, as we want to begin landing
 artifact-related code soon. So, I would appreciate your feedback
 during this week: here in the ML or in the comments to [3] review.
 

Hi. This is really interesting work and I'm glad Glance is growing into
an artifact catalog as I think it will assist cloud users and UI
development at the same time.

It seems to me that there are really only two reasons to care about the
content of the versions: sorting, and filtering. You want to make sure
if people upload artifacts named myapp like this:

myapp:1.0 myapp:2.0 myapp:1.1

That when they say show me the newest myapp they get 2.0, not 1.1.

And if they say show me the newest myapp in the 1.x series they get 1.1.

I am a little worried this is not something that can or should be made
generic in a micro service.

Here's a thought: You could just have the version, series, and sequence,
and let users manage the sequencing themselves on the client side. This
way if users want to use the _extremely_ difficult to program for Debian
packaging version, you don't have to figure out how to make 1.0~special
less than 1.0 and more than 0.9.

To start with, you can have a default strategy of a single series, and
max(sequence)+1000 if unspecified. Then teach the clients the various
semvers/pep440's/etc. etc. and let them choose their own sequencing and
series strategy.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev