Re: [openstack-dev] Golang technical requirements
On 2016-12-14 17:49:46 -0600 (-0600), Monty Taylor wrote: [...] > For things we generate in pbr - like AUTHORS and ChangeLog - assuming we > want to do the same thing in go, we could likely come up with something > similar. I also think we should consider that perhaps those two files > are not needed if the primary expectation for distribution is git. Completely agree on this point. We auto-generate those two files in particular as a means for us to export data from our revision control system (specifically data which is not normally represented in the file content under revision control) such that it can be conveyed within a tarball that would otherwise lack the relevant context to derive it. Some teams do hand-create those files instead and check them into revision control, but could of course continue to do so in Git repos of Go code as well with no real change to their usual process. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
On 12/14/2016 09:56 AM, Doug Hellmann wrote: > Excerpts from Dean Troyer's message of 2016-12-14 08:15:08 -0600: >> On Wed, Dec 14, 2016 at 7:46 AM, Doug Hellmann wrote: >>> FWIW, some of the deployment tool communities (ansible and puppet, >>> I think) rely on git repos without external artifacts, and we're >>> supporting them in our release tools today. I'm not sure how >>> downstream packagers feel about the lack of external artifacts. >> >> I recall zigo working hard for the Debian packaging to come strictly >> from Git repo tags but do not recall seeing opinions from other >> packagers. >> >>> Were you thinking of files other than the tarball when you say >>> "generated files"? >> >> Yes, in particular, files like AUTHORS and sample config files, which >> IIRC are generated and included in tarballs. > > OK, those should be easy enough to deal with if we publish tarballs > of golang source. Perhaps someone from one of our other projects > that uses git tags as releases can comment on how they handle those > things? In oaktree, which has golang bindings, and also because of gRPC/protobuf has generated files - I have followed what I could see of other folks and ... checked the generated files into git, along with a gate job to ensure that patches that change the source of those generated files are always accompanied by changes to the generated files (and that the generated file content is the same) It makes me feel dirty - but with the gate check it doesn't bother me as much as it would have in times past. Other people theoretically wanting to consume those go bindings as a library need to be able to do so with go get - and that means the files need to be there. For things we generate in pbr - like AUTHORS and ChangeLog - assuming we want to do the same thing in go, we could likely come up with something similar. I also think we should consider that perhaps those two files are not needed if the primary expectation for distribution is git. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
On 12/13/2016 04:45 PM, Dean Troyer wrote: > On Tue, Dec 13, 2016 at 4:26 PM, Jay Pipes wrote: >> Not sure if it's already been brought up, but I really like govendor: > > Govendor and glide are the two most promising candidates at this > point. I just read today's summary of the Package Management > Committee [0] and that sounds _really_ promising, if further out than > we can wait on, but there will be migration paths from many of the > existing tools. Both govendor and glide have devs in the Advisory > Group to the committee. Yup - looked at govendor, glide and godep back in May: https://review.openstack.org/315200 - glide https://review.openstack.org/315196 - govendor https://review.openstack.org/315029 - godep >> More than the decision between dependency management toolkits, though, is >> the decision over whether to have *any* vendored source code in the primary >> project's source tree itself (as opposed to only storing the >> vendor.json/glide.yaml|lock file that lists dependent library versions and >> locations). Please let's have a policy of keeping vendored source code *out* >> of the primary project source tree. > > This is the intention and I see no reason why it is not achievable. > It totally depends (ha!) on having better (any!) dependency management > in place. ++ I think we're all fairly well agreed on not vendoring code in OpenStack. IANAL but I also think our CLA would make a patch to vendor code into an OpenStack repo particularly legally strange. (a contributor quite simply does not have the legal standing to apply the terms of our CLA to the vendored code) I don't know for a fact that this is strictly problematic, but given that it does seem at best grey, and the fact that vendoring code is a TERRIBLE engineering decision, I'm pretty sure we can steer clear of it. I will be happy to write up such a policy if we think it would be good to have one on the books? > One thing that I have been pondering is resurrecting the old technique > of embedding version info into binaries so they can be examined to > discover which library versions were used in its build. This was > totally a thing when I was using rcs ($Id$ and ident(1) FTW) but that > particular technique is messy with distributed version control, and > may not be needed at all in a packaged distro environment. This may, > however, help ease the concerns many have around static linking of > libs. > > dt > > [0] https://blog.gopheracademy.com/advent-2016/saga-go-dependency-management/ > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
Excerpts from Dean Troyer's message of 2016-12-14 08:15:08 -0600: > On Wed, Dec 14, 2016 at 7:46 AM, Doug Hellmann wrote: > > FWIW, some of the deployment tool communities (ansible and puppet, > > I think) rely on git repos without external artifacts, and we're > > supporting them in our release tools today. I'm not sure how > > downstream packagers feel about the lack of external artifacts. > > I recall zigo working hard for the Debian packaging to come strictly > from Git repo tags but do not recall seeing opinions from other > packagers. > > > Were you thinking of files other than the tarball when you say > > "generated files"? > > Yes, in particular, files like AUTHORS and sample config files, which > IIRC are generated and included in tarballs. OK, those should be easy enough to deal with if we publish tarballs of golang source. Perhaps someone from one of our other projects that uses git tags as releases can comment on how they handle those things? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
On Wed, Dec 14, 2016 at 7:46 AM, Doug Hellmann wrote: > FWIW, some of the deployment tool communities (ansible and puppet, > I think) rely on git repos without external artifacts, and we're > supporting them in our release tools today. I'm not sure how > downstream packagers feel about the lack of external artifacts. I recall zigo working hard for the Debian packaging to come strictly from Git repo tags but do not recall seeing opinions from other packagers. > Were you thinking of files other than the tarball when you say > "generated files"? Yes, in particular, files like AUTHORS and sample config files, which IIRC are generated and included in tarballs. dt -- Dean Troyer dtro...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
On Wed, Dec 14, 2016 at 4:25 AM, Thierry Carrez wrote: > I don't think we need to stop producing and publishing source code > tarballs in the Go case, just because it's not the primary way people > consume the code. Publication of the source code is something we need to > do as part of the open source license we use, and some still consider > tarball publication to be a clearer form of "publication" than keeping a > git server up. I'll buy that. Overall, I don't think we are doing anything today that is incompatible with what I see golang needing. This is really a side-issue that I have heard you, and maybe others, ask about in general, so I included it to affirm the golang case. Making changes to the general case is separate in my mind, but this may be a good time to consider it in parallel since we're thinking about these things again in detail. > Beyond the source though, one question is whether we should build, sign > and distribute binary artifacts (compiled code), or if tagging a source > repo (and producing a source code tarball) is sufficient. And if we do > distribute binaries, would we only do that for some deliverables (like > the top ones that are supposed to be directly used by users) or for > everything ? I am not in favor of distributing binaries as a matter of course, although from a purely selfish point of view I want the ability to do that for a future CLI binary for Windows. That CLI doesn't exist yet, and I do not see another reason (yet?) to include this in the initial task list. dt -- Dean Troyer dtro...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
Excerpts from Dean Troyer's message of 2016-12-13 12:45:07 -0600: > Release Deliverables > > OpenStack still officially considers the tarballs generated during the > release rpocess to be our official deliverable. Many downstream > consumers, however, bypass those and go directly to the tagged > releases in the Git repos. The Golang community has no real culture > of using tarballs at all, given the 'go get' functionality bult in to > the tooling directly for checking out remote repos. One obvious > shortcoming in just defining the release to be Git repo tags is our > use of generated files. FWIW, some of the deployment tool communities (ansible and puppet, I think) rely on git repos without external artifacts, and we're supporting them in our release tools today. I'm not sure how downstream packagers feel about the lack of external artifacts. I think I would be satisfied to see tags, even if we do not build artifacts from them. We can still run other parts of the release process, such as building release notes and documentation with the version number. Were you thinking of files other than the tarball when you say "generated files"? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
[Sending again due to mail delivery issues.] Excerpts from Dean Troyer's message of 2016-12-13 12:45:07 -0600: > Release Deliverables > > OpenStack still officially considers the tarballs generated during the > release rpocess to be our official deliverable. Many downstream > consumers, however, bypass those and go directly to the tagged > releases in the Git repos. The Golang community has no real culture > of using tarballs at all, given the 'go get' functionality bult in to > the tooling directly for checking out remote repos. One obvious > shortcoming in just defining the release to be Git repo tags is our > use of generated files. FWIW, some of the deployment tool communities (ansible and puppet, I think) rely on git repos without external artifacts, and we're supporting them in our release tools today. I'm not sure how downstream packagers feel about the lack of external artifacts. I think I would be satisfied to see tags, even if we do not build artifacts from them. We can still run other parts of the release process, such as building release notes and documentation with the version number. Were you thinking of files other than the tarball when you say "generated files"? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
Monty Taylor wrote: > On 12/13/2016 12:45 PM, Dean Troyer wrote: >> Release Deliverables >> >> OpenStack still officially considers the tarballs generated during the >> release rpocess to be our official deliverable. Many downstream >> consumers, however, bypass those and go directly to the tagged >> releases in the Git repos. The Golang community has no real culture >> of using tarballs at all, given the 'go get' functionality bult in to >> the tooling directly for checking out remote repos. One obvious >> shortcoming in just defining the release to be Git repo tags is our >> use of generated files. > > Indeed. On the other hand - our use of git tags to drive our > deliverables (and pbr's use of them instead of content in a file in the > source repo) actually has us more closely aligned with this than other > projects might be. Yes, especially since we automated the tagging process (reducing the risk of a random tag being accidentally pushed to a repository), git tags *are* releases in openstack deliverables. I don't think we need to stop producing and publishing source code tarballs in the Go case, just because it's not the primary way people consume the code. Publication of the source code is something we need to do as part of the open source license we use, and some still consider tarball publication to be a clearer form of "publication" than keeping a git server up. Beyond the source though, one question is whether we should build, sign and distribute binary artifacts (compiled code), or if tagging a source repo (and producing a source code tarball) is sufficient. And if we do distribute binaries, would we only do that for some deliverables (like the top ones that are supposed to be directly used by users) or for everything ? >> If you have an interest in seeing Golang support implemented for >> OpenStack, please jump in here and help us nail down how to accomplish >> that "right". ++ -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
On Tue, Dec 13, 2016 at 4:26 PM, Jay Pipes wrote: > Not sure if it's already been brought up, but I really like govendor: Govendor and glide are the two most promising candidates at this point. I just read today's summary of the Package Management Committee [0] and that sounds _really_ promising, if further out than we can wait on, but there will be migration paths from many of the existing tools. Both govendor and glide have devs in the Advisory Group to the committee. > More than the decision between dependency management toolkits, though, is > the decision over whether to have *any* vendored source code in the primary > project's source tree itself (as opposed to only storing the > vendor.json/glide.yaml|lock file that lists dependent library versions and > locations). Please let's have a policy of keeping vendored source code *out* > of the primary project source tree. This is the intention and I see no reason why it is not achievable. It totally depends (ha!) on having better (any!) dependency management in place. One thing that I have been pondering is resurrecting the old technique of embedding version info into binaries so they can be examined to discover which library versions were used in its build. This was totally a thing when I was using rcs ($Id$ and ident(1) FTW) but that particular technique is messy with distributed version control, and may not be needed at all in a packaged distro environment. This may, however, help ease the concerns many have around static linking of libs. dt [0] https://blog.gopheracademy.com/advent-2016/saga-go-dependency-management/ -- Dean Troyer dtro...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
On 12/13/2016 03:17 PM, Monty Taylor wrote: On 12/13/2016 12:45 PM, Dean Troyer wrote: I am working on scoping the tasks required for the technical pieces of Golang adoption in OpenStack. This work has been informed somewhat by flaper87's reference doc proposal[0] for new language additions and is (mostly) compatible with it, pending that proposal's final approval by the TC. As a first step, I proposed a Common Testing Interface (CTI) document[1] for Go. There is, of course, much more than this, including a number of significant decisions that still need to be made. At this point I want to focus on identifying those decisions and other tasks rather than solving them immediately. Some of the previous work done around using Golang in OpenStack has been collected in an Etherpad[2] and summarized into major categories. I am confident that I have not found everything that has been done and would appreciate pointers to whatever resources you may have. Here are some highlights of areas that still require discussion and decisions: Common Libraries flaper87's reference doc touches on project requirements for new languages and Oslo-compatible implementations. At the time of this writing this is not yet finalized, with some discussion around how much to require, and if the first project to utilize a new language should shoulder all of the burden of also implementing those libraries. Dependency Management There are a number of tools available for Golang to manage dependencies. A number have been evaluated already with a couple of strong contenders identified. Once a tool is selected, process needs to be set up to track dependencies and versions tested. Figuring out how to get from that to pre-cached git repos in gate nodes is another important task. Actually cloning from github during CI jobs leads to wailing and gnashing of teeth pretty quickly. That said - godep and glide both lend themselves well towards doing such a thing. Not sure if it's already been brought up, but I really like govendor: https://github.com/kardianos/govendor for Golang dependency management. More than the decision between dependency management toolkits, though, is the decision over whether to have *any* vendored source code in the primary project's source tree itself (as opposed to only storing the vendor.json/glide.yaml|lock file that lists dependent library versions and locations). Please let's have a policy of keeping vendored source code *out* of the primary project source tree. Perhaps this is just me, but honestly, I don't see why some in the Golang community seem to just want to throw away decades of best practices from the development community that advises not to bundle all your dependent library source code in your own project's source tree. Basically... *don't mix your build artifacts with your source files*. Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Golang technical requirements
On 12/13/2016 12:45 PM, Dean Troyer wrote:
> I am working on scoping the tasks required for the technical pieces of
> Golang adoption in OpenStack. This work has been informed somewhat by
> flaper87's reference doc proposal[0] for new language additions and is
> (mostly) compatible with it, pending that proposal's final approval by
> the TC.
>
> As a first step, I proposed a Common Testing Interface (CTI)
> document[1] for Go. There is, of course, much more than this,
> including a number of significant decisions that still need to be
> made. At this point I want to focus on identifying those decisions
> and other tasks rather than solving them immediately.
>
> Some of the previous work done around using Golang in OpenStack has
> been collected in an Etherpad[2] and summarized into major categories.
> I am confident that I have not found everything that has been done and
> would appreciate pointers to whatever resources you may have. Here
> are some highlights of areas that still require discussion and
> decisions:
>
>
> Common Libraries
>
> flaper87's reference doc touches on project requirements for new
> languages and Oslo-compatible implementations. At the time of this
> writing this is not yet finalized, with some discussion around how
> much to require, and if the first project to utilize a new language
> should shoulder all of the burden of also implementing those
> libraries.
>
>
> Dependency Management
>
> There are a number of tools available for Golang to manage
> dependencies. A number have been evaluated already with a couple of
> strong contenders identified. Once a tool is selected, process needs
> to be set up to track dependencies and versions tested.
Figuring out how to get from that to pre-cached git repos in gate nodes
is another important task. Actually cloning from github during CI jobs
leads to wailing and gnashing of teeth pretty quickly. That said - godep
and glide both lend themselves well towards doing such a thing.
We also need to rework how zuul clones git repos a little bit so that
intra-OpenStack cross-repo dependencies work properly. In short, we
should have zuul clone to $BASE/src/git.openstack.org/openstack/nova
instead of $BASE/openstack/nova so that we can just point $GOPATH at
$BASE and have things just work. That obviously involves a change to
zuul to do the cloning, and to devstack to start grokking another level
of source repo organization. On the plus-side though, I think it'll be
nicer even for non-go related things, so is well worth doing.
Finally- (and this one is easy) - we need to put a pointer files at the
root of git.o.o and review.o.o to mark git.o.o/${repo}.git as the
canonical location of repositories.
https://golang.org/cmd/go/#hdr-Relative_import_paths
has instructions on this.
> Release Deliverables
>
> OpenStack still officially considers the tarballs generated during the
> release rpocess to be our official deliverable. Many downstream
> consumers, however, bypass those and go directly to the tagged
> releases in the Git repos. The Golang community has no real culture
> of using tarballs at all, given the 'go get' functionality bult in to
> the tooling directly for checking out remote repos. One obvious
> shortcoming in just defining the release to be Git repo tags is our
> use of generated files.
Indeed. On the other hand - our use of git tags to drive our
deliverables (and pbr's use of them instead of content in a file in the
source repo) actually has us more closely aligned with this than other
projects might be.
>
> If you have an interest in seeing Golang support implemented for
> OpenStack, please jump in here and help us nail down how to accomplish
> that "right".
>
> dt
>
> [0] https://review.openstack.org/#/c/398875/
> [1] https://review.openstack.org/410355
> [2] https://etherpad.openstack.org/p/golang-infra-work-plan
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Golang technical requirements
I am working on scoping the tasks required for the technical pieces of Golang adoption in OpenStack. This work has been informed somewhat by flaper87's reference doc proposal[0] for new language additions and is (mostly) compatible with it, pending that proposal's final approval by the TC. As a first step, I proposed a Common Testing Interface (CTI) document[1] for Go. There is, of course, much more than this, including a number of significant decisions that still need to be made. At this point I want to focus on identifying those decisions and other tasks rather than solving them immediately. Some of the previous work done around using Golang in OpenStack has been collected in an Etherpad[2] and summarized into major categories. I am confident that I have not found everything that has been done and would appreciate pointers to whatever resources you may have. Here are some highlights of areas that still require discussion and decisions: Common Libraries flaper87's reference doc touches on project requirements for new languages and Oslo-compatible implementations. At the time of this writing this is not yet finalized, with some discussion around how much to require, and if the first project to utilize a new language should shoulder all of the burden of also implementing those libraries. Dependency Management There are a number of tools available for Golang to manage dependencies. A number have been evaluated already with a couple of strong contenders identified. Once a tool is selected, process needs to be set up to track dependencies and versions tested. Release Deliverables OpenStack still officially considers the tarballs generated during the release rpocess to be our official deliverable. Many downstream consumers, however, bypass those and go directly to the tagged releases in the Git repos. The Golang community has no real culture of using tarballs at all, given the 'go get' functionality bult in to the tooling directly for checking out remote repos. One obvious shortcoming in just defining the release to be Git repo tags is our use of generated files. If you have an interest in seeing Golang support implemented for OpenStack, please jump in here and help us nail down how to accomplish that "right". dt [0] https://review.openstack.org/#/c/398875/ [1] https://review.openstack.org/410355 [2] https://etherpad.openstack.org/p/golang-infra-work-plan -- Dean Troyer dtro...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
