[go-nuts] Re: Using Go in a locked down SOC2 environment (dependency management hell)

[Diego Medina]

> All of our build servers are locked down and cannot reach public 
Internet, only our local library mirror.

With this in mind, you could use gvt https://github.com/FiloSottile/gvt

all your devs should use it to fetch dependencies, yes, they would use open 
internet, but only while they get their app working, at commit time, you 
only commit the manifest file, a sample file looks like:

{
"version": 0,
"dependencies": [
{
"importpath": "github.com/dgryski/dkeyczar",
"repository": "https://github.com/dgryski/dkeyczar;,
"vcs": "git",
"revision": "05c8ea381e02781927d6c633076910b86c6ce770",
"branch": "master",
"notests": true
},
{
"importpath": "github.com/golang/glog",
"repository": "https://github.com/golang/glog;,
"vcs": "git",
"revision": "23def4e6c14b4da8ac2ed8007337bc5eb5007998",
"branch": "master",
"notests": true
}
]
}


This is a simple json file.

You then need to provide your layers with a way to have in a central place, 
a list of the approved Go dependencies so you can pick these fields:

importpath": "github.com/golang/glog",
"repository": "https://github.com/golang/glog;,
"vcs": "git",
"revision": "23def4e6c14b4da8ac2ed8007337bc5eb5007998",
"branch": "master",

You will also create a builder $GOPATH with all these approved dependencies 
(like someone else already suggested)


Once you have that, the build server would first run your tool to make sure 
that every single entry on the manifest file is in the lawyer approved 
list, down to the commit, if that passes, then you run gvt restore, which 
will look at the manifest file and download all the dependencies into the 
vendor folder of the current project.

Oh, and just in case you have more of those non law abiding devs, as part 
of the build, you can first make sure that the vendor folder is empty and 
only has the manifest file, so nobody can sneak in a committed dependency.

gvt restore would use the approved GOPATH so it will not need internet 
access.

I think this covers all the restrictions you need.

Hope it helps.

Diego



On Friday, January 6, 2017 at 11:35:41 AM UTC-5, Jacek Furmankiewicz wrote:
>
> Hi everyone,
>
> We are operating in a SOC2 environment, which our customers demanded as we 
> host their systems and their data.
> It's a common requirement for many companies in a cloud environment.
>
> One of the key requirements of SOC2 is that *all* external 
> libraries/depdencies are mirrored internally and 
> *NOT*fetched directly from public Internet during the code building 
> process.
>
> With our Java apps, this is simple. We have an internal Artifactory 
> instance and we mirror all the Java libraries from the Maven Central 
> repository there
> (after each and every one of them goes through legal license review, to 
> exclude GPL, etc).
>
> All of our build servers are locked down and cannot reach public Internet, 
> only our local library mirror.
> All of our Gradle build scripts are locked down to allow fetching 
> dependencies from our local Maven mirror only.
>
> As you can imagine, this is anathema to how entire dependency management 
> works in Go (and all the related tools, like go get, etc).
>
> Even if we mirrored Go git repos for the libraries we want in our internal 
> Stash repository,
> pretty much all the current Go build tools (e,g. Glide) would still go to 
> the public internet to look for dependencies of any library.
>
> e.g.:
>
> https://github.com/Masterminds/glide/issues/729
>
>
> Just wanted to hear if there are any Go shops out there that operate in a 
> SOC2 environment and what combination of tools / procedures do you use.
>
> We are very interested in Go adoption for some of our real-time 
> server-side applications, but the way dependency management works in Go
> is a total showstopper due to the stringent security requirements of SOC2.
>
> I would greatly appreciate any input, comments, suggestions from the Go 
> community.
>
> Much appreciated
> Jacek
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Using Go in a locked down SOC2 environment (dependency management hell)

[Wojciech S. Czarnecki]

Dnia 2017-01-06, o godz. 14:19:28
Jacek Furmankiewicz  napisał(a):

> What happens when you have multiple projects, all with different versions
> of the same library (  some projects are newer, some older, some more
> maintained than others, etc).
> 
> Would you need a different GOPATH per project?

Jacek, take a look at gb [1] and vendoring [2].

Though for your needs there is a much simpler solution:

This is Go. It produces a single binary for deployment. For
SOC2 compliance you will use (you already use) a builder.
This is a single place where you exercise your control over what
can and what can not go into the executable. Make your lawyers
responsible for single 'official' gopath @builder.

Then tell your devs they may not use go get by themselves, everything
they need they should pull from the GOPATH@builder. If they won't abide
to this, their source simply will not compile @builder on commit hook;
hence it will not get into your controlled repo, ever. Voilà.

For the tagged production versions I would suggest to:
- make separate "version" branch
- copy used libs to the vendor/ [2] 
- optionally copy whole environment for reproductible build
All above may happen automagically on post-receive hook for a tag.

This way you will get CI on a single (dev/prod) repo and fully
reproductible [3] builds. All with official tools only and under
your tight control. 

[1] https://getgb.io/
[2] https://blog.gopheracademy.com/advent-2015/vendor-folder/
[3] as far as go compiler allows for https://github.com/golang/go/issues/16860

Hpoe this helps,

-- 
Wojciech S. Czarnecki
   ^oo^ OHIR-RIPE

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Using Go in a locked down SOC2 environment (dependency management hell)

[Justin Israel]

On Sat, Jan 7, 2017 at 9:19 AM Jacek Furmankiewicz 
wrote:

> What happens when you have multiple projects, all with different versions
> of the same library (  some projects are newer, some older, some more
> maintained than others, etc).
>
> Would you need a different GOPATH per project?
>

That in particular is a problem that vendoring is meant to address, as well
as the dependency/packaging group that is currently working to solve this
in an official manner.

Yea, if you are basing it on the filesystem, you would have explicit
checkouts, which wont work between different projects. You could use a
combination of that and a vendoring solution but not check the vendor
location into the repo. Then you would end up with a manifest (i.e using
Glide) which specifies the versions, and be able to run an update to pull
down your specific versions into the untracked vendor location.


>
>
> On Fri, Jan 6, 2017 at 2:15 PM, Justin Israel 
> wrote:
>
>
>
> On Sat, Jan 7, 2017, 9:04 AM Jacek Furmankiewicz 
> wrote:
>
> Hi Daniel.
>
> I participated in the great Go survey on dependency management a while
> back and raised these concerns there.
> I read the summary of that once it was completed and was kinda
> disappointed to see that none of this points seem to be getting addressed
> or even acknowledged as a problem.
>
>
> https://blog.gopheracademy.com/advent-2016/saga-go-dependency-management/?utm_source=golangweekly_medium=email
>
> Sure, govendor can force using a private repo for a library. Not it does
> not force it for all its dependencies, so you're back to the same clunky
> management of every single library with a custom path/repo for every single
> one of them.
>
> The core issue is Go's insistence on using the VCS path of a library as
> its import path, unlike any other language out there.
> There is no way to create a central repo with all packages and point to
> it. Even Rust got this right on day one, with crates.io.
>
>
> Go, as a language, does not insist on the import path being the vcs path.
> It is only a string, for flexibility reasons. It is the choice of the go
> tool to treat the import path as a vcs path. Other tooling can do what it
> wants.
>
> Would the multiple GOPATH setup work for you? This means you are not
> putting 3rd party code into the repo or giving wide open access to get any
> library you want. You could only import libraries that exist on the managed
> GOPATH. Then it just becomes a developer setup requirement to have the
> appropriate GOPATH set in the environment.
>
>
> I understand this was one of the opinionated decisions the Go creators
> made in the early days,
> but the end result is that we simply cannot even touch Go as a potential
> language.
>
> I don't think this is a side effect they envisioned when they made this
> decision. :-(
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Using Go in a locked down SOC2 environment (dependency management hell)

[Jacek Furmankiewicz]

What happens when you have multiple projects, all with different versions
of the same library (  some projects are newer, some older, some more
maintained than others, etc).

Would you need a different GOPATH per project?


On Fri, Jan 6, 2017 at 2:15 PM, Justin Israel 
wrote:

>
>
> On Sat, Jan 7, 2017, 9:04 AM Jacek Furmankiewicz 
> wrote:
>
>> Hi Daniel.
>>
>> I participated in the great Go survey on dependency management a while
>> back and raised these concerns there.
>> I read the summary of that once it was completed and was kinda
>> disappointed to see that none of this points seem to be getting addressed
>> or even acknowledged as a problem.
>>
>> https://blog.gopheracademy.com/advent-2016/saga-go-
>> dependency-management/?utm_source=golangweekly_medium=email
>>
>> Sure, govendor can force using a private repo for a library. Not it does
>> not force it for all its dependencies, so you're back to the same clunky
>> management of every single library with a custom path/repo for every single
>> one of them.
>>
>> The core issue is Go's insistence on using the VCS path of a library as
>> its import path, unlike any other language out there.
>> There is no way to create a central repo with all packages and point to
>> it. Even Rust got this right on day one, with crates.io.
>>
>
> Go, as a language, does not insist on the import path being the vcs path.
> It is only a string, for flexibility reasons. It is the choice of the go
> tool to treat the import path as a vcs path. Other tooling can do what it
> wants.
>
> Would the multiple GOPATH setup work for you? This means you are not
> putting 3rd party code into the repo or giving wide open access to get any
> library you want. You could only import libraries that exist on the managed
> GOPATH. Then it just becomes a developer setup requirement to have the
> appropriate GOPATH set in the environment.
>
>
>> I understand this was one of the opinionated decisions the Go creators
>> made in the early days,
>> but the end result is that we simply cannot even touch Go as a potential
>> language.
>>
>> I don't think this is a side effect they envisioned when they made this
>> decision. :-(
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Using Go in a locked down SOC2 environment (dependency management hell)

[Jacek Furmankiewicz]

Hi Daniel. 

I participated in the great Go survey on dependency management a while back 
and raised these concerns there.
I read the summary of that once it was completed and was kinda disappointed 
to see that none of this points seem to be getting addressed or even 
acknowledged as a problem.

https://blog.gopheracademy.com/advent-2016/saga-go-dependency-management/?utm_source=golangweekly_medium=email

Sure, govendor can force using a private repo for a library. Not it does 
not force it for all its dependencies, so you're back to the same clunky 
management of every single library with a custom path/repo for every single 
one of them.

The core issue is Go's insistence on using the VCS path of a library as its 
import path, unlike any other language out there.
There is no way to create a central repo with all packages and point to it. 
Even Rust got this right on day one, with crates.io.

I understand this was one of the opinionated decisions the Go creators made 
in the early days,
but the end result is that we simply cannot even touch Go as a potential 
language.

I don't think this is a side effect they envisioned when they made this 
decision. :-( 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Using Go in a locked down SOC2 environment (dependency management hell)

[Tamás Gulácsi]

2017. január 6., péntek 17:35:41 UTC+1 időpontban Jacek Furmankiewicz a 
következőt írta:
>
> Hi everyone,
>
> We are operating in a SOC2 environment, which our customers demanded as we 
> host their systems and their data.
> It's a common requirement for many companies in a cloud environment.
>
> One of the key requirements of SOC2 is that *all* external 
> libraries/depdencies are mirrored internally and 
> *NOT*fetched directly from public Internet during the code building 
> process.
>
>
$ go get -x -v google.golang.org/grpc 
Fetching 
https://google.golang.org/grpc?go-get=1 
 
Parsing meta tags from https://google.golang.org/grpc?go-get=1 (status code 
200) 
get "google.golang.org/grpc": found meta tag 
main.metaImport{Prefix:"google.golang.org/grpc", VCS:"git", 
RepoRoot:"https://github.com/grpc/grpc-go"} a
t https://google.golang.org/grpc?go-get=1


So maybe write a small HTTP proxy for HTTP_PROXY, which "ticks in" for 
?go-get=1 query string, and provides a proper document with proper meta 
tags?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.