Re: [gentoo-dev] network sandbox challenge

[Robin H. Johnson]

On Wed, Apr 01, 2020 at 11:30:00PM +0100, Samuel Bernardo wrote:
> Hi Robin,
> 
> On 4/1/20 11:07 PM, Robin H. Johnson wrote:
> >>> # I am considering removing this and just hard coding mirror://goproxy
> >>> # below, so please do not rely on it.
> >>> : "${_GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}"|
> >> So, go-module.eclass provides a good base to follow SRP pattern (single
> >> responsibility principle). Looking into that effort I would copy
> >> go-module.eclass to my overlay eclass directory and adapt it to use my
> >> mirror type.
> >>
> >> I would like to use directly Gentoo maintained go-module.eclass, but for
> >> that I would like to have access to set my own mirror type.
> >>
> >> I also ask you to update documentation since there is some details to
> >> use EGO_SUM, such as you stated in comment[4].
> > It's not clear why you need to modify the mirror behavior at all; maybe
> > you covered that elsewhere in this overly long thread.
> 
> Remembering that I'm in overlay context, lets consider the use case with
> a private repository with restricted access from where I need to get
> some go module. I need to use an alternative mirror not listed in
> default goproxy mirror type.
Ok, so what you're REALLY asking for here, is a cleaner way to specify
that a go module that exists OUTSIDE of the upstream Goproxy ecosystem.

The eclass as it stands was written as a way to CONSUME the upstream
Goproxy ecosystem efficiently, without producing a bundling mess.

This is not entirely a Gentoo packaging problem, but also intersects
upstream Go packaging, specifically that outside of the ebuild you have
to take special measures to cause 'go mod' to fetch from your
alternative mirror that contains private modules. Upstream also doesn't
have a good handle on how to solve this problem for ALL of the edge
cases (see GOPRIVATE and interactions with the GOSUMDB).

Your package that consumes that go module PROBABLY also consumes many
public go modules, so you don't want to modify or mirror every single
entry: just the private ones.

I propose the following rough improvement to the go-module.eclass for
EGO_SUM usage in the above case.

EGO_SUM=(
  "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod"
  "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59"
)
EGO_SUM_WITH_BASE=(
  "mirror://private/ internal.com/foo/disco v1.2.3/go.mod"
  "mirror://private/ internal.com/foo/disco v1.2.3"
  "https://site2/ mycompany.com/bar/dance v2.3.4/go.mod"
  "https://site2/ mycompany.com/bar/dance v2.3.4/"
)

EGO_SUM can be trivially transformed to EGO_SUM_WITH_BASE, by adding the
mirror://goproxy/ slug as a first argument.

The implementation of go-module_set_globals remains almost identical,
just using the new first argument rather than the
${_GOMODULE_GOPROXY_BASEURI} variable.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: PGP signature

Re: [gentoo-dev] network sandbox challenge

[Robin H. Johnson]

On Wed, Apr 01, 2020 at 04:14:48PM -0400, Michael Orlitzky wrote:
> On 4/1/20 4:03 PM, Samuel Bernardo wrote:
> > 
> > Couldn't security issue in a Go library be solved with keyword mask and
> > announce in portage? 
> 
> If there's an ebuild for the library, then yeah, you've got the right
> idea. But with the Go eclasses, there are no ebuilds for any of the
> dependencies.
The problem goes deeper than that, and is more of an upstream concern
than a Gentoo concern: because the Go module ecosystem pins exact
versions, not version ranges, and this is a crucial part of reproducible
builds.

Let the library be "L", with versions 1,2.
The vulnerable version is L-1, and L-2 contains the fix.
Let the package that uses the library be "P", with version 1 only.

If you wanted to USE L-2 in P-1, you generally must make modifications
to the consumers of that library. At the very least you have to modify
the go.mod file to use the new version. To satisfy the package
checksums/security in the Go ecosystem, you ALSO need to update the
go.sum file.

If the vulnerable library is a transitive dependency (e.g. indirect via
another library), then you ALSO need to update that other consumer.

At the point that there IS a reliable way to get lists of vulnerable
versions of Go modules, including the horrible timestamped v0.0.0
versions, the EGO_SUM work does contain enough data to identify ALL
vulnerable outputs on your system. That listing isn't available yet, due
to upstream working on it still:
https://github.com/golang/go/issues/24031

That listing would be transformed into a GLSA input criterion, to
identify vulnerable Gentoo packages (at which point upstream Golang has
hopefully also provided a cleaner way to bump/patch the dependency in
the scope of reproducible versions).

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: PGP signature

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Robin,

On 4/1/20 11:07 PM, Robin H. Johnson wrote:
>>> # I am considering removing this and just hard coding mirror://goproxy
>>> # below, so please do not rely on it.
>>> : "${_GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}"|
>> So, go-module.eclass provides a good base to follow SRP pattern (single
>> responsibility principle). Looking into that effort I would copy
>> go-module.eclass to my overlay eclass directory and adapt it to use my
>> mirror type.
>>
>> I would like to use directly Gentoo maintained go-module.eclass, but for
>> that I would like to have access to set my own mirror type.
>>
>> I also ask you to update documentation since there is some details to
>> use EGO_SUM, such as you stated in comment[4].
> It's not clear why you need to modify the mirror behavior at all; maybe
> you covered that elsewhere in this overly long thread.

Remembering that I'm in overlay context, lets consider the use case with
a private repository with restricted access from where I need to get
some go module. I need to use an alternative mirror not listed in
default goproxy mirror type.

Ok, you can say that since is not public what is the point of creating
an ebuild? Well the ebuild (so easy to do) turns smooth the maintenance,
the installation into my environment, to reproduce it later to install
in real infrastructure and also provide me a tar.gz or rpm package using
the awesome power of Gentoo!

Btw, my workspace have two overlays for now:

- local private overlay
- public overlay  (where I share what can be shared)

Thanks for your advices,

Samuel



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Robin H. Johnson]

On Wed, Apr 01, 2020 at 12:50:56PM +0100, Samuel Bernardo wrote:
> Hi Robin,
> 
> On 4/1/20 6:36 AM, Robin H. Johnson wrote:
> > Samuel:
> > I already proved that using go-module.eclass EGO_SUM it will NOT use Git
> > repositories, and all of the fetching will happen long before
> > src_unpack. Why do you persist with your statement to the contrary?
> 
> Sorry my misunderstanding, but I didn't get it right with what you
> mentioned to me before:
> 
> > Have you looked at the EGO_SUM in go-module? This already covers ANY go
> > package that uses go.mod + go.sum, in a fully reproducible way that does
> > not break network sandbox.
> I looked into the eclass documentation and I only found EGO_VENDOR[1].
> This seems so expensive since I need to parse all go.mod or go.sum files
> that I concluded would be better to use a tar.gz with all dependencies
> (remember that my use case is with my personal overlay).
Thanks, it seems the eclass HTML documentation is not updating
correctly (it's showing the content of the eclass-manpages-20200213
version right now, despite the "Updated: Apr 2020" text). I've pushed a
change and the corrected documentation is now visible.
Thank you for bringing this to our attention.

> Looking deeply into the source code I could find more details about
> EGO_SUM that only needs to set the urls to go.mod or/and go.sum files
> present in the repository. This is useful, but only to official gentoo
> developers, since if I want to use my own mirror type, for instance
> mirror://gooverlay, I wouldn't have that option[3]:
You don't need to override _GOMODULE_GOPROXY_BASEURI at all.

_GOMODULE_GOPROXY_BASEURI (with the default of mirror://goproxy/)
expands to the UPSTREAM locations that have the Go modules available.

It expects those mirrors to have the layout of files used by upstream.

> > # I am considering removing this and just hard coding mirror://goproxy
> > # below, so please do not rely on it.
> > : "${_GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}"|
> So, go-module.eclass provides a good base to follow SRP pattern (single
> responsibility principle). Looking into that effort I would copy
> go-module.eclass to my overlay eclass directory and adapt it to use my
> mirror type.
> 
> I would like to use directly Gentoo maintained go-module.eclass, but for
> that I would like to have access to set my own mirror type.
> 
> I also ask you to update documentation since there is some details to
> use EGO_SUM, such as you stated in comment[4].
It's not clear why you need to modify the mirror behavior at all; maybe
you covered that elsewhere in this overly long thread.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: PGP signature

Re: [gentoo-dev] network sandbox challenge

[Michael Orlitzky]

On 4/1/20 4:03 PM, Samuel Bernardo wrote:
> 
> Couldn't security issue in a Go library be solved with keyword mask and
> announce in portage? 

If there's an ebuild for the library, then yeah, you've got the right
idea. But with the Go eclasses, there are no ebuilds for any of the
dependencies.

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Michael,

On 4/1/20 6:01 PM, Michael Orlitzky wrote:
> On 4/1/20 11:49 AM, Alec Warner wrote:
>> Imagine a common dep (CommonFoo-x-y-z)
>> has a security problem, so we must upgrade to CommonFoo-y-z. In the
>> scenario where CommonFoo is a dynamically linked package we can
>> recompile it once[4] and new consumers will just use the new dynamic
>> shared object. In a bundling scenario, we will be forced to rebuild[5]
>> all consumers. 
> This is highly euphemistic. What actually happens is: someone discovers
> a security issue in a Go library. That library is not "in" Gentoo,
> because it only ever appears in a string inside of another ebuild that
> bundles everything. Thereafter, a whole lot of nothing happens. Users
> remain vulnerable "forever," until some other unrelated event causes
> both the ebuild and its dependency to be updated.

Couldn't security issue in a Go library be solved with keyword mask and
announce in portage?

The vulnerability care is not only related to the distro, but also to
the upstream. Gentoo already provides the option to downgrade to a
previous version (if that is an answer for the issue). Imagine Arch
distro where that is not an option or Debian Stable that is stuck in a
version?
I see so more troubles in other distros than Gentoo.

The choice is the responsibility of the end user and distro maintainers
don't have to provide every software. Providing the eclasses that allows
to produce the ebuild is a good option for those who need some software,
simplifying the development work. Overlaying is the solution, I think...

Best,

Samuel



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Alec,

Really great reading about packaging and the current challenges. This
would be great to be posted in Gentoo blog!

---

To all community:

I would like to commend the know how of Gentoo community and their
principles.

I would like to evidence relatively to other distributions some points
that IMHO turns Gentoo the greatest (win in all points):
- QA
- freedom
- tooling
- know how
- velocity
- security
- slotting
- learning
- sharing
- straightforwardness
- objectivity
- mirroring
- package system
- **extensibility**

Overlays, ebuild and profile definitions in Gentoo extends the freedom
of the ingenious distro ever made. There are no barriers, so please
don't forget the overlays when developing Gentoo code base that would
help everyone to extend their needs and follow the good QA practices.

There is nothing like Gentoo!

Thank you all that contribute to Gentoo!


On 4/1/20 4:49 PM, Alec Warner wrote:
>
> On Wed, Apr 1, 2020 at 5:14 AM Samuel Bernardo
> mailto:samuelbernardo.m...@gmail.com>>
> wrote:
>
> Forgive my noobishness in this matter that let Alec to comment
> over my own statement.
>
> Alec pointed out some very important issues in go development that
> break copyright infringement and security vulnerabilities, but I'm
> sure that is not related to the good work done in go-module.eclass
> to surpass all go mess. npm is worst and I take from go-module as
> a good pattern to apply also into there.
>
> I am antarus, not mjo (but more on that below!) I don't believe
> bundling presents many challenges with regards to copyright
> infringement. As a package maintainer you should know the licenses
> used in your packages. You are required to reflect any licenses used
> in the LICENSE ebuild variable. Obviously this becomes more work if
> you are using a bundle due to the fact that bundling will include more
> code. In the golang ecosystem there is a tool to help maintainers do
> this (https://packages.gentoo.org/packages/dev-go/golicense). I get
> that with bundling we cannot share the work from previous packages
> because packages are not shared in a bundled environment but I expect
> the golicense tool to have good coverage in practice. If the tool does
> the work, sharing the work becomes moot.
>
> I think licensing can be more challenging in other bundling scenarios
> where tooling is not provided; but note that this is not significantly
> different from the unbundled scenario in terms of license discovery.
> If I am packaging a new program (A) and it depends on (B,C,D) I have
> two options. I can either package [A,B,C,D] (normal gentoo way) or I
> can package [A] (with B,C,D bundled). The intersection of the LICENSE
> variables is the same effort for both here. The benefit of the
> multiple packages is that future users of B,C,D can re-use the license
> discovery work and that isn't nothing.
>
> Going back to my overlay use case, will go-modules download all
> modules to distfiles directory? The naming convention will assure
> that there will be no modules repetition?
>
> What about eclean-dist, will it work as expected for those modules
> dependencies?
>
> I think some of this answers would worth mention in documentation.
>
> Sorry for anything I wrongly stated and thank you very much for
> your help,
>
> Samuel
>
> I've chosen this part to write my treatise on packaging, but rest
> assured it's mostly intended as a response to mgorny and mjo; not
> specifically in response to you.
>  
> The very long answer is that Gentoo was designed around a paradigm of
> programs written primarily in C. In C programs you have the ability to
> link to libraries which offer APIs and in the ideal case, each API is
> offered via a unique SONAME[0]. Upstream packages were written and
> built in this way (with dynamic linking). So in the case of package A,
> that uses libraries B, C, and D; the result in many distributions is 4
> packages (A,B,C,D) and users who want A will get B, C, and D
> installed. This in fact was a major selling point of package managers
> at the time because finding these dependencies by hand and building
> and merging them all was painful.
>
> Many applications break this trend; I don't think golang or nodejs are
> particularly new (python and ruby have had (pip, venv) and rubygems[1]
> for years, for example, which are similar bundling paradigms.) The
> struggle as packagers and distribution managers is when upstream
> decides "my software should be installed via a bundling solution
> (golang, node, pip, rubygems, and so on)" we are left to decide both
> whether to map this to the ebuild paradigm (no bundling of
> dependencies) or omit ebuilds entirely. In the former case we are
> often left working at odds with upstream (who are confused by our
> decomposition of their application) and in the latter case, users
> often use the bundle anyway (e.g. they install the packages by hand or
> use the ruby gems or whatever.) 

Re: [gentoo-dev] network sandbox challenge

[Michael Orlitzky]

On 4/1/20 11:49 AM, Alec Warner wrote:
> Imagine a common dep (CommonFoo-x-y-z)
> has a security problem, so we must upgrade to CommonFoo-y-z. In the
> scenario where CommonFoo is a dynamically linked package we can
> recompile it once[4] and new consumers will just use the new dynamic
> shared object. In a bundling scenario, we will be forced to rebuild[5]
> all consumers. 

This is highly euphemistic. What actually happens is: someone discovers
a security issue in a Go library. That library is not "in" Gentoo,
because it only ever appears in a string inside of another ebuild that
bundles everything. Thereafter, a whole lot of nothing happens. Users
remain vulnerable "forever," until some other unrelated event causes
both the ebuild and its dependency to be updated.

Your license scenario is also wishful thinking. All of the LICENSE bugs
reported when this eclass was proposed have been sitting open for six
months. As soon as the eclass was committed, that shit went out the door
and the developers moved on to make more money at our expense. You got
scammed.

Re: [gentoo-dev] network sandbox challenge

[Alec Warner]

On Wed, Apr 1, 2020 at 5:14 AM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi Robin,
> On 4/1/20 6:36 AM, Robin H. Johnson wrote:
>
> Normally we don't bundle dependencies, avoiding that problem entirely.
> The Go eclasses however are badly designed, committed against protest by
> paid corporate interests, and serve only to facilitate large-scale
> copyright infringement and security vulnerabilities. If you're looking
> for a consistent explanation of how they're supposed to work with the
> rest of Gentoo, you won't find one.
>
> mjo: Can you please substantiate your claims?
>
> It would have been nice to have heard your concerns during February, any
> of one the three times that William and I posted the go-module.eclass
> EGO_SUM development work for review on this mailing list. I don't see a
> single email from you during that entire period.
>
> The EGO_SUM support explicitly ensured that upstream distfiles (for each
> dependency) remained absolutely as upstream provided them, without
> merging the distfiles together or altering their content in way (I admit
> that the exact naming of the distfiles changed, because it was terrible,
> v0.0.0-20190311183353-d8887717615a.zip for example).
>
> Forgive my noobishness in this matter that let Alec to comment over my own
> statement.
>
> Alec pointed out some very important issues in go development that break
> copyright infringement and security vulnerabilities, but I'm sure that is
> not related to the good work done in go-module.eclass to surpass all go
> mess. npm is worst and I take from go-module as a good pattern to apply
> also into there.
>
I am antarus, not mjo (but more on that below!) I don't believe bundling
presents many challenges with regards to copyright infringement. As a
package maintainer you should know the licenses used in your packages. You
are required to reflect any licenses used in the LICENSE ebuild variable.
Obviously this becomes more work if you are using a bundle due to the fact
that bundling will include more code. In the golang ecosystem there is a
tool to help maintainers do this (
https://packages.gentoo.org/packages/dev-go/golicense). I get that with
bundling we cannot share the work from previous packages because packages
are not shared in a bundled environment but I expect the golicense tool to
have good coverage in practice. If the tool does the work, sharing the work
becomes moot.

I think licensing can be more challenging in other bundling scenarios where
tooling is not provided; but note that this is not significantly different
from the unbundled scenario in terms of license discovery. If I am
packaging a new program (A) and it depends on (B,C,D) I have two options. I
can either package [A,B,C,D] (normal gentoo way) or I can package [A] (with
B,C,D bundled). The intersection of the LICENSE variables is the same
effort for both here. The benefit of the multiple packages is that future
users of B,C,D can re-use the license discovery work and that isn't nothing.

> Going back to my overlay use case, will go-modules download all modules to
> distfiles directory? The naming convention will assure that there will be
> no modules repetition?
>
What about eclean-dist, will it work as expected for those modules
> dependencies?
>
> I think some of this answers would worth mention in documentation.
>
> Sorry for anything I wrongly stated and thank you very much for your help,
>
> Samuel
>
I've chosen this part to write my treatise on packaging, but rest assured
it's mostly intended as a response to mgorny and mjo; not specifically in
response to you.

The very long answer is that Gentoo was designed around a paradigm of
programs written primarily in C. In C programs you have the ability to link
to libraries which offer APIs and in the ideal case, each API is offered
via a unique SONAME[0]. Upstream packages were written and built in this
way (with dynamic linking). So in the case of package A, that uses
libraries B, C, and D; the result in many distributions is 4 packages
(A,B,C,D) and users who want A will get B, C, and D installed. This in fact
was a major selling point of package managers at the time because finding
these dependencies by hand and building and merging them all was painful.

Many applications break this trend; I don't think golang or nodejs are
particularly new (python and ruby have had (pip, venv) and rubygems[1] for
years, for example, which are similar bundling paradigms.) The struggle as
packagers and distribution managers is when upstream decides "my software
should be installed via a bundling solution (golang, node, pip, rubygems,
and so on)" we are left to decide both whether to map this to the ebuild
paradigm (no bundling of dependencies) or omit ebuilds entirely. In the
former case we are often left working at odds with upstream (who are
confused by our decomposition of their application) and in the latter case,
users often use the bundle anyway (e.g. they install the packages by hand
or 

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Robin,

On 4/1/20 6:36 AM, Robin H. Johnson wrote:
>> Normally we don't bundle dependencies, avoiding that problem entirely.
>> The Go eclasses however are badly designed, committed against protest by
>> paid corporate interests, and serve only to facilitate large-scale
>> copyright infringement and security vulnerabilities. If you're looking
>> for a consistent explanation of how they're supposed to work with the
>> rest of Gentoo, you won't find one.
> mjo: Can you please substantiate your claims? 
>
> It would have been nice to have heard your concerns during February, any
> of one the three times that William and I posted the go-module.eclass
> EGO_SUM development work for review on this mailing list. I don't see a
> single email from you during that entire period.
>
> The EGO_SUM support explicitly ensured that upstream distfiles (for each
> dependency) remained absolutely as upstream provided them, without
> merging the distfiles together or altering their content in way (I admit
> that the exact naming of the distfiles changed, because it was terrible,
> v0.0.0-20190311183353-d8887717615a.zip for example).

Forgive my noobishness in this matter that let Alec to comment over my
own statement.

Alec pointed out some very important issues in go development that break
copyright infringement and security vulnerabilities, but I'm sure that
is not related to the good work done in go-module.eclass to surpass all
go mess. npm is worst and I take from go-module as a good pattern to
apply also into there.

Going back to my overlay use case, will go-modules download all modules
to distfiles directory? The naming convention will assure that there
will be no modules repetition?

What about eclean-dist, will it work as expected for those modules
dependencies?

I think some of this answers would worth mention in documentation.

Sorry for anything I wrongly stated and thank you very much for your help,

Samuel



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Robin,

On 4/1/20 6:36 AM, Robin H. Johnson wrote:
> Samuel:
> I already proved that using go-module.eclass EGO_SUM it will NOT use Git
> repositories, and all of the fetching will happen long before
> src_unpack. Why do you persist with your statement to the contrary?

Sorry my misunderstanding, but I didn't get it right with what you
mentioned to me before:

> Have you looked at the EGO_SUM in go-module? This already covers ANY go
> package that uses go.mod + go.sum, in a fully reproducible way that does
> not break network sandbox.
I looked into the eclass documentation and I only found EGO_VENDOR[1].
This seems so expensive since I need to parse all go.mod or go.sum files
that I concluded would be better to use a tar.gz with all dependencies
(remember that my use case is with my personal overlay).

Looking deeply into the source code I could find more details about
EGO_SUM that only needs to set the urls to go.mod or/and go.sum files
present in the repository. This is useful, but only to official gentoo
developers, since if I want to use my own mirror type, for instance
mirror://gooverlay, I wouldn't have that option[3]:

> |
> # I am considering removing this and just hard coding mirror://goproxy
> # below, so please do not rely on it.
> : "${_GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}"|
So, go-module.eclass provides a good base to follow SRP pattern (single
responsibility principle). Looking into that effort I would copy
go-module.eclass to my overlay eclass directory and adapt it to use my
mirror type.

I would like to use directly Gentoo maintained go-module.eclass, but for
that I would like to have access to set my own mirror type.

I also ask you to update documentation since there is some details to
use EGO_SUM, such as you stated in comment[4].

Thanks,

Samuel

[1]
https://devmanual.gentoo.org/eclass-reference/go-module.eclass/index.html

[2] https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/go-module.eclass

[3]
https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/go-module.eclass#n164

[4]
https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/go-module.eclass#n31



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Michał Górny]

On Wed, 2020-04-01 at 05:36 +, Robin H. Johnson wrote:
> On Tue, Mar 31, 2020 at 09:18:32PM -0400, Michael Orlitzky wrote:
> > On 3/31/20 8:48 PM, Samuel Bernardo wrote:
> > > My question started with the network sandbox issue when we need to load
> > > external code dependencies. For example, a go project will download all
> > > dependencies from git repositories that will happen after src_unpack. In
> > > this case I need to add an additional tar.gz with that code along with
> > > the software release tar.gz.
> Samuel:
> I already proved that using go-module.eclass EGO_SUM it will NOT use Git
> repositories, and all of the fetching will happen long before
> src_unpack. Why do you persist with your statement to the contrary?
> 
> > > That additional tar.gz needs to be stored somewhere and as I understand
> > > local mirror could be the right place.
> > 
> > Normally we don't bundle dependencies, avoiding that problem entirely.
> > The Go eclasses however are badly designed, committed against protest by
> > paid corporate interests, and serve only to facilitate large-scale
> > copyright infringement and security vulnerabilities. If you're looking
> > for a consistent explanation of how they're supposed to work with the
> > rest of Gentoo, you won't find one.
> mjo: Can you please substantiate your claims? 
> 
> It would have been nice to have heard your concerns during February, any
> of one the three times that William and I posted the go-module.eclass
> EGO_SUM development work for review on this mailing list. I don't see a
> single email from you during that entire period.
> 

Do you really expect people to repeat themselves every time Go packaging
is discussed, and their concerns are ignored because upstream?  For one,
I've decided to focus on what's new in the eclasses, and just let
the tower topple of its own.

-- 
Best regards,
Michał Górny



signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] network sandbox challenge

[Michael Orlitzky]

On 4/1/20 1:36 AM, Robin H. Johnson wrote:
> mjo: Can you please substantiate your claims? 
> 
> It would have been nice to have heard your concerns during February, any
> of one the three times that William and I posted the go-module.eclass
> EGO_SUM development work for review on this mailing list. I don't see a
> single email from you during that entire period.
> 

Seriously?

Re: [gentoo-dev] network sandbox challenge

[Robin H. Johnson]

On Tue, Mar 31, 2020 at 09:18:32PM -0400, Michael Orlitzky wrote:
> On 3/31/20 8:48 PM, Samuel Bernardo wrote:
> > 
> > My question started with the network sandbox issue when we need to load
> > external code dependencies. For example, a go project will download all
> > dependencies from git repositories that will happen after src_unpack. In
> > this case I need to add an additional tar.gz with that code along with
> > the software release tar.gz.
Samuel:
I already proved that using go-module.eclass EGO_SUM it will NOT use Git
repositories, and all of the fetching will happen long before
src_unpack. Why do you persist with your statement to the contrary?

> > That additional tar.gz needs to be stored somewhere and as I understand
> > local mirror could be the right place.
> 
> Normally we don't bundle dependencies, avoiding that problem entirely.
> The Go eclasses however are badly designed, committed against protest by
> paid corporate interests, and serve only to facilitate large-scale
> copyright infringement and security vulnerabilities. If you're looking
> for a consistent explanation of how they're supposed to work with the
> rest of Gentoo, you won't find one.
mjo: Can you please substantiate your claims? 

It would have been nice to have heard your concerns during February, any
of one the three times that William and I posted the go-module.eclass
EGO_SUM development work for review on this mailing list. I don't see a
single email from you during that entire period.

The EGO_SUM support explicitly ensured that upstream distfiles (for each
dependency) remained absolutely as upstream provided them, without
merging the distfiles together or altering their content in way (I admit
that the exact naming of the distfiles changed, because it was terrible,
v0.0.0-20190311183353-d8887717615a.zip for example).

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: PGP signature

Re: [gentoo-dev] network sandbox challenge

[Michael Orlitzky]

On 3/31/20 8:48 PM, Samuel Bernardo wrote:
> 
> My question started with the network sandbox issue when we need to load
> external code dependencies. For example, a go project will download all
> dependencies from git repositories that will happen after src_unpack. In
> this case I need to add an additional tar.gz with that code along with
> the software release tar.gz.
> That additional tar.gz needs to be stored somewhere and as I understand
> local mirror could be the right place.

Normally we don't bundle dependencies, avoiding that problem entirely.
The Go eclasses however are badly designed, committed against protest by
paid corporate interests, and serve only to facilitate large-scale
copyright infringement and security vulnerabilities. If you're looking
for a consistent explanation of how they're supposed to work with the
rest of Gentoo, you won't find one.

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Michael,

Thank you for pointing that out.

My use case was about my personal overlay that is not mirrored by Gentoo
infrastructure.

My question started with the network sandbox issue when we need to load
external code dependencies. For example, a go project will download all
dependencies from git repositories that will happen after src_unpack. In
this case I need to add an additional tar.gz with that code along with
the software release tar.gz.
That additional tar.gz needs to be stored somewhere and as I understand
local mirror could be the right place.

Thanks,

Samuel

On 3/31/20 11:47 PM, Michael Orlitzky wrote:
> On 3/31/20 6:21 PM, Samuel Bernardo wrote:
>> But after your explanation, I understand now that mirror types provides
>> alias to use in ebuild SRC_URI, specially useful for the update task
>> (awesome).
>>
> Beware: thirdpartymirrors doesn't really do anything useful for normal
> ebuilds in ::gentoo. The gentoo infrastructure will automatically mirror
> anything in SRC_URI unless you set RESTRICT=mirror -- and that's where
> most people will download them from -- so (for example) adding a bunch
> of extra thirdpartymirrors entries to ensure that the distfiles are
> accessible is counter-productive: everyone will get the stuff from the
> gentoo mirrors anyway, and you wind up with a bunch of dead entries in
> the thirdpartymirrors file (at least half of the entries in there right
> now are dead, and no one is ever going to clean them up because no one
> uses them).
>



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Alec,

Thank you very much for your explanation.

I'll keep it in my notes.

Best,

Samuel

On 3/31/20 11:41 PM, Alec Warner wrote:
> In general I'd avoid using the mirror system as URI simplification too
> much; a lot of the idea is to avoid hardcoding specific hosts. E.g.
> for the gentoo mirrors we want to avoid hardcoding a *specific* mirror
> in the SRC_URI, so mirror://gentoo lets us say "any gentoo mirror will
> do." Often other thirdpartymirror systems act similarly. You see this
> commonly where mirrors are divided by region.
>
> example.com/mirror  # "global / us"
> example.au/mirror  # "australia"
> example.co.uk/mirror  # "UK"
>
> From a layout perspective these are the same, but you would not want
> to hardcode "https://example.com/mirror" in the ebuild, because then
> australian users are routed to the US..which is wasteful. So we use a
> generic:
>
> "mirror://example"
>
> And we potentially pick the best mirror at fetch time.
>
> This is mostly spelled out
> in: https://devmanual.gentoo.org/ebuild-writing/variables/#third-party-mirrors
>
> If you have a bunch of packages that share a common domain and its not
> sharded (there is only 1 host serving it) then you should consider
> using an eclass to share data between your ebuilds and not use
> thirdpartymirrors just to share a hostname.
>
> -A


signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Michael Orlitzky]

On 3/31/20 6:21 PM, Samuel Bernardo wrote:
> 
> But after your explanation, I understand now that mirror types provides
> alias to use in ebuild SRC_URI, specially useful for the update task
> (awesome).
> 

Beware: thirdpartymirrors doesn't really do anything useful for normal
ebuilds in ::gentoo. The gentoo infrastructure will automatically mirror
anything in SRC_URI unless you set RESTRICT=mirror -- and that's where
most people will download them from -- so (for example) adding a bunch
of extra thirdpartymirrors entries to ensure that the distfiles are
accessible is counter-productive: everyone will get the stuff from the
gentoo mirrors anyway, and you wind up with a bunch of dead entries in
the thirdpartymirrors file (at least half of the entries in there right
now are dead, and no one is ever going to clean them up because no one
uses them).

Re: [gentoo-dev] network sandbox challenge

[Alec Warner]

On Tue, Mar 31, 2020 at 3:21 PM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi,
> On 3/31/20 9:25 PM, Alec Warner wrote:
>
> From thirdpartymirrors file I can see more examples... The mirror type
>> can be any label that I decide to use?
>>
>
> man portage(5) says:
> Whenever portage encounters a mirror:// style URI it will look up the
> actual hosts here.  If the mirror set is not found here, it  will  check
>  the
> global  mirrors file at /var/db/repos/gentoo/profiles/thirdpartymirrors.
> You may also set a special mirror type called "local".  This list of
> mirrors will be checked before GENTOO_MIRRORS and will be used even if the
> package has RESTRICT="mirror" or RESTRICT="fetch".
>
> We have a bunch of existing mirror types (as you noticed). This lets us do
> things like "mirror://github" and then globally change the uri for github
> easily without editing the entire set of ebuilds because we late-bind the
> URI matching.
>
> Ok, so it works as an alias... The format definition led me to
> misunderstand:
>
> - mirror type followed by a list of hosts
>
> I interpreted mirror type as some internal definition not an alias.
>
>
>> Is URI mirror:// style deprecated in ebuilds and should use
>> restric="mirror" instead?
>>
>
> I don't understand how you could arrive at this conclusion. What would
> this do?
>
> RESTRICT=mirror is for things we are not legally allowed to mirror. That's
> basically all its for; it has nothing to do with custom mirrors or anything.
>
> Sorry for my dumb question, but my previous understanding of the context
> was that, since mirror:// is always accessed before the SRC_URI, there is
> no need to use mirror:// in ebuild SRC_URI, with the sources being
> downloaded from any available mirror (also remembering the dev manuals
> documentation related to  mirror://gentoo policy for auto mirroring the
> sources).
>
> But after your explanation, I understand now that mirror types provides
> alias to use in ebuild SRC_URI, specially useful for the update task
> (awesome).
>
> Looking into overlay context, I think that I could add thirpartymirrors
> file inside overlay profiles directory to use mirror:// in SRC_URI
> to simplify ebuild URI management in the end. For example, to all JetBrains
> software this will be awesome.
>
In general I'd avoid using the mirror system as URI simplification too
much; a lot of the idea is to avoid hardcoding specific hosts. E.g. for the
gentoo mirrors we want to avoid hardcoding a *specific* mirror in the
SRC_URI, so mirror://gentoo lets us say "any gentoo mirror will do." Often
other thirdpartymirror systems act similarly. You see this commonly where
mirrors are divided by region.

example.com/mirror # "global / us"
example.au/mirror # "australia"
example.co.uk/mirror # "UK"

>From a layout perspective these are the same, but you would not want to
hardcode "https://example.com/mirror; in the ebuild, because then
australian users are routed to the US..which is wasteful. So we use a
generic:

"mirror://example"

And we potentially pick the best mirror at fetch time.

This is mostly spelled out in:
https://devmanual.gentoo.org/ebuild-writing/variables/#third-party-mirrors

If you have a bunch of packages that share a common domain and its not
sharded (there is only 1 host serving it) then you should consider using an
eclass to share data between your ebuilds and not use thirdpartymirrors
just to share a hostname.

-A


>
Thanks,
>
> Samuel
>

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi,

On 3/31/20 9:25 PM, Alec Warner wrote:
>
> From thirdpartymirrors file I can see more examples... The mirror type
> can be any label that I decide to use?
>
>
> man portage(5) says:
> Whenever portage encounters a mirror:// style URI it will look up the
> actual hosts here.  If the mirror set is not found here, it  will
>  check  the
> global  mirrors file at
> /var/db/repos/gentoo/profiles/thirdpartymirrors.  You may also set a
> special mirror type called "local".  This list of mirrors will be
> checked before GENTOO_MIRRORS and will be used even if the package has
> RESTRICT="mirror" or RESTRICT="fetch".
>
> We have a bunch of existing mirror types (as you noticed). This lets
> us do things like "mirror://github" and then globally change the uri
> for github easily without editing the entire set of ebuilds because we
> late-bind the URI matching.

Ok, so it works as an alias... The format definition led me to
misunderstand:

> - mirror type followed by a list of hosts

I interpreted mirror type as some internal definition not an alias.

>
> Is URI mirror:// style deprecated in ebuilds and should use
> restric="mirror" instead?
>
>
> I don't understand how you could arrive at this conclusion. What would
> this do?
>
> RESTRICT=mirror is for things we are not legally allowed to mirror.
> That's basically all its for; it has nothing to do with custom mirrors
> or anything.

Sorry for my dumb question, but my previous understanding of the context
was that, since mirror:// is always accessed before the SRC_URI, there
is no need to use mirror:// in ebuild SRC_URI, with the sources being
downloaded from any available mirror (also remembering the dev manuals
documentation related to  mirror://gentoo policy for auto mirroring the
sources).

But after your explanation, I understand now that mirror types provides
alias to use in ebuild SRC_URI, specially useful for the update task
(awesome).

Looking into overlay context, I think that I could add thirpartymirrors
file inside overlay profiles directory to use mirror:// in
SRC_URI to simplify ebuild URI management in the end. For example, to
all JetBrains software this will be awesome.

Thanks,

Samuel



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Alec Warner]

On Tue, Mar 31, 2020 at 12:58 PM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi Alec,
>
> On 3/27/20 11:20 PM, Alec Warner wrote:
> >
> > I should point you at man portage(5) (search for mirrors), which has
> > more detail on how to set up a non-gentoo mirror network.
>
> Reading portage manpage about mirrors I didn't find the mirror type
> possible values. As I could understand, there is type local as a special
> mirror and then, from the example, sourceforge and gnu. Didn't
> understand right how this works.
>
> From thirdpartymirrors file I can see more examples... The mirror type
> can be any label that I decide to use?
>

man portage(5) says:
Whenever portage encounters a mirror:// style URI it will look up the
actual hosts here.  If the mirror set is not found here, it  will  check
 the
global  mirrors file at /var/db/repos/gentoo/profiles/thirdpartymirrors.
You may also set a special mirror type called "local".  This list of
mirrors will be checked before GENTOO_MIRRORS and will be used even if the
package has RESTRICT="mirror" or RESTRICT="fetch".

We have a bunch of existing mirror types (as you noticed). This lets us do
things like "mirror://github" and then globally change the uri for github
easily without editing the entire set of ebuilds because we late-bind the
URI matching.


>
> Is URI mirror:// style deprecated in ebuilds and should use
> restric="mirror" instead?
>

I don't understand how you could arrive at this conclusion. What would this
do?

RESTRICT=mirror is for things we are not legally allowed to mirror. That's
basically all its for; it has nothing to do with custom mirrors or anything.


>
> I think that, in mirrors description for /etc/portage/, first paragraph
> should be divided in two: the new one in  the phrase that starts with
> "You may also set a special mirror type called local".
>

Feel free to submit a patch; the source is on github:
https://github.com/gentoo/portage/blob/master/man/portage.5

-A


>
> Thanks,
>
> Samuel
>
>
>

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Alec,

On 3/27/20 11:20 PM, Alec Warner wrote:
>
> I should point you at man portage(5) (search for mirrors), which has
> more detail on how to set up a non-gentoo mirror network.

Reading portage manpage about mirrors I didn't find the mirror type
possible values. As I could understand, there is type local as a special
mirror and then, from the example, sourceforge and gnu. Didn't
understand right how this works.

From thirdpartymirrors file I can see more examples... The mirror type
can be any label that I decide to use?

Is URI mirror:// style deprecated in ebuilds and should use
restric="mirror" instead?

I think that, in mirrors description for /etc/portage/, first paragraph
should be divided in two: the new one in  the phrase that starts with
"You may also set a special mirror type called local".

Thanks,

Samuel




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Thank you very much for you detailed answers Alec.

I will add them to my FAQ,

Best,

Samuel

On 3/27/20 11:20 PM, Alec Warner wrote:
>
> On Fri, Mar 27, 2020 at 3:59 PM Alec Warner  > wrote:
>
>
> On Fri, Mar 27, 2020 at 3:10 PM Samuel Bernardo
>  > wrote:
>
> So what happens when RESTRIC=mirror is used inside ebuild for
> an overlay
> in git.gentoo.org ?
>
>
> I want to again re-iterate; gentoo doesn't mirror anything outside
> of gentoo.git, even if its hosted on git.gentoo.org
> . gentoo.git is literally the only repo the
> Gentoo Mirror system is processing.
>
> RESTRICT itself then has two potential usage sites.
>  - MIrroring. We already determined that for overlays, no
> mirroring occurs, so we can ignore that for your case.
>  - Fetching. Basically if something is RESTRICT=mirror, then we
> don't need to bother consulting the mirror network, because we
> know from the RESTRICT that mirror network will not have a copy.
> Fetching then proceeds from the origin URI (e.g. the URI in SRC_URI.)
>
>
> I should point you at man portage(5) (search for mirrors), which has
> more detail on how to set up a non-gentoo mirror network.
>
> So, thinking on site reliability, should it be a good choice
> to upload
> the necessary tar.gz, for example, to gitlab or github community
> services using git lfs as an alternative to
> "https://dev.gentoo.org/;?
>
>
> For most content served from dev.gentoo.org
>  its not RESTRICT=mirror, so the
> reliability of the origin URI is not an issue in most cases
> (because it should be on the gentoo mirror network.)
> Cases where it is not include:
>  - RESTRICT=mirror content.
>  - Content that is pushed to an ebuild but hasn't been mirrored yet.
>    - This can take up to 4h (or longer if the origin is unavailable.)
>
> In practice this hasn't been a big enough issue to do something
> more complicated. Many proposals have already been floated but
> none have ever been put into place.
> It also turns out that most of the time dev.gentoo.org
>  is available (and so again, its
> reliability is not a major issue.) I understand that it recently
> had an incident; I'm not really convinced it was high enough
> impact to make operational changes.
>


signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Alec Warner]

On Fri, Mar 27, 2020 at 3:59 PM Alec Warner  wrote:

>
>
> On Fri, Mar 27, 2020 at 3:10 PM Samuel Bernardo <
> samuelbernardo.m...@gmail.com> wrote:
>
>> Hi Alec,
>>
>> On 3/27/20 7:27 PM, Alec Warner wrote:
>> > The Gentoo Mirror system is basically a set of scripts that syncs the
>> > ::gentoo repository, enumerates all URIs in SRC_URI for all ebuilds,
>> > and fetches them.
>> > It doesn't enumerate anything in any overlays. Overlay authors are
>> > required to point SRC_URI somewhere useful (typically to an upstream
>> URI.)
>> >
>> > Gentoo Developers use "https://dev.gentoo.org/~user/distfiles; as an
>> > origin URI for items where either Gentoo *is* the upstream, or there
>> > is no upstream (e.g. a custom Gentoo patchset.) Any origin URI can be
>> > used; this just happens to be some free hosting that Gentoo Developers
>> > have access to use.
>> >
>> > -A
>>
>> Thank you for your clarification.
>>
>> So what happens when RESTRIC=mirror is used inside ebuild for an overlay
>> in git.gentoo.org?
>>
>
> I want to again re-iterate; gentoo doesn't mirror anything outside of
> gentoo.git, even if its hosted on git.gentoo.org. gentoo.git is literally
> the only repo the Gentoo Mirror system is processing.
>
> RESTRICT itself then has two potential usage sites.
>  - MIrroring. We already determined that for overlays, no mirroring
> occurs, so we can ignore that for your case.
>  - Fetching. Basically if something is RESTRICT=mirror, then we don't need
> to bother consulting the mirror network, because we know from the RESTRICT
> that mirror network will not have a copy. Fetching then proceeds from the
> origin URI (e.g. the URI in SRC_URI.)
>

I should point you at man portage(5) (search for mirrors), which has more
detail on how to set up a non-gentoo mirror network.

-A


>
>
>>
>> So, thinking on site reliability, should it be a good choice to upload
>> the necessary tar.gz, for example, to gitlab or github community
>> services using git lfs as an alternative to "https://dev.gentoo.org/;?
>>
>
> For most content served from dev.gentoo.org its not RESTRICT=mirror, so
> the reliability of the origin URI is not an issue in most cases (because it
> should be on the gentoo mirror network.)
> Cases where it is not include:
>  - RESTRICT=mirror content.
>  - Content that is pushed to an ebuild but hasn't been mirrored yet.
>- This can take up to 4h (or longer if the origin is unavailable.)
>
> In practice this hasn't been a big enough issue to do something more
> complicated. Many proposals have already been floated but none have ever
> been put into place.
> It also turns out that most of the time dev.gentoo.org is available (and
> so again, its reliability is not a major issue.) I understand that it
> recently had an incident; I'm not really convinced it was high enough
> impact to make operational changes.
>
> -A
>
>
>>
>> Best,
>>
>> Samuel
>>
>>
>>

Re: [gentoo-dev] network sandbox challenge

[Alec Warner]

On Fri, Mar 27, 2020 at 3:10 PM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi Alec,
>
> On 3/27/20 7:27 PM, Alec Warner wrote:
> > The Gentoo Mirror system is basically a set of scripts that syncs the
> > ::gentoo repository, enumerates all URIs in SRC_URI for all ebuilds,
> > and fetches them.
> > It doesn't enumerate anything in any overlays. Overlay authors are
> > required to point SRC_URI somewhere useful (typically to an upstream
> URI.)
> >
> > Gentoo Developers use "https://dev.gentoo.org/~user/distfiles; as an
> > origin URI for items where either Gentoo *is* the upstream, or there
> > is no upstream (e.g. a custom Gentoo patchset.) Any origin URI can be
> > used; this just happens to be some free hosting that Gentoo Developers
> > have access to use.
> >
> > -A
>
> Thank you for your clarification.
>
> So what happens when RESTRIC=mirror is used inside ebuild for an overlay
> in git.gentoo.org?
>

I want to again re-iterate; gentoo doesn't mirror anything outside of
gentoo.git, even if its hosted on git.gentoo.org. gentoo.git is literally
the only repo the Gentoo Mirror system is processing.

RESTRICT itself then has two potential usage sites.
 - MIrroring. We already determined that for overlays, no mirroring occurs,
so we can ignore that for your case.
 - Fetching. Basically if something is RESTRICT=mirror, then we don't need
to bother consulting the mirror network, because we know from the RESTRICT
that mirror network will not have a copy. Fetching then proceeds from the
origin URI (e.g. the URI in SRC_URI.)


>
> So, thinking on site reliability, should it be a good choice to upload
> the necessary tar.gz, for example, to gitlab or github community
> services using git lfs as an alternative to "https://dev.gentoo.org/;?
>

For most content served from dev.gentoo.org its not RESTRICT=mirror, so the
reliability of the origin URI is not an issue in most cases (because it
should be on the gentoo mirror network.)
Cases where it is not include:
 - RESTRICT=mirror content.
 - Content that is pushed to an ebuild but hasn't been mirrored yet.
   - This can take up to 4h (or longer if the origin is unavailable.)

In practice this hasn't been a big enough issue to do something more
complicated. Many proposals have already been floated but none have ever
been put into place.
It also turns out that most of the time dev.gentoo.org is available (and so
again, its reliability is not a major issue.) I understand that it recently
had an incident; I'm not really convinced it was high enough impact to make
operational changes.

-A


>
> Best,
>
> Samuel
>
>
>

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Alec,

On 3/27/20 7:27 PM, Alec Warner wrote:
> The Gentoo Mirror system is basically a set of scripts that syncs the
> ::gentoo repository, enumerates all URIs in SRC_URI for all ebuilds,
> and fetches them.
> It doesn't enumerate anything in any overlays. Overlay authors are
> required to point SRC_URI somewhere useful (typically to an upstream URI.)
>
> Gentoo Developers use "https://dev.gentoo.org/~user/distfiles; as an
> origin URI for items where either Gentoo *is* the upstream, or there
> is no upstream (e.g. a custom Gentoo patchset.) Any origin URI can be
> used; this just happens to be some free hosting that Gentoo Developers
> have access to use.
>
> -A

Thank you for your clarification.

So what happens when RESTRIC=mirror is used inside ebuild for an overlay
in git.gentoo.org?

So, thinking on site reliability, should it be a good choice to upload
the necessary tar.gz, for example, to gitlab or github community
services using git lfs as an alternative to "https://dev.gentoo.org/;?

Best,

Samuel




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Alec Warner]

On Fri, Mar 27, 2020 at 5:17 AM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi again Michał,
> On 3/27/20 11:48 AM, Michał Górny wrote:
> > Nope, just ::gentoo.  Minus ebuilds with RESTRICT=mirror.
>
> I have some doubts after reading the mirror documentation[1] in the
> context of personal overlays (not official).
>
> There is two procedures defined as I could understand:
> - manually upload a file to mirror://gentoo, scp it to
> dev.gentoo.org:/space/distfiles-local
>
> - having SRC_URI defined as
> SRC_URI="https://dev.gentoo.org/~myname/distfiles/${P}.tar.gz; to avoid
> namespace collisions with RESTRIC=mirror in ebuild would be uploaded
> automatically
>
> The use of mirror://gentoo directly is a deprecated policy. So it must
> be used https instead.
>
> 1) Did I understand it right?
>
> 2) What is dev.gentoo.org:~/public_html? Do this means that is only
> available to Gentoo official developers the access to the mirror[2]?
>
>
The Gentoo Mirror system is basically a set of scripts that syncs the
::gentoo repository, enumerates all URIs in SRC_URI for all ebuilds, and
fetches them.
It doesn't enumerate anything in any overlays. Overlay authors are required
to point SRC_URI somewhere useful (typically to an upstream URI.)

Gentoo Developers use "https://dev.gentoo.org/~user/distfiles; as an origin
URI for items where either Gentoo *is* the upstream, or there is no
upstream (e.g. a custom Gentoo patchset.) Any origin URI can be used; this
just happens to be some free hosting that Gentoo Developers have access to
use.

-A



> Best,
>
> Samuel
>
> [1] https://devmanual.gentoo.org/general-concepts/mirrors/index.html
>
> [2] https://wiki.gentoo.org/wiki/Project:Infrastructure/Developer_Webspace
>
>
>

Re: [gentoo-dev] network sandbox challenge

[Michał Górny]

On Fri, 2020-03-27 at 12:17 +, Samuel Bernardo wrote:
> Hi again Michał,
> On 3/27/20 11:48 AM, Michał Górny wrote:
> > Nope, just ::gentoo.  Minus ebuilds with RESTRICT=mirror.
> 
> I have some doubts after reading the mirror documentation[1] in the
> context of personal overlays (not official).
> 
> There is two procedures defined as I could understand:
> - manually upload a file to mirror://gentoo, scp it to
> dev.gentoo.org:/space/distfiles-local
> 
> - having SRC_URI defined as
> SRC_URI="https://dev.gentoo.org/~myname/distfiles/${P}.tar.gz; to avoid
> namespace collisions with RESTRIC=mirror in ebuild would be uploaded
> automatically
> 
> The use of mirror://gentoo directly is a deprecated policy. So it must
> be used https instead.
> 
> 1) Did I understand it right?

Yes, I think so.

> 
> 2) What is dev.gentoo.org:~/public_html? Do this means that is only
> available to Gentoo official developers the access to the mirror[2]?

It's just a web server.  Technically, you can use any server, either
private or public, as long as it provides reliable download.

Yes, only developers and proxied maintainers whose packages are
in ::gentoo.  In the latter case, we either upload their files for them
or they use some other hosting.

> 
> Best,
> 
> Samuel
> 
> [1] https://devmanual.gentoo.org/general-concepts/mirrors/index.html
> 
> [2] https://wiki.gentoo.org/wiki/Project:Infrastructure/Developer_Webspace
> 
> 

-- 
Best regards,
Michał Górny



signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] network sandbox challenge

[Rich Freeman]

On Fri, Mar 27, 2020 at 7:33 AM Michał Górny  wrote:
>
> On Fri, 2020-03-27 at 11:29 +, Samuel Bernardo wrote:
>
> > Same question for unpack context when using directly the source
> > repository with vcs functions.
>
> VCS ebuilds generally suck, for multiple reasons.  We allow users to use
> them but with minimal support.  However, e.g. git-r3 supports local
> mirrors to resolve some problems.
>

Any guides on using that from an ebuild maintenance standpoint?  The
eclass docs make it appear more for local sysadmin use vs as a way to
distribute things, but I could be reading into it.

Usually my solution to VCS ebuilds when that is all upstream supports
is to just create my own github mirror of the repo, tag an appropriate
commit, and then fetch the resulting tarball directly as a non-VCS
ebuild.  Essentially I end up running my own private fork of upstream.
At least, this is what I do for actual releases that upstream can't be
bothered to release properly - for a live - VCS ebuild I just
point to upstream.

I don't believe Gentoo has any kind of recommended/standardized
solution for this, but having ebuilds point to my own private fork of
things obviously is non-ideal.  However, I think it is still more
transparent than just bundling up a tarball manually and sticking it
in my devspace since at least with my forked repo everybody can see
where it came from and what state it is in, and of course it is easier
to maintain.

If there is a more preferred way of doing this I'd be interested to
hear about it.

-- 
Rich

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi again Michał,
On 3/27/20 11:48 AM, Michał Górny wrote:
> Nope, just ::gentoo.  Minus ebuilds with RESTRICT=mirror.

I have some doubts after reading the mirror documentation[1] in the
context of personal overlays (not official).

There is two procedures defined as I could understand:
- manually upload a file to mirror://gentoo, scp it to
dev.gentoo.org:/space/distfiles-local

- having SRC_URI defined as
SRC_URI="https://dev.gentoo.org/~myname/distfiles/${P}.tar.gz; to avoid
namespace collisions with RESTRIC=mirror in ebuild would be uploaded
automatically

The use of mirror://gentoo directly is a deprecated policy. So it must
be used https instead.

1) Did I understand it right?

2) What is dev.gentoo.org:~/public_html? Do this means that is only
available to Gentoo official developers the access to the mirror[2]?

Best,

Samuel

[1] https://devmanual.gentoo.org/general-concepts/mirrors/index.html

[2] https://wiki.gentoo.org/wiki/Project:Infrastructure/Developer_Webspace




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Michał Górny]

On Fri, 2020-03-27 at 11:45 +, Samuel Bernardo wrote:
> Hi Michał,
> 
> On 3/27/20 11:33 AM, Michał Górny wrote:
> > SRC_URI is well-defined, and that makes it possible for us and users to
> > develop consistent solutions.  We have Gentoo mirror network to increase
> > reliability when upstream servers fail.  Users can deploy local mirrors
> > to increase reliability further, improve throughput and make things work
> > in semi-isolated networks.
> 
> This is news for me. So to see if I understand the Gentoo mirror
> network, everything I place in SRC_URI is already mirrored when using my
> personal overlay in git.gentoo.org?

Nope, just ::gentoo.  Minus ebuilds with RESTRICT=mirror.

> 
> > > Same question for unpack context when using directly the source
> > > repository with vcs functions.
> > VCS ebuilds generally suck, for multiple reasons.  We allow users to use
> > them but with minimal support.  However, e.g. git-r3 supports local
> > mirrors to resolve some problems.
> 
> So, using local mirrors means that is the responsibility of the end user
> to review the ebuilds and create, for example, the local git mirror
> repository and then define EGIT_MIRROR_URI in make.conf to override for
> all ebuilds?
> 

Yes.  The mirrors use the same format as git-r3.eclass distdir, so it's
mostly useful to clone repositories from your other Gentoo system.

-- 
Best regards,
Michał Górny



signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Michał,

On 3/27/20 11:33 AM, Michał Górny wrote:
> SRC_URI is well-defined, and that makes it possible for us and users to
> develop consistent solutions.  We have Gentoo mirror network to increase
> reliability when upstream servers fail.  Users can deploy local mirrors
> to increase reliability further, improve throughput and make things work
> in semi-isolated networks.

This is news for me. So to see if I understand the Gentoo mirror
network, everything I place in SRC_URI is already mirrored when using my
personal overlay in git.gentoo.org?

>> Same question for unpack context when using directly the source
>> repository with vcs functions.
> VCS ebuilds generally suck, for multiple reasons.  We allow users to use
> them but with minimal support.  However, e.g. git-r3 supports local
> mirrors to resolve some problems.

So, using local mirrors means that is the responsibility of the end user
to review the ebuilds and create, for example, the local git mirror
repository and then define EGIT_MIRROR_URI in make.conf to override for
all ebuilds?

Thanks,

Samuel



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Michał Górny]

On Fri, 2020-03-27 at 11:29 +, Samuel Bernardo wrote:
> Hi Michał,
> 
> On 3/27/20 5:59 AM, Michał Górny wrote:
> > Stop here.  If you think that you need to 'break network sandbox', you
> > already have the wrong attitude and shouldn't continue.  Network sandbox
> > is not your enemy.  Using network is.
> > 
> > Network sandbox protects users from paying extra because you've written
> > a bad ebuild that unexpectedly downloads lot of data on their mobile
> > connection.  Network sandbox makes sure that we don't end up delivering
> > stuff that doesn't work to people who are on isolated networks or simply
> > have non-permanent connections.  Network sandbox makes sure that these
> > ebuilds will work three months from now when upstream randomly decides
> > to remove old files or shuffle servers, or just get hits by a temporary
> > issue.
> > 
> > There's no 'breaking the network sandbox'.  You must fix the ebuild not
> > to require Internet.
> 
> That is an awesome concept for producing ebuilds, since I could be using
> dist-cc and compiling multiple profiles using dedicated computing
> cluster leveraging available resources within a sandbox with very
> restricted access. This is a very nice pattern on resource management.
> This is another reason why I like Gentoo very much, with the SQA
> assurance of the high quality rules, and persuade me to invest my time
> using this wonderful distro.
> 
> I understand that network sandbox only restricts the build environment,
> but wouldn't the urls in SRC_URI be a problem when referencing sites
> that are not reliable? Shouldn't be relevant to define those sites that
> give better assurance for syncing the required binaries?

SRC_URI is well-defined, and that makes it possible for us and users to
develop consistent solutions.  We have Gentoo mirror network to increase
reliability when upstream servers fail.  Users can deploy local mirrors
to increase reliability further, improve throughput and make things work
in semi-isolated networks.

> Same question for unpack context when using directly the source
> repository with vcs functions.

VCS ebuilds generally suck, for multiple reasons.  We allow users to use
them but with minimal support.  However, e.g. git-r3 supports local
mirrors to resolve some problems.

-- 
Best regards,
Michał Górny



signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Michał,

On 3/27/20 5:59 AM, Michał Górny wrote:
> Stop here.  If you think that you need to 'break network sandbox', you
> already have the wrong attitude and shouldn't continue.  Network sandbox
> is not your enemy.  Using network is.
>
> Network sandbox protects users from paying extra because you've written
> a bad ebuild that unexpectedly downloads lot of data on their mobile
> connection.  Network sandbox makes sure that we don't end up delivering
> stuff that doesn't work to people who are on isolated networks or simply
> have non-permanent connections.  Network sandbox makes sure that these
> ebuilds will work three months from now when upstream randomly decides
> to remove old files or shuffle servers, or just get hits by a temporary
> issue.
>
> There's no 'breaking the network sandbox'.  You must fix the ebuild not
> to require Internet.

That is an awesome concept for producing ebuilds, since I could be using
dist-cc and compiling multiple profiles using dedicated computing
cluster leveraging available resources within a sandbox with very
restricted access. This is a very nice pattern on resource management.
This is another reason why I like Gentoo very much, with the SQA
assurance of the high quality rules, and persuade me to invest my time
using this wonderful distro.

I understand that network sandbox only restricts the build environment,
but wouldn't the urls in SRC_URI be a problem when referencing sites
that are not reliable? Shouldn't be relevant to define those sites that
give better assurance for syncing the required binaries?
Same question for unpack context when using directly the source
repository with vcs functions.

Best,
Samuel




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Robin,

On 3/27/20 3:03 AM, Robin H. Johnson wrote:
> Have you looked at the EGO_SUM in go-module? This already covers ANY go
> package that uses go.mod + go.sum, in a fully reproducible way that does
> not break network sandbox.

I didn't understand EGO_SUM right.

Thank you for mentioned it.

Best,

Samuel




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Hi Haelwenn,
On 3/27/20 1:50 AM, Haelwenn (lanodan) Monnier wrote:
> Couldn't the snapd_${PV}.vendor.tar.xz available in 
> https://github.com/snapcore/snapd/releases 
> work in your case to avoid downloading tarballs?
> And probably consider using go-modules.eclass, which can also allow
> packaging when there is no vendoring done by the upstream by just
> cut(1)'ing the content of go.sum
I'm using the archive tarball... You're right I will try it instead and
thanks for the heads-up about controlling the content of go.sum using
go-modules.eclass.
> And I'm not so sure why you want to apparently host a tarball?
> Maybe you're not aware that SRC_URI accepts multiple tarballs? And 
> btw you strongly should only use upstream URLs in it, they don't
> need to be mirrored in distfiles.gentoo.org for the ebuild to work.

The reason to host the tarball was regarded to my understanding of
requirements pointed out in src_test[1] function notes about network
sandbox. Not related to the build process, but thinking on an
environment with restricted proxy access to only trusted locations there
would be probably a limitation to access any url. So for distributing
additional sources behind well known places such as gentoo mirrors,
github or gitlab could be an issue.

What do you think about this?

Thanks

[1]
https://devmanual.gentoo.org/ebuild-writing/functions/src_test/index.html




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] network sandbox challenge

[Michał Górny]

On Fri, 2020-03-27 at 01:16 +, Samuel Bernardo wrote:
> Dear all,
> 
> Fulfilling the linting of ebuild code style design for software projects
> that loads their dependencies during build, such as go based projects or
> npm as an example, could be very nasty.
> 
> Looking into source code of snapd or opennebula as two examples, I need
> to break network sandbox to get all dependencies for snapd go modules or
> opennebula sunstone npm code.
> 

Stop here.  If you think that you need to 'break network sandbox', you
already have the wrong attitude and shouldn't continue.  Network sandbox
is not your enemy.  Using network is.

Network sandbox protects users from paying extra because you've written
a bad ebuild that unexpectedly downloads lot of data on their mobile
connection.  Network sandbox makes sure that we don't end up delivering
stuff that doesn't work to people who are on isolated networks or simply
have non-permanent connections.  Network sandbox makes sure that these
ebuilds will work three months from now when upstream randomly decides
to remove old files or shuffle servers, or just get hits by a temporary
issue.

There's no 'breaking the network sandbox'.  You must fix the ebuild not
to require Internet.

-- 
Best regards,
Michał Górny



signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] network sandbox challenge

[Robin H. Johnson]

On Fri, Mar 27, 2020 at 01:16:43AM +, Samuel Bernardo wrote:
> Dear all,
> 
> Fulfilling the linting of ebuild code style design for software projects
> that loads their dependencies during build, such as go based projects or
> npm as an example, could be very nasty.
> 
> Looking into source code of snapd or opennebula as two examples, I need
> to break network sandbox to get all dependencies for snapd go modules or
> opennebula sunstone npm code.
Have you looked at the EGO_SUM in go-module? This already covers ANY go
package that uses go.mod + go.sum, in a fully reproducible way that does
not break network sandbox.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: PGP signature

Re: [gentoo-dev] network sandbox challenge

[Haelwenn (lanodan) Monnier]

[2020-03-27 01:16:43+] Samuel Bernardo:
> 2) For snapd I need to load previously the remote repositories
> dependencies into a tar.gz that need to be stored in ebuild files. This
> is ugly, I know, but there is no distfiles trusted repository
> alternative where I can place them.
> As a workaround, I could create gitlab or github repository and use git
> lfs to upload the artifacts that need to be loaded (since I didn't know
> any free Artifactory or NexusOSS repositories community supported). Then
> use the provided url to download the files.

Couldn't the snapd_${PV}.vendor.tar.xz available in 
https://github.com/snapcore/snapd/releases 
work in your case to avoid downloading tarballs?
And probably consider using go-modules.eclass, which can also allow
packaging when there is no vendoring done by the upstream by just
cut(1)'ing the content of go.sum

And I'm not so sure why you want to apparently host a tarball?
Maybe you're not aware that SRC_URI accepts multiple tarballs? And 
btw you strongly should only use upstream URLs in it, they don't
need to be mirrored in distfiles.gentoo.org for the ebuild to work.

[gentoo-dev] network sandbox challenge

[Samuel Bernardo]

Dear all,

Fulfilling the linting of ebuild code style design for software projects
that loads their dependencies during build, such as go based projects or
npm as an example, could be very nasty.

Looking into source code of snapd or opennebula as two examples, I need
to break network sandbox to get all dependencies for snapd go modules or
opennebula sunstone npm code.

1) For opennebula I can use the full releases that brings already
generated sunstone code, with the penalty to loose some patches or
updates in the middle.

2) For snapd I need to load previously the remote repositories
dependencies into a tar.gz that need to be stored in ebuild files. This
is ugly, I know, but there is no distfiles trusted repository
alternative where I can place them.
As a workaround, I could create gitlab or github repository and use git
lfs to upload the artifacts that need to be loaded (since I didn't know
any free Artifactory or NexusOSS repositories community supported). Then
use the provided url to download the files.

What do think about this two cases and what are your suggestions?

Thanks,

Samuel




signature.asc
Description: OpenPGP digital signature