For some dependency management systems, trust is implied and not verified. The MacPorts approach can definitely provide benefits here: a dependency is declared with checksums, and it is compared against the downloaded version. This can happen with Ruby gems, Cargo crates, Python packages, NPM, etc. Where there is a single distribution arm (even if mirrored like MacPorts itself), and presumably for some "distributed" distribution systems (Maven? I don't do Java), then MacPorts provides benefits. In most cases, these dependencies are *packages*, which means that they are fancy tarballs of a subset of the whole development tree.
This is not the case for Go, where `go.sum` stores the digest of the dependencies and the "distribution" system is git. Like the Node community, there is a common pattern (especially with large API providers like AWS and Google) of using monorepos. There are ways to use git to only clone specific subfolders (git clone --filter and git sparse-checkout) at tags, and while I cannot guarantee that this is what Go uses, it certainly does not do what the Golang portgroup is doing when you use go.vendors. A new release of the whole AWS SDK v2 for Go was cut yesterday, but that's not how the go dependencies are computed. If I just need S3 and ACM, I would use `import "github.com/aws/aws-sdk-go-v2/service/s3"` and `import " github.com/aws/aws-sdk-go-v2/service/acm"`. I might do `go get github.com/aws/aws-sdk-go-v2/service/[email protected]` and `go get github.com/aws/aws-sdk-go-v2/[email protected]` (those are both the latest versions of the individual service SDKs). For go.vendors to work, I have to download the WHOLE repository at `service/s3/v1.49.0` and `service/acm/v1.23.1` tags, which is dozens of times larger than what `go get` pulls down (almost 70Mb each). `go get` on ACM downloads about 7MB of code and dependencies. If I use ten different AWS dependencies, each MacPorts build machine (and the developer trying to update `go.vendors`) has to download nearly 700Mb of dependencies instead of 50–100Mb (go get will not download dependencies already in its cache) for those ten dependencies. This is why I focused exclusively on the Golang portgroup. I don't think that the MacPorts model is fundamentally broken. I appreciate it. But I think that go.vendors is the worst approach for Go, and I think that it makes things actively worse from a port and dependency management perspective because it adds little over Go's existing online mechanisms. If Go mirrors could be treated as targets for `GOPROXY`, then I think we would see substantial benefit. There's a bootstrapping question on this, for how a dependency would make it onto the mirrors in the first place, but I’m raising this to try to get possible ideas for the solution. -a On Fri, Feb 16, 2024 at 9:58 AM Kirill A. Korinsky <[email protected]> wrote: > On Fri, 16 Feb 2024 10:24:59 +0100, > Nils Breunese wrote: > > > > Allowing ports to download dependencies at build time carries the risk of > > those dependencies not being available at build time, which I guess is > why > > MacPorts is not a fan of this method. In a corporate setting this is > typically > > mitigated by using an in-house repository manager (e.g. Artifactory or > Nexus), > > which caches these dependency artifacts. This is kind of similar to > MacPorts > > servers storing files, but a repository manager provides interfaces that > can > > be directly used by build tools to fetch dependencies, so if we’d have a > > MacPorts repository manager a port developer (or even better: port group) > > could configure a Maven/Gradle/NPM/Yarn/whatever port to use the MacPorts > > repository manager, and then port build should no longer depend on > publicly > > available dependencies after the initial build. > > > > Let switch to another and usually missed aspect of automatic downloading > dependency from interten to build a software: security. > > For example Maven as the most popular Java build system doesn't care about > checksums for dependency. > > It ultimatley trust used repository and when it builds software it can > inject > any unexpected code. > > From another side, MacPorts enforces checksuming of dependency that > garantue > that it is the same artifact that was used before. > > -- > wbr, Kirill > -- Austin Ziegler • [email protected] • [email protected] http://www.halostatue.ca/ • http://twitter.com/halostatue
