Hi, Here is a diff that adds go support in portgen(1). It's a combination of diffs from my self and afresh1@. There are a few issues, but on the whole this is the direction I think we should take with regard to porting Go application.
The biggest issue with the diff is it's inability to cope with the way Go escapes[1] uppercase letters in URLs. This means that this implementation can't package things like "github.com/gohugoio/hugo" as it has some dependencies like: "github.com/BurntSushi/toml". As for usage, one needs to get the "module" name from a given program. This is the first line of the "mo.mod" file contained in the program being ported. Once you have that it's simply a call to portgen: portgen go gthub.com/qbit/gavin This causes portgen to: 1) Fetch https://proxy.golang.org/github.com/qbit/gavin/@latest 2) Grab the latest version. 3) Fetch the .mod file: https://proxy.golang.org/github.com/qbit/gavin/@v/v0.0.0-20191129154004-5242047b11bb.mod 4) Call 'go mod graph' to get a list of all dependencies and their versions. We need the .zip files and a handful of .mod files (as described in [1]) to complete the build. 5) Generate the port. We are using SUPDISTFILES for the .mod and .zip files as Go doesn't need the zip files to be extracted on disk. With this diff I can package a number of applications (wireguard, honk and a number of pet projects I have) without issue! I'd like to get this diff into the tree to make collaboration easier. As it is now, I can build other ports (with portgen) fine, and in-tree go packages build as expected. [1] https://tip.golang.org/cmd/go/#hdr-Module_proxy_protocol ** MMITYWTR (Much More Info Than You Wanted To Read) Here is an outline of the current methods for creating a Go port and a bit of reasoning as to why I went with the 'go mod' method. 1) Using GOPATH. *Advantages*: - Minimal changes to our tree. - Easy. *Disadvantages*: - Going away in future releases. - Prevents Go from using versioned modules (everything basically builds off of 'master' of what ever dependency. This will likely introduce issues that seem like upstream problems, but are actually local to our port. 2) Using the vendor directory. *Advantages*: - Easy. - Currently in use. - Maintains "build integrity" by keeping versioned dependencies in the 'vendor' directory. *Disadvantages*: - Requires building / hosting of 'vendor'ed distfiles. - Might go away in the future. 3) Using 'go mod'. *Advantages*: - The New© Go⑨ Way™. Likely to stay the defacto method for some time. - Maintains "build integrity". The binaries we produce in the ports tree are the same as the binaries upstream will produce. *Disadvantages*: - More complex to implement. - Makes patching an application's dependencies difficult[2]. But one could still fall back to the 'vendor'ed method if patches are absolutely required (this breaks "build integrity" though). For portgen(1) I have opted to use method 3. [2] I see this as an advantage, as it will encourage people to upstream diffs first. This raises awareness of OpenBSD and lubes the interactions between program and library developers. diff --git a/infrastructure/bin/portgen b/infrastructure/bin/portgen index ad5ab17f3cf..b7316d42b64 100755 --- a/infrastructure/bin/portgen +++ b/infrastructure/bin/portgen @@ -32,6 +32,7 @@ use lib ( "$portdir/infrastructure/lib", "$FindBin::Bin/../lib" ); use OpenBSD::PortGen::Port::CPAN; use OpenBSD::PortGen::Port::PyPI; use OpenBSD::PortGen::Port::Ruby; +use OpenBSD::PortGen::Port::Go; my ( $type, $module ) = @ARGV; @@ -44,6 +45,8 @@ if ( $type eq 'p5' ) { $o = OpenBSD::PortGen::Port::PyPI->new(); } elsif ( $type eq 'ruby' ) { $o = OpenBSD::PortGen::Port::Ruby->new(); +} elsif ( $type eq 'go' ) { + $o = OpenBSD::PortGen::Port::Go->new(); } else { die "unknown module type\n"; } diff --git a/infrastructure/lib/OpenBSD/PortGen/Port.pm b/infrastructure/lib/OpenBSD/PortGen/Port.pm index dc4c6bd6b7b..06814357f67 100644 --- a/infrastructure/lib/OpenBSD/PortGen/Port.pm +++ b/infrastructure/lib/OpenBSD/PortGen/Port.pm @@ -203,9 +203,7 @@ sub set_build_deps { my ( $self, $build_deps ) = @_; - # Makefile.template is missing a tab for BUILD_DEPENDS - # and we want the port to be pretty, so add one - $self->{BUILD_DEPENDS} = "\t" . $build_deps if $build_deps; + $self->{BUILD_DEPENDS} = $build_deps; } sub set_run_deps @@ -344,6 +342,22 @@ sub write_makefile delete $configs{EXTRACT_SUFX} if $configs{EXTRACT_SUFX} and $configs{EXTRACT_SUFX} eq '.tar.gz'; + my $format = sub { + my ($key, $value, %opts) = @_; + + my $tabs = "\t" x ( $opts{tabs} || 1 ); + $key .= $opts{equal} || $default_equal; + + if (ref $value eq 'ARRAY') { + my $key_tabs = "\t" x ( length($key) / 8 ); + $value = join " \\\n$key_tabs$tabs", @{ $value } + } + + $key .= $tabs if length $value; + + return $key . $value; + }; + my @makefile; foreach my $line (@template) { next # no more than one blank line @@ -358,10 +372,7 @@ sub write_makefile next if $key !~ /^[\p{Upper}_]+(?:-\w+)?$/; my $value = $configs{$key}; next unless defined $value; - - my $print_key = "$key$default_equal"; - $print_key .= "\t" if length $value; - push @additions, "$print_key$value"; + push @additions, $format->($key, $value); } if (@additions) { push @makefile, @@ -382,19 +393,12 @@ sub write_makefile } # If we didn't get a value, copy from the template - if ( not $value and %copy_values ) { - $value = $line->{value} - if $copy_values{$key} - and not $reset_values{$key}; - } + $value ||= $line->{value} + if $copy_values{$key} + and not $reset_values{$key}; next unless defined $value; - my $equal = $line->{equal} || $default_equal; - my $tabs = "\t" x ( $line->{tabs} || 1 ); - my $print_key = "$key$equal"; - $print_key .= $tabs if length $value; - if ( $key eq 'PERMIT_PACKAGE' && $license ) { # guess that the comment before this was # the license marker. @@ -402,7 +406,7 @@ sub write_makefile push @makefile, "# $license"; } - push @makefile, "$print_key$value"; + push @makefile, $format->($key, $value, %{$line}); } else { push @makefile, $line; } diff --git a/infrastructure/lib/OpenBSD/PortGen/Port/Go.pm b/infrastructure/lib/OpenBSD/PortGen/Port/Go.pm new file mode 100644 index 00000000000..d9739d69e44 --- /dev/null +++ b/infrastructure/lib/OpenBSD/PortGen/Port/Go.pm @@ -0,0 +1,217 @@ +# $OpenBSD: Go.pm,v 1.16 2019/05/16 16:01:10 afresh1 Exp $ +# +# Copyright (c) 2019 Aaron Bieber <[email protected]> +# +# Permission to use, copy, modify, and distribute this software for any +# purpose with or without fee is hereby granted, provided that the above +# copyright notice and this permission notice appear in all copies. +# +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +package OpenBSD::PortGen::Port::Go; + +use 5.028; +use utf8; +use warnings; +use strict; +use warnings qw(FATAL utf8); # fatalize encoding glitches +use open qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8 + +use parent 'OpenBSD::PortGen::Port'; + +use Carp; +use Cwd; +use File::Temp qw/ tempdir /; + +use OpenBSD::PortGen::Dependency; + +sub ecosystem_prefix +{ + my $self = shift; + return ''; +} + +sub base_url +{ + my $self = shift; + return 'https://proxy.golang.org/'; +} + +sub get_dist_info +{ + my ( $self, $module ) = @_; + + my $json = $self->get_json( $module . '/@latest' ); + $json->{Name} = ( split '/', $module )[-1]; + $json->{Module} = $module; + + my %mods; + for ( $self->_go_mod_graph($json) ) { + my ($direct, $ephemeral) = @{$_}; + + for my $d ( $direct, $ephemeral ) { + #next if $d->{Module} eq $module; + next unless $d->{Version}; + $mods{ $d->{Module} }{ $d->{Version} } ||= $d; + } + } + + # Filter dependencies to the one with the highest version + foreach my $mod ( sort keys %mods ) { + # Sort semver numerically then the rest alphanumeric. + # This is overkill for sorting, but I already wrote it + my @versions = + map { $_->[0] } + sort { + $a->[1] <=> $b->[1] + || $a->[2] <=> $b->[2] + || $a->[3] <=> $b->[3] + || $a->[4] cmp $b->[4] + } + map { $_->[1] =~ s/^v//; $_ } + map { [ $_, split /[\.-]/, $_, 4 ] } + keys %{ $mods{$mod} }; + + push @{ $json->{Dist} }, $mods{$mod}{ $versions[-1] }; + push @{ $json->{Mods} }, map { $mods{$mod}{$_} } @versions; + } + + return $json; +} + +sub _go_mod_graph +{ + my ($self, $json) = @_; + my $dir = tempdir(CLEANUP => 1); + + my $mod = $self->get("$json->{Module}/\@v/$json->{Version}.mod"); + my ($module) = $mod =~ /\bmodule\s+(.*)\n/; + $module =~ s/\s+$//; + unless ( $json->{Module} eq $module ) { + my $msg = "Module $json->{Module} doesn't match $module"; + warn "$msg\n"; + croak $msg; + } + + { + open my $fh, '>', $dir . "/go.mod" or die $!; + print $fh $mod; + close $fh; + } + + my $old_cwd = getcwd(); + chdir $dir or die "Unable to chdir '$dir': $!"; + + my @mods; + { + # Outputs: "direct_dep ephemeral_dep" + local $ENV{GOPATH} = "$dir/go"; + open my $fh, '-|', qw< go mod graph > + or die "Unable to spawn 'go mod path': $!"; + @mods = readline $fh; + close $fh + or die "Error closing pipe to 'go mod path': $!"; + } + + chdir $old_cwd or die "Unable to chdir '$old_cwd': $!"; + + chomp @mods; + + # parse the graph into pairs of hashrefs + return map { [ + map { + my ($m, $v) = split /@/; + { Module => $m, Version => $v }; + } split /\s/ + ] } grep { $_ } @mods; +} + +sub get_ver_info +{ + my ( $self, $module ) = @_; + return $self->get_json( $module . '/@latest' ); +} + +sub name_new_port +{ + my ( $self, $di ) = @_; + + my $name = $di->{Name}; + $name = $self->SUPER::name_new_port($name); + $name = "go/$name" unless $name =~ m{/}; + + return $name; +} + +sub fill_in_makefile +{ + my ( $self, $di, $vi ) = @_; + + $self->set_modules('lang/go'); + $self->set_comment("todo"); + $self->set_descr("TODO"); + $self->set_license("unknown license"); + $self->set_other( MODGO_MODNAME => $di->{Module} ); + $self->set_other( MODGO_VERSION => $di->{Version} ); + $self->set_distname($di->{Name} . '-${MODGO_VERSION}'); + + my @parts = split("-", $di->{Version}); + if (@parts > 1) { + $self->set_pkgname($di->{Name} . "-" . $parts[1]) + if $parts[1] =~ m/\d{6}/; + } else { + $parts[0] =~ s/^v//; + $self->set_pkgname($di->{Name} . "-" . $parts[0]); + } + + my @dist = map { [ $_->{Module}, $_->{Version} ] } + @{ $di->{Dist} || [] }; + my @mods = map { [ $_->{Module}, $_->{Version} ] } + @{ $di->{Mods} }; + + # Turn the deps into tab separated columns + foreach my $s ( \@dist, \@mods ) { + next unless @{$s}; # if there aren't any, don't try + my ($length) = sort { $b <=> $a } map { length $_->[0] } @$s; + my $n = ( 1 + int $length / 8 ); + @{$s} = map { + ( my $m = $_->[0] ) =~ s/\p{Upper}/!\L$&/g; + $m =~ s/[^\w\/\.]/sprintf "%%%02x", ord $&/ge; + my $tabs = "\t" x ( $n - int( length($m) / 8 ) ); + "$m$tabs$_->[1]" + } @{$s}; + } + + $self->set_other( MODGO_MODULES => \@dist ) if @dist; + $self->set_other( MODGO_MODFILES => \@mods ) if @mods; +} + +sub try_building +{ + my $self = shift; + $self->make_fake(); +} + +sub postextract +{ +} + +sub get_deps +{ + my ( $self, $di, $wrksrc ) = @_; + my $deps = OpenBSD::PortGen::Dependency->new(); + + return $deps->format; +} + +sub get_config_style +{ +} + +1; diff --git a/infrastructure/templates/Makefile.template b/infrastructure/templates/Makefile.template index 1f9264b7336..6889a51a600 100644 --- a/infrastructure/templates/Makefile.template +++ b/infrastructure/templates/Makefile.template @@ -22,7 +22,13 @@ COMMENT = ??? # #MODPY_EGG_VERSION = ??? +# MODGO_MODNAME should be set to the 'module' specified in the 'go.mod' file. +#MODGO_MODNAME = github.com/test/app # +# Version of port if using lang/go and MODGO_MODULES +# +#MODGO_VERSION = 0.1.1 + # What port/package will be created # # DISTNAME should not include suffix (like .tar.gz .tgz .tar.bz2 etc.) @@ -125,6 +131,15 @@ MASTER_SITES = ??? # If port is python3 only #MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3} +# +# MODGO_ settings for when using lang/go module +# +# Get source from proxy.golang.org +#MODGO_MODULES = modulename version +# These are needed for dependency resolution. We don't actually need the +# coresponding code +#MODGO_MODFILES = modulename version + # Dependencies #BUILD_DEPENDS = ??? #RUN_DEPENDS = ??? diff --git a/lang/go/go.port.mk b/lang/go/go.port.mk index 983c3990706..96851a3c001 100644 --- a/lang/go/go.port.mk +++ b/lang/go/go.port.mk @@ -4,6 +4,13 @@ ONLY_FOR_ARCHS ?= ${GO_ARCHS} MODGO_BUILDDEP ?= Yes +MODGO_DIST_SUBDIR ?= go_modules + +MASTER_SITE_ATHENS = https://proxy.golang.org/ + +MODGO_MASTER_SITESN = 9 +MASTER_SITES${MODGO_MASTER_SITESN} ?= ${MASTER_SITE_ATHENS} + MODGO_RUN_DEPENDS = lang/go MODGO_BUILD_DEPENDS = lang/go @@ -33,17 +40,12 @@ MODGO_TYPE ?= bin MODGO_WORKSPACE ?= ${WRKDIR}/go MODGO_GOCACHE ?= ${WRKDIR}/go-cache MODGO_GOPATH ?= ${MODGO_WORKSPACE}:${MODGO_PACKAGE_PATH} -MAKE_ENV += GOCACHE="${MODGO_GOCACHE}" \ - GOPATH="${MODGO_GOPATH}" \ - GO111MODULE=off # We cannot assume that the maching running the built code will have SSE, # even though the machine building the package has SSE. As such, we need # to explicitly disable SSE on i386 builds. MAKE_ENV += GO386=387 -# Ports are not allowed to fetch from the network at build time; point -# GOPROXY at an unreachable host so that failures are also visible to -# developers who don't have PORTS_PRIVSEP and a "deny .. _pbuild" PF rule. -MAKE_ENV += GOPROXY=invalid://ports.should.not.fetch.at.buildtime/ +MAKE_ENV += GOCACHE="${MODGO_GOCACHE}" + MODGO_CMD ?= ${SETENV} ${MAKE_ENV} go MODGO_BUILD_CMD = ${MODGO_CMD} install ${MODGO_FLAGS} MODGO_TEST_CMD = ${MODGO_CMD} test ${MODGO_FLAGS} ${MODGO_TEST_FLAGS} @@ -54,19 +56,35 @@ MODGO_BUILD_CMD += -ldflags="${MODGO_LDFLAGS}" MODGO_TEST_CMD += -ldflags="${MODGO_LDFLAGS}" .endif -.if defined(GH_ACCOUNT) && defined(GH_PROJECT) -ALL_TARGET ?= github.com/${GH_ACCOUNT}/${GH_PROJECT} +.if defined(MODGO_MODNAME) +EXTRACT_SUFX ?= .zip +PKGNAME ?= ${DISTNAME:S/-v/-/} +ALL_TARGET ?= ${MODGO_MODNAME} +DISTFILES = ${DISTNAME}${EXTRACT_SUFX}{${MODGO_VERSION}${EXTRACT_SUFX}} +MASTER_SITES ?= ${MASTER_SITE_ATHENS}${MODGO_MODNAME}/@v/ +. for _modname _modver in ${MODGO_MODULES} +SUPDISTFILES += ${MODGO_DIST_SUBDIR}/${_modname}/@v/${_modver}.zip{${_modname}/@v/${_modver}.zip}:${MODGO_MASTER_SITESN} +. endfor +. for _modname _modver in ${MODGO_MODFILES} +SUPDISTFILES += ${MODGO_DIST_SUBDIR}/${_modname}/@v/${_modver}.mod{${_modname}/@v/${_modver}.mod}:${MODGO_MASTER_SITESN} +. endfor +MAKE_ENV += GOPROXY=file://${DISTDIR}/${MODGO_DIST_SUBDIR} +MAKE_ENV += GO111MODULE=on GOPATH="${MODGO_GOPATH}" +.else +# ports are not allowed to fetch from the network at build time; point +# GOPROXY at an unreachable host so that failures are also visible to +# developers who don't have PORTS_PRIVSEP and a "deny .. _pbuild" PF rule. +MAKE_ENV += GOPROXY=invalid://ports.should.not.fetch.at.buildtime/ +MAKE_ENV += GO111MODULE=off GOPATH="${MODGO_GOPATH}" .endif -TEST_TARGET ?= ${ALL_TARGET} -SEPARATE_BUILD ?= Yes -WRKSRC ?= ${MODGO_WORKSPACE}/src/${ALL_TARGET} +MODGO_TEST_TARGET ?= cd ${WRKSRC} && ${MODGO_CMD} test ${ALL_TARGET} -MODGO_SETUP_WORKSPACE = mkdir -p ${WRKSRC:H}; mv ${MODGO_SUBDIR} ${WRKSRC}; +SEPARATE_BUILD ?= Yes CATEGORIES += lang/go -MODGO_BUILD_TARGET = ${MODGO_BUILD_CMD} ${ALL_TARGET} +MODGO_BUILD_TARGET = ${MODGO_BUILD_CMD} MODGO_FLAGS += -v -p ${MAKE_JOBS} .if empty(DEBUG) @@ -76,6 +94,14 @@ MODGO_LDFLAGS += -s -w MODGO_FLAGS += -x .endif +.if empty(MODGO_MODNAME) +WRKSRC ?= ${MODGO_WORKSPACE}/src/${ALL_TARGET} +MODGO_SETUP_WORKSPACE = mkdir -p ${WRKSRC:H}; mv ${MODGO_SUBDIR} ${WRKSRC}; +.else +WRKSRC ?= ${WRKDIR}/${MODGO_MODNAME}@${MODGO_VERSION} +MODGO_SETUP_WORKSPACE = ln -sf ${WRKSRC} ${WRKDIR}/${MODGO_MODNAME} +.endif + INSTALL_STRIP = .if ${MODGO_TYPE:L:Mbin} MODGO_INSTALL_TARGET = ${INSTALL_PROGRAM_DIR} ${PREFIX}/${MODGO_BINDIR} && \ @@ -99,14 +125,13 @@ MODGO_INSTALL_TARGET += ${INSTALL_DATA_DIR} ${MODGO_PACKAGE_PATH} && \ RUN_DEPENDS += ${MODGO_RUN_DEPENDS} .endif -MODGO_TEST_TARGET = ${MODGO_TEST_CMD} ${TEST_TARGET} - .if empty(CONFIGURE_STYLE) MODGO_pre-configure += ${MODGO_SETUP_WORKSPACE} . if !target(do-build) do-build: - ${MODGO_BUILD_TARGET} + cd ${WRKSRC} && \ + ${MODGO_BUILD_TARGET} . endif . if !target(do-install) -- PGP: 0x1F81112D62A9ADCE / 3586 3350 BFEA C101 DB1A 4AF0 1F81 112D 62A9 ADCE
