On Mon, Aug 07, 2023 at 09:17:05PM +0100, Stuart Henderson wrote:
[...]
> I think maybe I'd prefer to have some variable that could be used
> *instead* of the existing GH_* variables rather than in conjunction with
> them (so they can be used for all GH archive ports, rather than have
> them a special case for multi-distfile ports). If that's the standard
> way to do things we can have a sweep of the tree converting other
> ports (or at least the ones that don't use go.port.mk ;)
>
> It would be kind-of helpful if it could support more than just github
> too (gitlab.com, sr.ht, ..). While that could be done with different
> variables (GH_xx, GL_xx, SRHT_xx etc) they're all a similar enough
> layout to each other that making the site part of the variable itself
> rather than the name would be simpler and easier to add more sites
> (plus it covers the case where you have some port using one file from
> github and one from gitlab, etc).
>
> Playing with syntax ideas, maybe something like this would be easy to
> use for pprts not needing a rename -
>
> SOMEVAR+= github vim vim refs/tags/v9.0.1677
> SOMEVAR+= github vim colorschemes 22986fa2a3d2f7229efd4019fcbca411caa6afbb
>
> or with some auto-renaming (and specifying more of the path to avoid the
> extra GH_WRKSRC which I think might not be enough in some cases anyway -
> a port may have several distfiles that need to go into different base
> dirs) -
>
> SOMEVAR+= github fortran-lang fpm refs/tags/v0.7.0
> OTHERVAR+= github toml-f toml-f e49f5523e4ee67db6628618864504448fb8c8939
> vendor/toml-f
> OTHERVAR+= github urbanjost M_CLI2
> 90a1a146e19c8ad37b0469b8cbd04bc28eb67a50 vendor/M_CLI2
>
> (no idea what to use as real names instead of SOMEVAR/OTHERVAR though!)
>
> How does that sort of thing seem to you? (i.e. using the same basic idea as
> you have for submodules, but making it the standard for all gh distfiles)?
I ran with your suggestion and came up with a solution that I've named
distexpand. The idea is to use templates for commonly used,
automatically generated and therefore predictably named, stored, and
packaged dist files. 2 variables take different arguments/parts that
are 'expanded' with the template to working MASTER_SITESn and
DISTFILES.
The current configuration in the ports Makefile is done like this,
after putting distexpand.port.mk into /usr/ports/infrastructure/mk/:
MODULES += distexpand
DISTEXPAND += template account1 project1 id1(commithash/tag)
DISTEXPANDX += template account2 project2 id2(commithash/tag) targetdir
'template' is currently set up for github, gitlab, and sourcehut. You
can use multiple DISTEXPAND and DISTEXPANDX as needed. This will _not_
use up more MASTER_SITESn, as long as the template stays the same.
Regarding the naming, I'm definitely open to discuss other suggestions.
DISTEXPAND is what I've been able to think of that most clearly conveys
the use of the fragments that are expanded to a full address for
fetching the distfile. DISTEXPANDX - the last 'X' is meant to stand for
'extended' as this is the version that relocates the extracted files to
a target dir. I'm slightly partial to consider naming the variables
instead 'DISTEXPAND4' and 'DISTEXPAND5' which would remind the porter
of the number of components for each version.
For the templating, I used %account, %project, %id, %subdir as the
placeholders. Those are substituted later with :S. I'm open to
suggestions if there may be a more established pattern for placeholders
in strings in Makefile context.
This can replace GH_{ACCOUNT,PROJECT,TAGNAME,COMMIT}. Tags are detected
as such, and in that case a DISTNAME will be set to $project-$tag if
not otherwise set. In other scenarios, a DISTNAME or PKGNAME may need
to be set.
A couple of other things to note compared to before:
- GH_WRKSRC is gone without replacement. Its usefulness was
questionable.
- It includes logic that finds the first MASTER_SITESn that isn't
otherwise used, and throws an ERROR if it overruns past
MASTER_SITES9.
- Using tags is now by just proving '0.1.0' or 'v0.11.2' or other
non-commithash string (the heuristic checks for length to determine
if this is a tag or a commit hash).
- It currently uses 2 longer for-loops that are almost identical, but
one for DISTEXPAND, and the other one for DISTEXPANDX. Given the
limitations in Makefiles, I couldn't think of a way to reuse more
code there.
This doesn't need to be in a module, but this way it's easy to plug in
and experiment with.
I'm attaching the distexpand.port.mk, as well as the patch for using it
with neovim as an example. I've tested this with about 3 dozen ports
that use combinations of mostly github sites, but also a gitlab and a
sourcehut dist source [1].
[1] https://thfr.info/tmp/distexpand-ports.txt
Index: Makefile
===================================================================
RCS file: /cvs/ports/editors/neovim/Makefile,v
retrieving revision 1.37
diff -u -p -r1.37 Makefile
--- Makefile 14 Jun 2023 07:47:57 -0000 1.37
+++ Makefile 9 Aug 2023 04:53:18 -0000
@@ -11,9 +11,18 @@ EMBED_LUAJIT = No
COMMENT = continuation and extension of Vim
-GH_ACCOUNT = neovim
-GH_PROJECT = neovim
-GH_TAGNAME = v0.9.1
+DISTEXPAND += github neovim neovim v0.9.1
+
+# The versions listed here must match those in cmake.deps/CMakeLists.txt.
+# Move static deps source code under WRKDIST so that they can be patched.
+DISTEXPANDX+= github luvit luv 093a977b82077591baefe1e880d37dfa2730bd54 \
+ static-deps/luv # Apache-2.0
+DISTEXPANDX+= github LuaJIT LuaJIT 505e2c03de35e2718eef0d2d3660712e06dadf1f \
+ static-deps/luajit # MIT + public domain
+DISTEXPANDX+= github lunarmodules lua-compat-5.3 \
+ 7b783fb8efac60de8be91522d5731a9716e83d56 \
+ static-deps/lua-compat-5.3 # MIT
+STATIC_DEPS_WRKSRC = ${WRKDIST}/static-deps
REVISION = 0
# embedded luajit
@@ -23,22 +32,7 @@ CATEGORIES = editors devel
HOMEPAGE = https://neovim.io
MAINTAINER = Edd Barrett <e...@openbsd.org>
-# The versions listed here must match those in cmake.deps/CMakeLists.txt.
-LUV_VER = 093a977b82077591baefe1e880d37dfa2730bd54
-LUAJIT_VER = 505e2c03de35e2718eef0d2d3660712e06dadf1f
-LUACOMPAT_VER = v0.9
-
-MASTER_SITES0 = https://github.com/luvit/luv/archive/
-MASTER_SITES1 = https://github.com/LuaJIT/LuaJIT/archive/
-MASTER_SITES2 = https://github.com/keplerproject/lua-compat-5.3/archive/
-DISTFILES = ${DISTNAME}${EXTRACT_SUFX} \
- luv-{}${LUV_VER}${EXTRACT_SUFX}:0 \
- luajit-{}${LUAJIT_VER}${EXTRACT_SUFX}:1 \
- lua-compat-5.3-{}${LUACOMPAT_VER}${EXTRACT_SUFX}:2
-
-# Neovim: Apache 2.0 + Vim License
-# LuaJIT: MIT + public domain
-# libluv: Apache 2.0
+# Apache 2.0 + Vim License
PERMIT_PACKAGE = Yes
DEBUG_PACKAGES = ${BUILD_PACKAGES}
@@ -52,7 +46,8 @@ WANTLIB += ${MODLUA_WANTLIB}
COMPILER = base-clang ports-gcc
-MODULES = devel/cmake \
+MODULES = distexpand \
+ devel/cmake \
lang/lua \
textproc/intltool
@@ -105,15 +100,6 @@ MODPY_BUILDDEP =No
TEST_DEPENDS = shells/bash \
editors/py-neovim \
editors/py-neovim${MODPY_FLAVOR}
-
-# Move static deps source code under WRKDIST so that they can be patched.
-STATIC_DEPS_WRKSRC=${WRKDIST}/static-deps/
-post-extract:
- mkdir ${STATIC_DEPS_WRKSRC}
- mv ${WRKDIR}/LuaJIT-${LUAJIT_VER} ${STATIC_DEPS_WRKSRC}/luajit
- mv ${WRKDIR}/luv-${LUV_VER} ${STATIC_DEPS_WRKSRC}/luv
- mv ${WRKDIR}/lua-compat-5.3-${LUACOMPAT_VER:C/^v//} \
- ${STATIC_DEPS_WRKSRC}/lua-compat-5.3
# Build LuaJIT (if required) and libluv as static libraries.
#
Index: distinfo
===================================================================
RCS file: /cvs/ports/editors/neovim/distinfo,v
retrieving revision 1.20
diff -u -p -r1.20 distinfo
--- distinfo 5 Jun 2023 13:22:15 -0000 1.20
+++ distinfo 9 Aug 2023 04:53:18 -0000
@@ -1,8 +1,8 @@
-SHA256 (lua-compat-5.3-v0.9.tar.gz) =
rQVUDS2WpIclu3mh3vNc9mUqTi7CY3biYXyM4rqm9BY=
-SHA256 (luajit-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) =
Z8iDmbkBoi6aI29Ld+b+Oa8A9rcUTOndb1EUHZIfEHY=
-SHA256 (luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) =
Iis4tkJfCSYhjhTn2oFIH93m+WYMH+rCWlPm+1LohuY=
-SHA256 (neovim-0.9.1.tar.gz) = jbF8Kh9HdtzaAOWUieoNmLqC99Go6gMoHWQOWNijoA4=
-SIZE (lua-compat-5.3-v0.9.tar.gz) = 53599
-SIZE (luajit-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) = 1075264
-SIZE (luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) = 127241
-SIZE (neovim-0.9.1.tar.gz) = 11567358
+SHA256 (LuaJIT-LuaJIT-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) =
Z8iDmbkBoi6aI29Ld+b+Oa8A9rcUTOndb1EUHZIfEHY=
+SHA256
(lunarmodules-lua-compat-5.3-7b783fb8efac60de8be91522d5731a9716e83d56.tar.gz) =
6JL6QFiIxNX1DS4fX3EXDw6sTmBnWdjJgLXbxep/yq8=
+SHA256 (luvit-luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) =
Iis4tkJfCSYhjhTn2oFIH93m+WYMH+rCWlPm+1LohuY=
+SHA256 (neovim-neovim-v0.9.1.tar.gz) =
jbF8Kh9HdtzaAOWUieoNmLqC99Go6gMoHWQOWNijoA4=
+SIZE (LuaJIT-LuaJIT-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) = 1075264
+SIZE
(lunarmodules-lua-compat-5.3-7b783fb8efac60de8be91522d5731a9716e83d56.tar.gz) =
53632
+SIZE (luvit-luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) = 127241
+SIZE (neovim-neovim-v0.9.1.tar.gz) = 11567358
Index: patches/patch-static-deps_luajit_src_lj_arch_h
===================================================================
RCS file:
/cvs/ports/editors/neovim/patches/patch-static-deps_luajit_src_lj_arch_h,v
retrieving revision 1.3
diff -u -p -r1.3 patch-static-deps_luajit_src_lj_arch_h
--- patches/patch-static-deps_luajit_src_lj_arch_h 4 Jul 2022 14:49:03
-0000 1.3
+++ patches/patch-static-deps_luajit_src_lj_arch_h 9 Aug 2023 04:53:18
-0000
@@ -5,7 +5,7 @@ ignore the gcc version check ifdef hell
Index: static-deps/luajit/src/lj_arch.h
--- static-deps/luajit/src/lj_arch.h.orig
+++ static-deps/luajit/src/lj_arch.h
-@@ -432,7 +432,7 @@
+@@ -442,7 +442,7 @@
/* -- Checks for requirements --------------------------------------------- */
/* Check for minimum required compiler versions. */
# DISTEXPAND: basic form, 4 elements, doesn't move files after extraction
# Syntax:
# DISTEXPAND += template account project id(commit/tag) # license
#
# Examples:
# DISTEXPAND += github vim vim v9.0.1677 # VIM License / donation-ware
# DISTEXPAND += gitlab Mr_Goldberg goldberg_emulator \
# 475342f0d8b2bd7eb0d93bd7cfdd61e3ae7cda24 # LGPLv3
DISTEXPAND ?=
# DISTEXPANDX: extended, 5 elements, last one is target directory to move files
to
# Syntax:
# DISTEXPANDX += template account project id(commit/tag) targetdir #
license
#
# Example:
# DISTEXPANDX += github FNA-XNA FNA.NetStub
ebff244074bb3c28aeeb8cf7b383b5a029d7e28d \
# ../FNA.NetStub # Ms-PL
#
# Caveats:
# If DISTNAME isn't set and a tag is used for id, project-tag will be
# set as DISTNAME.
# If neither DISTNAME nor WRKDIST/WRKSRC are set in Makefile, this might
# extract into WRKDIR.
DISTEXPANDX ?=
# Templates:
# %account
# %project
# %id: commit hash/tag
# GitHub
MASTER_SITES_github = https://github.com/
DISTFILES_github =
%account-%project-{%account/%project/archive/%subdir}%id.tar.gz
HOMEPAGE_github = https://github.com/%account/%project
# GitLab
MASTER_SITES_gitlab = https://gitlab.com/
DISTFILES_gitlab =
%account-%project-{%account/%project/-/archive/%id/}%id.tar.gz
HOMEPAGE_gitlab = https://gitlab.com/%account/%project
# sr.ht
MASTER_SITES_sourcehut= https://git.sr.ht/
DISTFILES_sourcehut =
%account-%project-{~%account/%project/archive/}%id.tar.gz
HOMEPAGE_sourcehut = https://git.sr.ht/~%account/%project
# needed to work with traditional MASTER_SITES + DISTNAME
.if defined(DISTNAME) && defined(MASTER_SITES)
DISTFILES ?= ${DISTNAME}${EXTRACT_SUFX}
.endif
# abort if overrun_flag
_overrun_flag =
# DISTEXPAND (basic template)
.if !empty(DISTEXPAND)
. for _template _account _project _id in ${DISTEXPAND}
# check if _template is valid and no overrun so far
. if empty(MASTER_SITES_${_template})
ERRORS += "Fatal: invalid choice for distexpand: ${_template}"
. elif empty(_overrun_flag)
_template_master_sitesn =
# check which MASTER_SITESn to use
. for _n in 0 1 2 3 4 5 6 7 8 9 _overrun
. if ("${_n}" == "_overrun") && empty(_template_master_sitesn)
ERRORS += "Fatal: too many MASTER_SITESn needed"
_overrun_flag = 1
. else
. if empty(_template_master_sitesn)
# reuse the MASTER_SITESn where possible
. if empty(MASTER_SITES${_n}) || "${MASTER_SITES${_n}}" ==
"${MASTER_SITES_${_template}}"
_template_master_sitesn = ${_n}
. endif
. endif
. endif
. endfor
# modifications based on _id characteristics
# _id: tag or commit hash? heuristic using length > 39 characters
_subdir =
_id_length_test = ${_id:M????????????????????????????????????????}
. if empty(_id_length_test)
# set DISTNAME if not done by the port, based on _project + _id tag
DISTNAME ?= ${_project}-${_id:S/^v//}
. if "MASTER_SITES_${_template}" == "MASTER_SITES_github"
_subdir = refs/tags/
. endif
. endif
# set the variables for bsd.port.mk
. if empty(_overrun_flag)
MASTER_SITES${_template_master_sitesn} = ${MASTER_SITES_${_template}}
DISTFILES +:=
${DISTFILES_${_template}:S/%account/${_account}/g:S/%project/${_project}/g:S/%id/${_id}/g:S/%subdir/${_subdir}/g}:${_template_master_sitesn}
. if !empty(HOMEPAGE_${_template})
HOMEPAGE ?=
${HOMEPAGE_${_template}:S/%account/${_account}/g:S/%project/${_project}/g}
. endif
. endif
. endif
. endfor
.endif
# DISTEXPANDX (extended template with target directory)
.if empty(_overrun_flag) && !empty(DISTEXPANDX)
. for _template _account _project _id _targetdir in ${DISTEXPANDX}
# check if _template is valid
. if empty(MASTER_SITES_${_template})
ERRORS += "Fatal: invalid choice for distexpand: ${_template}"
. elif empty(_overrun_flag)
_template_master_sitesn =
# check which MASTER_SITESn to use
. for _n in 0 1 2 3 4 5 6 7 8 9 _overrun
. if "${_n}" == "_overrun" && empty(_template_master_sitesn)
ERRORS += "Fatal: too many MASTER_SITESn needed"
_overrun_flag = 1
. else
. if empty(_template_master_sitesn)
# reuse the MASTER_SITESn where possible
. if empty(MASTER_SITES${_n}) || "${MASTER_SITES${_n}}" ==
"${MASTER_SITES_${_template}}"
_template_master_sitesn = ${_n}
. endif
. endif
. endif
. endfor
# modifications based on _id characteristics
# _id: tag or commit hash? heuristic using length > 39 characters
_subdir =
_id_length_test = ${_id:M????????????????????????????????????????}
. if empty(_id_length_test)
# set DISTNAME if not done by the port, based on _project + _id tag
DISTNAME ?= ${_project}-${_id:S/^v//}
. if "MASTER_SITES_${_template}" == "MASTER_SITES_github"
_subdir = refs/tags/
. endif
. endif
# set the variables for bsd.port.mk
. if empty(_overrun_flag)
MASTER_SITES${_template_master_sitesn} = ${MASTER_SITES_${_template}}
DISTFILES +=
${DISTFILES_${_template}:S/%account/${_account}/g:S/%project/${_project}/g:S/%id/${_id}/g:S/%subdir/${_subdir}/g}:${_template_master_sitesn}
. if !empty(HOMEPAGE_${_template})
HOMEPAGE ?=
${HOMEPAGE_${_template}:S/%account/${_account}/g:S/%project/${_project}/g}
. endif
. endif
# fix/add post-extract target
MODDISTEXPAND_post-extract += \
[[ -d ${WRKSRC}/${_targetdir} ]] && rmdir ${WRKSRC}/${_targetdir} \
|| mkdir -p `dirname ${WRKSRC}/${_targetdir}` ; \
mv
${WRKDIR}/${_project}-${_id:S/refs\/tags\///:S/^v//} ${WRKSRC}/${_targetdir} ;
. endif
. endfor
.endif
