Hi,
GitHub projects using submodules seems to be a common enough case that
I'm wondering if it would be helpful to simplify and standardize it. I
got the idea when I saw how the modules for go and cargo pull in their
modules or crates, respectively.
The current state is that projects with submodules are either manually
packaged and hosted, or a combination of of MASTER_SITES0 through 9,
manual DISTFILES setting, and post-extract movement are used. What both
have in common is at least some maintenance burden and a significant
barrier for especially for those newer to the ports system.
The diff below proposes a module github.port.mk that aims to simplify
this process. At its core, the main way to manage submodules is to add
a line like the following:
GH_SUBMODULES+= luvit luv 093a977b82077591baefe1e880d37dfa2730bd54 \
luv # Apache-2.0
This way the GitHub tarball from the commit 093a977b from the project
luv by account luvit is added to the distfiles, and then the extracted
files are moved to ${GHSM_WRKSRC}/luv (GHSM_WRKSRC defaults to
${WRKSRC}). I saw license comments used with MODCARGO_CRATES and think
this would also be a good location for the submodule licenses.
This way, the multi-step setup and maintenance in MASTER_SITESX,
DISTFILES, and post-extract is reduced to essentially one location.
The current draft hijacks post-extract target, but it would be easy to
add this to _post-extract-finalize in bsd.port.mk similar to how the
post-extract commands from modules are handled, if this is of interest.
I'm attaching the github.port.mk file, as well as 3 diffs to show how
the simplified Makefiles look with this. A quick grep through the ports
tree shows there are at least a couple of dozen ports that could
benefit from a rework...
# List of static dependencies. The format is:
# account project tag_or_commit target_dir # license
# Example:
# GH_SUBMODULES += moonlight-stream moonlight-common-c \
# c9426a6a71c4162e65dde8c0c71a25f1dbca46ba \
# third-party/moonlight-common-c # GPL-v3.0+
GH_SUBMODULES ?=
# Master site for github tarballs
GH_MASTER_SITES ?= https://github.com/
# where submodule distfiles will be stored
GHSM_DIST_SUBDIR ?= gh-submodules
# where submodules will be extracted to
GHSM_WRKSRC ?= ${WRKSRC}
# Grab submodules by default with MASTER_SITES8. (Don't use 9 to avoid collision
# with language-specific mechanisms, like devel/cargo or lang/go.)
GHSM_MASTER_SITESN ?= 8
MASTER_SITES${GHSM_MASTER_SITESN} ?= ${GH_MASTER_SITES}
# Default GitHub distfile suffix
GH_SUFX ?= .tar.gz
.if defined(DISTNAME)
DISTFILES ?= ${DISTNAME}${EXTRACT_SUFX}
.elif !empty(GH_ACCOUNT) && !empty(GH_PROJECT)
DISTFILES ?= ${GH_DISTFILE}
.endif
.for _ghaccount _ghproject _ghtagcommit _targetdir in ${GH_SUBMODULES}
DISTFILES +=
${GHSM_DIST_SUBDIR}/{}${_ghaccount}/${_ghproject}/archive/${_ghtagcommit}${GH_SUFX}:${GHSM_MASTER_SITESN}
.endfor
# post-extract target for moving the submodules to the target directories
GHSM_post-extract =
.for _ghaccount _ghproject _ghtagcommit _targetdir in ${GH_SUBMODULES}
GHSM_post-extract += \
test -d ${GHSM_WRKSR}/${_targetdir} || rm -rf
${GHSM_WRKSRC}/${_targetdir} ; \
mv ${WRKDIR}/${_ghproject}-${_ghtagcommit} ${GHSM_WRKSRC}/${_targetdir}
;
.endfor
# XXX: would best belong in _post-extract-finalize in bsd.port.mk rather than
# hijacking post-extract here
post-extract:
@${ECHO_MSG} "moving GitHub submodules to ${GHSM_WRKSRC}" ;
mkdir -p ${GHSM_WRKSRC} ;
${GHSM_post-extract}
Index: Makefile
===================================================================
RCS file: /cvs/ports/editors/neovim/Makefile,v
retrieving revision 1.37
diff -u -p -r1.37 Makefile
--- Makefile 14 Jun 2023 07:47:57 -0000 1.37
+++ Makefile 5 Aug 2023 19:06:09 -0000
@@ -23,22 +23,20 @@ CATEGORIES = editors devel
HOMEPAGE = https://neovim.io
MAINTAINER = Edd Barrett <e...@openbsd.org>
+# Move static deps source code under WRKDIST so that they can be patched.
+STATIC_DEPS_WRKSRC = ${WRKDIST}/static-deps
+GHSM_WRKSRC = ${STATIC_DEPS_WRKSRC}
+
# The versions listed here must match those in cmake.deps/CMakeLists.txt.
-LUV_VER = 093a977b82077591baefe1e880d37dfa2730bd54
-LUAJIT_VER = 505e2c03de35e2718eef0d2d3660712e06dadf1f
-LUACOMPAT_VER = v0.9
-
-MASTER_SITES0 = https://github.com/luvit/luv/archive/
-MASTER_SITES1 = https://github.com/LuaJIT/LuaJIT/archive/
-MASTER_SITES2 = https://github.com/keplerproject/lua-compat-5.3/archive/
-DISTFILES = ${DISTNAME}${EXTRACT_SUFX} \
- luv-{}${LUV_VER}${EXTRACT_SUFX}:0 \
- luajit-{}${LUAJIT_VER}${EXTRACT_SUFX}:1 \
- lua-compat-5.3-{}${LUACOMPAT_VER}${EXTRACT_SUFX}:2
-
-# Neovim: Apache 2.0 + Vim License
-# LuaJIT: MIT + public domain
-# libluv: Apache 2.0
+GH_SUBMODULES+= luvit luv 093a977b82077591baefe1e880d37dfa2730bd54 \
+ luv # Apache-2.0
+GH_SUBMODULES+= LuaJIT LuaJIT 505e2c03de35e2718eef0d2d3660712e06dadf1f \
+ luajit # MIT + public domain
+GH_SUBMODULES+= lunarmodules lua-compat-5.3 \
+ 7b783fb8efac60de8be91522d5731a9716e83d56 \
+ lua-compat-5.3 # MIT
+
+# Apache 2.0 + Vim License
PERMIT_PACKAGE = Yes
DEBUG_PACKAGES = ${BUILD_PACKAGES}
@@ -106,15 +104,6 @@ TEST_DEPENDS = shells/bash \
editors/py-neovim \
editors/py-neovim${MODPY_FLAVOR}
-# Move static deps source code under WRKDIST so that they can be patched.
-STATIC_DEPS_WRKSRC=${WRKDIST}/static-deps/
-post-extract:
- mkdir ${STATIC_DEPS_WRKSRC}
- mv ${WRKDIR}/LuaJIT-${LUAJIT_VER} ${STATIC_DEPS_WRKSRC}/luajit
- mv ${WRKDIR}/luv-${LUV_VER} ${STATIC_DEPS_WRKSRC}/luv
- mv ${WRKDIR}/lua-compat-5.3-${LUACOMPAT_VER:C/^v//} \
- ${STATIC_DEPS_WRKSRC}/lua-compat-5.3
-
# Build LuaJIT (if required) and libluv as static libraries.
#
# We opted not to create a libluv port because it must be built for a specific
@@ -197,4 +186,5 @@ do-test:
${MAKE_PROGRAM} -C ${WRKSRC}/test/old/testdir \
HOME=${WRKBUILD} NVIM_PRG=${WRKBUILD}/bin/nvim ${MAKE_FLAGS}
+.include "/usr/ports/mystuff/github.port.mk"
.include <bsd.port.mk>
Index: distinfo
===================================================================
RCS file: /cvs/ports/editors/neovim/distinfo,v
retrieving revision 1.20
diff -u -p -r1.20 distinfo
--- distinfo 5 Jun 2023 13:22:15 -0000 1.20
+++ distinfo 5 Aug 2023 19:06:09 -0000
@@ -1,8 +1,8 @@
-SHA256 (lua-compat-5.3-v0.9.tar.gz) =
rQVUDS2WpIclu3mh3vNc9mUqTi7CY3biYXyM4rqm9BY=
-SHA256 (luajit-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) =
Z8iDmbkBoi6aI29Ld+b+Oa8A9rcUTOndb1EUHZIfEHY=
-SHA256 (luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) =
Iis4tkJfCSYhjhTn2oFIH93m+WYMH+rCWlPm+1LohuY=
+SHA256
(gh-submodules/LuaJIT/LuaJIT/archive/505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz)
= Z8iDmbkBoi6aI29Ld+b+Oa8A9rcUTOndb1EUHZIfEHY=
+SHA256
(gh-submodules/lunarmodules/lua-compat-5.3/archive/7b783fb8efac60de8be91522d5731a9716e83d56.tar.gz)
= 6JL6QFiIxNX1DS4fX3EXDw6sTmBnWdjJgLXbxep/yq8=
+SHA256
(gh-submodules/luvit/luv/archive/093a977b82077591baefe1e880d37dfa2730bd54.tar.gz)
= Iis4tkJfCSYhjhTn2oFIH93m+WYMH+rCWlPm+1LohuY=
SHA256 (neovim-0.9.1.tar.gz) = jbF8Kh9HdtzaAOWUieoNmLqC99Go6gMoHWQOWNijoA4=
-SIZE (lua-compat-5.3-v0.9.tar.gz) = 53599
-SIZE (luajit-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) = 1075264
-SIZE (luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) = 127241
+SIZE
(gh-submodules/LuaJIT/LuaJIT/archive/505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz)
= 1075264
+SIZE
(gh-submodules/lunarmodules/lua-compat-5.3/archive/7b783fb8efac60de8be91522d5731a9716e83d56.tar.gz)
= 53632
+SIZE
(gh-submodules/luvit/luv/archive/093a977b82077591baefe1e880d37dfa2730bd54.tar.gz)
= 127241
SIZE (neovim-0.9.1.tar.gz) = 11567358
Index: patches/patch-static-deps_luajit_src_lj_arch_h
===================================================================
RCS file:
/cvs/ports/editors/neovim/patches/patch-static-deps_luajit_src_lj_arch_h,v
retrieving revision 1.3
diff -u -p -r1.3 patch-static-deps_luajit_src_lj_arch_h
--- patches/patch-static-deps_luajit_src_lj_arch_h 4 Jul 2022 14:49:03
-0000 1.3
+++ patches/patch-static-deps_luajit_src_lj_arch_h 5 Aug 2023 19:06:09
-0000
@@ -5,7 +5,7 @@ ignore the gcc version check ifdef hell
Index: static-deps/luajit/src/lj_arch.h
--- static-deps/luajit/src/lj_arch.h.orig
+++ static-deps/luajit/src/lj_arch.h
-@@ -432,7 +432,7 @@
+@@ -442,7 +442,7 @@
/* -- Checks for requirements --------------------------------------------- */
/* Check for minimum required compiler versions. */
Index: Makefile
===================================================================
RCS file: /cvs/ports/games/fs2open/Makefile,v
retrieving revision 1.17
diff -u -p -r1.17 Makefile
--- Makefile 22 Feb 2023 06:51:24 -0000 1.17
+++ Makefile 5 Aug 2023 19:05:53 -0000
@@ -22,17 +22,10 @@ WANTLIB += ${COMPILER_LIBCXX} SDL2 avcod
WANTLIB += jansson jpeg ${MODLUA_WANTLIB} m openal png swresample swscale
WANTLIB += vulkan z
-RPAVLIK_CMAKE_MOD = 7cef9577d6fc35057ea57f46b4986a8a28aeff50
-LIBROCKET = ecd648a43aff8a9f3daf064d75ca5725237d5b38
-
-MASTER_SITES0 = https://github.com/asarium/cmake-modules/archive/
-MASTER_SITES1 = https://github.com/asarium/libRocket/archive/
-
-DISTFILES = ${DISTNAME}${EXTRACT_SUFX} \
- ${RPAVLIK_CMAKE_MOD}.tar.gz:0 \
- ${LIBROCKET}.tar.gz:1
-
-DIST_SUBDIR = fs2open-${V}
+GH_SUBMODULES+= asarium cmake-modules
7cef9577d6fc35057ea57f46b4986a8a28aeff50 \
+ cmake/external/rpavlik-cmake-modules
+GH_SUBMODULES+= asarium libRocket
ecd648a43aff8a9f3daf064d75ca5725237d5b38 \
+ lib/libRocket
# c++11
COMPILER = base-clang ports-gcc
@@ -59,14 +52,8 @@ CONFIGURE_ARGS = -DCMAKE_INSTALL_PREFIX=
NO_TEST = Yes
# remove bundled SDL
-post-extract:
+pre-configure:
rm -rf ${WRKSRC}/lib/libsdl
- rmdir ${WRKSRC}/cmake/external/rpavlik-cmake-modules
- ln -s ${WRKDIR}/cmake-modules-${RPAVLIK_CMAKE_MOD} \
- ${WRKSRC}/cmake/external/rpavlik-cmake-modules
- rmdir ${WRKSRC}/lib/libRocket
- ln -s ${WRKDIR}/libRocket-${LIBROCKET} \
- ${WRKSRC}/lib/libRocket
do-gen:
${SUBST_CMD} ${WRKSRC}/CMakeLists.txt
@@ -74,4 +61,5 @@ do-gen:
post-install:
${SUBST_CMD} -c -m 755 ${FILESDIR}/fs2open ${PREFIX}/bin/fs2open
+.include "/usr/ports/mystuff/github.port.mk"
.include <bsd.port.mk>
Index: distinfo
===================================================================
RCS file: /cvs/ports/games/fs2open/distinfo,v
retrieving revision 1.7
diff -u -p -r1.7 distinfo
--- distinfo 22 Feb 2023 06:51:24 -0000 1.7
+++ distinfo 5 Aug 2023 19:05:53 -0000
@@ -1,6 +1,6 @@
-SHA256 (fs2open-23.0.0/7cef9577d6fc35057ea57f46b4986a8a28aeff50.tar.gz) =
e2kS2gGlbuyljBP4u7B7YSCvjwRyIxZ0fcgTKo7GWso=
-SHA256 (fs2open-23.0.0/ecd648a43aff8a9f3daf064d75ca5725237d5b38.tar.gz) =
JcjDM2xciQqxzt/90Z17tubwjVy+HASUyiAs+5h/Tfc=
-SHA256 (fs2open-23.0.0/fs2open-23.0.0.tar.gz) =
AZok/u73pI+BcPAznFUwEisya9iPtz6ddRW/6eU4Gqg=
-SIZE (fs2open-23.0.0/7cef9577d6fc35057ea57f46b4986a8a28aeff50.tar.gz) = 267464
-SIZE (fs2open-23.0.0/ecd648a43aff8a9f3daf064d75ca5725237d5b38.tar.gz) = 2417905
-SIZE (fs2open-23.0.0/fs2open-23.0.0.tar.gz) = 12210430
+SHA256 (fs2open-23.0.0.tar.gz) = AZok/u73pI+BcPAznFUwEisya9iPtz6ddRW/6eU4Gqg=
+SHA256
(gh-submodules/asarium/cmake-modules/archive/7cef9577d6fc35057ea57f46b4986a8a28aeff50.tar.gz)
= e2kS2gGlbuyljBP4u7B7YSCvjwRyIxZ0fcgTKo7GWso=
+SHA256
(gh-submodules/asarium/libRocket/archive/ecd648a43aff8a9f3daf064d75ca5725237d5b38.tar.gz)
= JcjDM2xciQqxzt/90Z17tubwjVy+HASUyiAs+5h/Tfc=
+SIZE (fs2open-23.0.0.tar.gz) = 12210430
+SIZE
(gh-submodules/asarium/cmake-modules/archive/7cef9577d6fc35057ea57f46b4986a8a28aeff50.tar.gz)
= 267464
+SIZE
(gh-submodules/asarium/libRocket/archive/ecd648a43aff8a9f3daf064d75ca5725237d5b38.tar.gz)
= 2417905
Index: Makefile
===================================================================
RCS file: /cvs/ports/games/fna/Makefile,v
retrieving revision 1.17
diff -u -p -r1.17 Makefile
--- Makefile 15 Jul 2023 23:24:35 -0000 1.17
+++ Makefile 5 Aug 2023 19:10:00 -0000
@@ -7,21 +7,16 @@ HOMEPAGE = https://fna-xna.github.io/
MAINTAINER = Thomas Frohwein <t...@openbsd.org>
# MS-PL, includes lzxdecoder (dual MSPL/LGPL) and Mono.Xna (MIT)
-# zlib (Vorbisfile-CS)
PERMIT_PACKAGE = Yes
-NETSTUB = ebff244074bb3c28aeeb8cf7b383b5a029d7e28d
-VORBISFILE = 521c8532f03b3608a141b36d7c1343e816b46cb1
+GH_SUBMODULES+= FNA-XNA FNA.NetStub \
+ ebff244074bb3c28aeeb8cf7b383b5a029d7e28d \
+ ../FNA.NetStub # Ms-PL
+GH_SUBMODULES+= flibitijibibo Vorbisfile-CS \
+ 521c8532f03b3608a141b36d7c1343e816b46cb1 \
+ Vorbisfile-CS # zlib
MASTER_SITES = https://github.com/FNA-XNA/FNA/releases/download/${V}/
-MASTER_SITES0 = https://github.com/FNA-XNA/FNA.NetStub/archive/
-MASTER_SITES1 = https://github.com/flibitijibibo/Vorbisfile-CS/archive/
-
-DISTFILES = ${DISTNAME}${EXTRACT_SUFX} \
- ${NETSTUB}.tar.gz:0 \
- ${VORBISFILE}.tar.gz:1
-
-DIST_SUBDIR = fna-${V}
EXTRACT_SUFX = .zip
MODULES = lang/mono
@@ -31,14 +26,9 @@ RUN_DEPENDS = audio/faudio \
WRKDIST = ${WRKDIR}/FNA
NO_TEST = Yes
ALL_TARGET = release
-SUBST_VARS += VORBISFILE
-
-post-extract:
- ln -sf ${WRKDIR}/FNA.NetStub-${NETSTUB} ${WRKDIR}/FNA.NetStub
do-gen:
cp ${FILESDIR}/FNA.Settings.props ${WRKSRC}/
- ${SUBST_CMD} ${WRKSRC}/FNA.Settings.props
# need to rm bin,obj before FNA.sln to build with FNA.Settings.props
do-build:
@@ -63,4 +53,5 @@ do-install:
${PREFIX}/share/FNA/
${INSTALL_DATA} ${FILESDIR}/FNA.dll.config ${PREFIX}/share/FNA/
+.include "/usr/ports/mystuff/github.port.mk"
.include <bsd.port.mk>
Index: distinfo
===================================================================
RCS file: /cvs/ports/games/fna/distinfo,v
retrieving revision 1.15
diff -u -p -r1.15 distinfo
--- distinfo 15 Jul 2023 23:24:35 -0000 1.15
+++ distinfo 5 Aug 2023 19:10:00 -0000
@@ -1,6 +1,6 @@
-SHA256 (fna-23.07/521c8532f03b3608a141b36d7c1343e816b46cb1.tar.gz) =
Cj2HaaOFBDsXtb/LPiedFdKzTb59XU9nMU7fuR9C6gc=
-SHA256 (fna-23.07/ebff244074bb3c28aeeb8cf7b383b5a029d7e28d.tar.gz) =
34qgjqOOFTODn6zSKEs6Gt9jbdaJi9MR/7rw12tlXIA=
-SHA256 (fna-23.07/fna-2307.zip) = 6ZmbKqwz0X+RKXGvQTgqsZtLDtPvdcNURGoROTZye0s=
-SIZE (fna-23.07/521c8532f03b3608a141b36d7c1343e816b46cb1.tar.gz) = 5607
-SIZE (fna-23.07/ebff244074bb3c28aeeb8cf7b383b5a029d7e28d.tar.gz) = 19825
-SIZE (fna-23.07/fna-2307.zip) = 3916766
+SHA256 (fna-2307.zip) = 6ZmbKqwz0X+RKXGvQTgqsZtLDtPvdcNURGoROTZye0s=
+SHA256
(gh-submodules/FNA-XNA/FNA.NetStub/archive/ebff244074bb3c28aeeb8cf7b383b5a029d7e28d.tar.gz)
= 34qgjqOOFTODn6zSKEs6Gt9jbdaJi9MR/7rw12tlXIA=
+SHA256
(gh-submodules/flibitijibibo/Vorbisfile-CS/archive/521c8532f03b3608a141b36d7c1343e816b46cb1.tar.gz)
= Cj2HaaOFBDsXtb/LPiedFdKzTb59XU9nMU7fuR9C6gc=
+SIZE (fna-2307.zip) = 3916766
+SIZE
(gh-submodules/FNA-XNA/FNA.NetStub/archive/ebff244074bb3c28aeeb8cf7b383b5a029d7e28d.tar.gz)
= 19825
+SIZE
(gh-submodules/flibitijibibo/Vorbisfile-CS/archive/521c8532f03b3608a141b36d7c1343e816b46cb1.tar.gz)
= 5607
Index: files/FNA.Settings.props
===================================================================
RCS file: /cvs/ports/games/fna/files/FNA.Settings.props,v
retrieving revision 1.1.1.1
diff -u -p -r1.1.1.1 FNA.Settings.props
--- files/FNA.Settings.props 16 Nov 2020 23:06:07 -0000 1.1.1.1
+++ files/FNA.Settings.props 5 Aug 2023 19:10:00 -0000
@@ -1,6 +1,6 @@
<Project DefaultTargets="Build" ToolsVersion="4.0"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003";>
<ItemGroup>
<Compile Include="lib/SDL2-CS/src/SDL2_image.cs" />
- <Compile Include="../Vorbisfile-CS-${VORBISFILE}/Vorbisfile.cs"
/>
+ <Compile Include="Vorbisfile-CS/Vorbisfile.cs" />
</ItemGroup>
</Project>
