On Mon, Oct 11, 2021 at 11:46:53AM +0200, Sebastien Marie wrote: > What I mean is it might helps too much to package experimental and > possibly unmaintained code.
I understand, on the other hand porting stuff helps development; we can still judge/decide on import as well as later on whether premature ports should be imported or rotten ports should be removed. > My point is we are following closely rustc release cycle, and regulary > crates are breaking. So it could add work for rustc maintainer (me) to > deal with unmaintained forked crates. I'd say if those git-crate based ports break they break. We can try fixing stuff but if it doesn't work can be marked BROKEN or removed. > Maybe we could try to names the destination with the hash included ? > But I am unsure how cargo is fluent to searching crates in vendor > directory. I will try to test it. But it would be fine without that > too. Not sure, either. For now, this diff does what I want and it the topic is hackish enough already. > Just a question about MODCARGO_CRATES_GIT format. Does it is the right > one ? > > Currently, it is defined as: > > MODCARGO_CRATES_GIT += name version url > MODCARGO_CRATES_GIT += miniaudio 0.10.0 > https://github.com/klemensn/miniaudio-rs/archive/4d429a322513803992cfd03d7b3a487a3465a504 > > And we are doing parse in the module to extract the path > (author/project/archive/hash) and the short commit id. That's how the last diff does is, yes, and it works. > Do an alternate format might helps more ? > > MODCARGO_CRATES_GIT += name version project hash > MODCARGO_CRATES_GIT += miniaudio 0.10.0 klemensn/miniaudio-rs > 4d429a322513803992cfd03d7b3a487a3465a504 > > _MODCARGO_CRATE_PATH_${_cratename} = ${_crateproject}/archive > _MODCARGO_CRATE_COMMIT_${_cratename} = ${_cratehash:C/(........).*/\1/} > _MODCARGO_CRATE_DIR_${_cratename} = > ${_crateproject:C/^[^\/]+\///}-${_cratehash} > > Or maybe: > MODCARGO_CRATES_GIT += name version provider project hash > MODCARGO_CRATES_GIT += miniaudio 0.10.0 github > klemensn/miniaudio-rs 4d429a322513803992cfd03d7b3a487a3465a504 > > (in order to support at some point alternate provider) Good idea, I implemented your first suggestion. We can still support other providers later on, so I keep it simple enough for now, but the separation of the commit ID does indeed help, thanks. Diff below also incorporates your follow-up mail's feedback regarding metadata generation, thanks. I have not however run into download issues as you described them. Index: cargo.port.mk =================================================================== RCS file: /cvs/ports/devel/cargo/cargo.port.mk,v retrieving revision 1.24 diff -u -p -r1.24 cargo.port.mk --- cargo.port.mk 11 Sep 2021 07:13:13 -0000 1.24 +++ cargo.port.mk 11 Oct 2021 13:06:05 -0000 @@ -42,10 +42,15 @@ _MODCARGO_DIST_SUBDIR = ${MODCARGO_DIST_ .endif # Use MASTER_SITES9 to grab crates by default. -# Could be changed by setting MODCARGO_MASTER_SITESN. MODCARGO_MASTER_SITESN ?= 9 MASTER_SITES${MODCARGO_MASTER_SITESN} ?= ${MASTER_SITES_CRATESIO} +# Use MASTER_SITES8 to grab crates as GitHub tarballs by default. +MODCARGO_MASTER_SITES_GITN ?= 8 +# This assumes that all git crates come from the same site +# or that all sites support the same URL/download API. +MASTER_SITES${MODCARGO_MASTER_SITES_GITN} ?= https://github.com/ + # per crates options MODCARGO_CRATES_SQLITE3_BUNDLED ?= No @@ -54,6 +59,16 @@ MODCARGO_CRATES_SQLITE3_BUNDLED ?= No DISTFILES += ${_MODCARGO_DIST_SUBDIR}${_cratename}-${_cratever}.tar.gz{${_cratename}/${_cratever}/download}:${MODCARGO_MASTER_SITESN} .endfor +.for _cratename _cratever _crateurl _cratecommit in ${MODCARGO_CRATES_GIT} +# strip leading master site, e.g. https://github.com/author/project -> author/project +_MODCARGO_CRATE_REPO_${_cratename} = ${_crateurl:${MASTER_SITES${MODCARGO_MASTER_SITES_GITN}}%=} +# create downloadable path, e.g. author/project hash -> author/project/archive/hash +_MODCARGO_CRATE_PATH_${_cratename} = ${_MODCARGO_CRATE_REPO_${_cratename}}/archive/${_cratecommit} +# keep hash in filename (from bsd.port.mk's GH_COMMIT code) +_MODCARGO_CRATE_ARCHIVE_${_cratename} = ${_cratename}-${_cratever}-${_cratecommit:C/(........).*/\1/} +DISTFILES += ${_MODCARGO_DIST_SUBDIR}${_MODCARGO_CRATE_ARCHIVE_${_cratename}}{${_MODCARGO_CRATE_PATH_${_cratename}}}.tar.gz:${MODCARGO_MASTER_SITES_GITN} +.endfor + # post-extract target for preparing crates directory. # It will put all crates in the local crates directory. MODCARGO_post-extract = \ @@ -63,6 +78,15 @@ MODCARGO_post-extract = \ MODCARGO_post-extract += \ mv ${WRKDIR}/${_cratename}-${_cratever} ${MODCARGO_VENDOR_DIR}/${_cratename}-${_cratever} ; .endfor +.for _cratename _cratever _craterepo _cratecommit in ${MODCARGO_CRATES_GIT} +# strip author and turn path into unique directory name, e.g. author/project hash -> project-hash +_MODCARGO_CRATE_DIR_${_cratename} = ${_MODCARGO_CRATE_REPO_${_cratename}:T}-${_cratecommit} +# do not move a commit based directory as it may contain multiple crates, e.g. +# druid-d815bc... contains druid-0.7.0, druid-derive-0.4.0 and druid-shell-0.7.0 +MODCARGO_post-extract += \ + ln -s ${WRKDIR}/${_MODCARGO_CRATE_DIR_${_cratename}} \ + ${MODCARGO_VENDOR_DIR}/${_cratename}-${_cratever} ; +.endfor # post-extract target to provide clean environment for specific crates # in order to avoid rebuilding libraries from source behind us. @@ -179,6 +203,14 @@ MODCARGO_post-patch += \ ${MODCARGO_VENDOR_DIR}/${_cratename}-${_cratever} ; .endfor +.for _cratename _cratever _crateurlUNUSED _cratecommitUNUSED in ${MODCARGO_CRATES_GIT} +MODCARGO_post-patch += \ + ${ECHO_MSG} "[modcargo] Generating metadata for ${_MODCARGO_CRATE_ARCHIVE_${_cratename}}" ; \ + ${LOCALBASE}/bin/cargo-generate-vendor \ + ${FULLDISTDIR}/${_MODCARGO_DIST_SUBDIR}${_MODCARGO_CRATE_ARCHIVE_${_cratename}}.tar.gz \ + ${MODCARGO_VENDOR_DIR}/${_cratename}-${_cratever} ; +.endfor + # configure hook. Place a config file for overriding crates-io index by # local source directory, and set compilation options (based on release). # Enabled by use of "CONFIGURE_STYLE=cargo". @@ -351,8 +383,13 @@ modcargo-gen-crates: extract @echo '# run: make modcargo-gen-crates-licenses' @awk ' /^name = / { n=$$3; gsub("\"", "", n); } \ /^version = / { v=$$3; gsub("\"", "", v); } \ - /^source = "registry\+https:\/\/github.com\/rust-lang\/crates\.io-index"/ \ - { print "MODCARGO_CRATES += " n " " v; }' \ + /^source = "registry\+https:\/\/github\.com\/rust-lang\/crates\.io-index"$$/ \ + { print "MODCARGO_CRATES += " n " " v; } \ + /^source = "git\+https:\/\/github\.com\/.+"$$/ \ + { s=$$3; gsub("\"", "", s); \ + sub("git\\+", "", s); sub("(\\?.*)?#", "#", s); \ + split(s, a, "#"); \ + print "MODCARGO_CRATES_GIT += " n " " v " " a[1] " " a[2]; }' \ <${MODCARGO_CARGOTOML:toml=lock} # modcargo-gen-crates-licenses will try to grab license information from downloaded crates. @@ -360,6 +397,11 @@ modcargo-gen-crates-licenses: configure @echo '# $$Open''BSD$$' .for _cratename _cratever in ${MODCARGO_CRATES} @echo -n "MODCARGO_CRATES += ${_cratename} ${_cratever} # " + @sed -ne '/^license.*=/{;s/^license.*= *"\([^"]*\)".*/\1/p;q;};$$s/^.*$$/XXX missing license/p' \ + ${MODCARGO_VENDOR_DIR}/${_cratename}-${_cratever}/Cargo.toml +.endfor +.for _cratename _cratever _crateurl _cratecommit in ${MODCARGO_CRATES_GIT} + @echo -n "MODCARGO_CRATES_GIT += ${_cratename} ${_cratever} ${_crateurl} ${_cratecommit} # " @sed -ne '/^license.*=/{;s/^license.*= *"\([^"]*\)".*/\1/p;q;};$$s/^.*$$/XXX missing license/p' \ ${MODCARGO_VENDOR_DIR}/${_cratename}-${_cratever}/Cargo.toml .endfor
