Re: RFC: svn for make fetch
On Wed, 18 Nov 2009 11:43:07 +, Thomas Sandford
wrote:
> (Sorry, Ive lost track of the higher level attributions, but)
> Eitan Adler wrote:
Creating deterministic tars (ignoring "metadeta") sounds like it
should be a solved problem by now. If it isn't then I will have to
make it my next project ;)
>>> Instead of creating tar files, create zip files and then run them
>>> through torrentzip
>>>
>>> http://www.freebsd.org/cgi/cvsweb.cgi/ports/archivers/torrentzip/
>>>
>>> Torrentzip resets the date/time on the files and directories in the
>>> zip archive so that the checksum of the file will match, no matter who
>>> builds the zip file using the same set of files.
>>>
>>> Scot
>>>
>>
>> Does such a tool exist for tar archives?
>
> Couldn't you achieve the same thing by the use of touch(1)
Probably, but my idea was at the time to provide a simple way to support
non-releases (like mplayer) and devel snapshots for which a tarball is
not or rarely rolled, with minimal change to bsd.port.mk.
Hence the abuse of FETCH_*. I found it again, I'll clean it up when I have
more time - there's more stuff in my bsd.local.mk so not a clean diff, but
the crux is this:
FETCH_BINARY=${SVN_CMD}
FETCH_ARGS=${SVN_QUIET} export
FETCH_BEFORE_ARGS=-r${SVN_REVISION}
DISABLE_SIZE=yes # screws with FETCH_BEFORE_ARGS
# cwd at this point is ${_DISTDIR}
FETCH_AFTER_ARGS=${_SVN_STAGE} && ${TAR} -C ${SVN_STAGE} -czf \
${PORTNAME}-${PORTVERSION}.tar.gz ${PORTNAME}-${PORTVERSION} \
WRKSRC?=${WRKDIR}/${SVN_ROOT}
FETCH_NO_APPEND_FILE=yes
pre-fetch:
@${MKDIR} ${SVN_STAGE}
# Once bsdtar supports remove after archiving, we don't need this anymore.
post-fetch:
@${RM} -rf ${_SVN_STAGE}
(Yes, I realize I should inject my own targets and not abuse
pre|post-fetch).
bsd.port.mk diff snippet (long line):
--- Mk/bsd.port.mk 21 Sep 2009 19:13:51 - 1.629
+++ Mk/bsd.port.mk 22 Sep 2009 01:21:26 -
@@ -3490,11 +3490,15 @@
for site in `eval $$SORTED_MASTER_SITES_CMD_TMP
${_RANDOMIZE_SITES}`; do \
${ECHO_MSG} "=> Attempting to fetch from $${site}."; \
CKSIZE=`alg=SIZE; ${DISTINFO_DATA}`; \
- case $${file} in \
- */*) ${MKDIR} $${file%/*}; \
- args="-o $${file} $${site}$${file}";; \
- *)args=$${site}$${file};; \
- esac; \
+ if test x"${FETCH_NO_APPEND_FILE}" != x"yes"; then \
+ case $${file} in \
+ */*) ${MKDIR} $${file%/*}; \
+args="-o $${file} $${site}$${file}";; \
+ *)args=$${site}$${file};; \
+ esac; \
+ else \
+ args=$${site}; \
+ fi; \
if ${SETENV} ${FETCH_ENV} ${FETCH_CMD} ${FETCH_BEFORE_ARGS}
$${args} ${FETCH_AFTER_ARGS}; then \
continue 2; \
fi; \
--
--
Mel
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
(Sorry, Ive lost track of the higher level attributions, but) Eitan Adler wrote: Creating deterministic tars (ignoring "metadeta") sounds like it should be a solved problem by now. If it isn't then I will have to make it my next project ;) Instead of creating tar files, create zip files and then run them through torrentzip http://www.freebsd.org/cgi/cvsweb.cgi/ports/archivers/torrentzip/ Torrentzip resets the date/time on the files and directories in the zip archive so that the checksum of the file will match, no matter who builds the zip file using the same set of files. Scot Does such a tool exist for tar archives? Couldn't you achieve the same thing by the use of touch(1) -- Thomas Sandford ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Wed, Nov 18, 2009 at 2:19 AM, Scot Hetzel wrote: > On Tue, Nov 17, 2009 at 3:59 PM, Eitan Adler wrote: >>> Actually, I *had* a patch that got the source from svn, tarred it >>> and checksummed it, with little modification to the do-fetch target >>> and abusing FETCH_* variables. >>> The unsolvable problem I ran in to, is that svn doesn't adjust >>> timestamps for directories on export, so the checksum for the tar >>> was always different. Hacking svn export was outside my timeframe >>> and hacking tar to grow an option that sets all created dirs to >>> a fixed time stamp, seemed too hackish, so I let it go. >> >> Creating deterministic tars (ignoring "metadeta") sounds like it >> should be a solved problem by now. If it isn't then I will have to >> make it my next project ;) >> > Instead of creating tar files, create zip files and then run them > through torrentzip > > http://www.freebsd.org/cgi/cvsweb.cgi/ports/archivers/torrentzip/ > > Torrentzip resets the date/time on the files and directories in the > zip archive so that the checksum of the file will match, no matter who > builds the zip file using the same set of files. > > Scot > Does such a tool exist for tar archives? ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Tue, Nov 17, 2009 at 3:59 PM, Eitan Adler wrote: >> Actually, I *had* a patch that got the source from svn, tarred it >> and checksummed it, with little modification to the do-fetch target >> and abusing FETCH_* variables. >> The unsolvable problem I ran in to, is that svn doesn't adjust >> timestamps for directories on export, so the checksum for the tar >> was always different. Hacking svn export was outside my timeframe >> and hacking tar to grow an option that sets all created dirs to >> a fixed time stamp, seemed too hackish, so I let it go. > > Creating deterministic tars (ignoring "metadeta") sounds like it > should be a solved problem by now. If it isn't then I will have to > make it my next project ;) > Instead of creating tar files, create zip files and then run them through torrentzip http://www.freebsd.org/cgi/cvsweb.cgi/ports/archivers/torrentzip/ Torrentzip resets the date/time on the files and directories in the zip archive so that the checksum of the file will match, no matter who builds the zip file using the same set of files. Scot ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
> Actually, I *had* a patch that got the source from svn, tarred it > and checksummed it, with little modification to the do-fetch target > and abusing FETCH_* variables. > The unsolvable problem I ran in to, is that svn doesn't adjust > timestamps for directories on export, so the checksum for the tar > was always different. Hacking svn export was outside my timeframe > and hacking tar to grow an option that sets all created dirs to > a fixed time stamp, seemed too hackish, so I let it go. Creating deterministic tars (ignoring "metadeta") sounds like it should be a solved problem by now. If it isn't then I will have to make it my next project ;) Anyway lets take your script above modify it so that it uploads the tarball to freebsd mirrors and includes the checksum as of the time the maintainer created the tar. This would be much closer to my proposal. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Mon, 16 Nov 2009 15:16:22 +0300, Dmitry Marakasov wrote: > * Eitan Adler (eitanadlerl...@gmail.com) wrote: > >> I was hoping to get a bit more of a response to a recent posting of >> mine with regard to using svn to fetch files for ports >> My proposal: >> http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html >> A summary of what has been going on: >> http://wiki.freebsd.org/EitanAdler/ports-svn > > This was discussed many times before, and this is a bad idea. > - No checksums -> security issue > - No distfile caching/mirroring -> excess traffic consumption and > horrible availability Actually, I *had* a patch that got the source from svn, tarred it and checksummed it, with little modification to the do-fetch target and abusing FETCH_* variables. The unsolvable problem I ran in to, is that svn doesn't adjust timestamps for directories on export, so the checksum for the tar was always different. Hacking svn export was outside my timeframe and hacking tar to grow an option that sets all created dirs to a fixed time stamp, seemed too hackish, so I let it go. -- Mel ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
Thomas Sandford wrote: > If it were just one port and/or just a port maintainers tool I'd agree. > But this is something that affects MULTIPLE ports. You still seem to be advocating the idea of making this a mechanism for users, in spite of the fact that numerous people have said that idea is not likely to ever be approved. Given the universe where this feature is only used by maintainers, a tool script is a good solution. Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
>> I was hoping to get a bit more of a response to a recent posting of >> mine with regard to using svn to fetch files for ports >> My proposal: >> http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html >> A summary of what has been going on: >> http://wiki.freebsd.org/EitanAdler/ports-svn > >This was discussed many times before, and this is a bad idea. >- No checksums -> security issue >- No distfile caching/mirroring -> excess traffic consumption and > horrible availability >and I hope you're not even thinking of fetching from VCS without >specifying exact revision. > >Some standard way of generating distfiles from VCS may be nice though, >if it goes into separate mk-file and supports all major VCS. I also think this is a bad idea, even as a separate mk-file, for many of the same reasons. But it could be useful as a script in ports/Tools. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
* Eitan Adler (eitanadlerl...@gmail.com) wrote: > I was hoping to get a bit more of a response to a recent posting of > mine with regard to using svn to fetch files for ports > My proposal: > http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html > A summary of what has been going on: > http://wiki.freebsd.org/EitanAdler/ports-svn This was discussed many times before, and this is a bad idea. - No checksums -> security issue - No distfile caching/mirroring -> excess traffic consumption and horrible availability and I hope you're not even thinking of fetching from VCS without specifying exact revision. Some standard way of generating distfiles from VCS may be nice though, if it goes into separate mk-file and supports all major VCS. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
Peter Jeremy wrote:
On 2009-Nov-14 15:46:27 +, Thomas Sandford
wrote:
Wesley Shields wrote:
Sure, but it doesn't belong in bsd.*.mk. Turn it into a script and
submit it as a regular port.
...
Surely the whole value/purpose of the ports build infrastructure is to
present a consistent way of doing things rather than different
maintainers doing their own thing and solving problems in different, and
quite possibly sub-optimal ways and/or bloating multiple individual port
Makefiles with what could be kept in a single bsd.*.mk file.
If it affected several hundred ports and/or was visible to the end
user then this might be justification for embedding it into bsd.*.mk.
The ports build infrastructure is already quite large (>20K LOC) and
difficult to follow. The overheads associated with loading bsd.*.mk
files also makes operations like "make index" very time-consuming.
IMHO, bloating it further to marginally simplify life for the
maintainers of ~15 ports is not a good tradeoff.
Wesley's suggestion above sounds like the best solution.
And if the file were (say) bsd.vcs.mk and were pulled in only if one of
USE_SVNFETCH
USE_CVSFETCH
USE_GITFETCH
etc were defined then the impact of the bloat on other ports is minimal.
If you still want to go this way, I'd suggest writing a stand-alone
bsd.vcs.mk that can be .include'd by the port when it needs the
functionality.
Personally I think the performance difference between
.ifdef USE_SVNFETCH
.include "${PORTSDIR}/Mk/bsd.vcs.mk"
.endif
in bsd.port.mk
and only explicitly including it in the ports that use it is marginal,
but I could certainly live with this as a compromise.
--
Thomas Sandford
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On 2009-Nov-14 15:46:27 +, Thomas Sandford wrote: >Wesley Shields wrote: >> Sure, but it doesn't belong in bsd.*.mk. Turn it into a script and >> submit it as a regular port. > >If it were just one port and/or just a port maintainers tool I'd agree. >But this is something that affects MULTIPLE ports. Based on your numbers, 15 ports - less that 0.1% of the ports tree. >Surely the whole value/purpose of the ports build infrastructure is to >present a consistent way of doing things rather than different >maintainers doing their own thing and solving problems in different, and >quite possibly sub-optimal ways and/or bloating multiple individual port >Makefiles with what could be kept in a single bsd.*.mk file. If it affected several hundred ports and/or was visible to the end user then this might be justification for embedding it into bsd.*.mk. The ports build infrastructure is already quite large (>20K LOC) and difficult to follow. The overheads associated with loading bsd.*.mk files also makes operations like "make index" very time-consuming. IMHO, bloating it further to marginally simplify life for the maintainers of ~15 ports is not a good tradeoff. Wesley's suggestion above sounds like the best solution. >And if the file were (say) bsd.vcs.mk and were pulled in only if one of > >USE_SVNFETCH >USE_CVSFETCH >USE_GITFETCH > >etc were defined then the impact of the bloat on other ports is minimal. If you still want to go this way, I'd suggest writing a stand-alone bsd.vcs.mk that can be .include'd by the port when it needs the functionality. -- Peter Jeremy pgpEvU8kYZsOq.pgp Description: PGP signature
Re: RFC: svn for make fetch
Wesley Shields wrote: On Fri, Nov 13, 2009 at 10:00:08AM +0200, Eitan Adler wrote: Actually I was thinking of eventually adding non-svn support as well I don't think bloating bsd.*.mk for the most common VCS out there is a good idea, not to mention what happens when someone wants support for some oddball VCS that is not normally used? The reason I started on this project is because the version of mplayer in ports is severely out of date. When I tried to update to port I noticed that the project wants you to compile and install from svn. I also noticed a few other ports that have hacks to let the maintainers "use his/her custom scripts" stuck into the port's Makefile. I think it would be good if there was some standardized way of solving both of these problems... Sure, but it doesn't belong in bsd.*.mk. Turn it into a script and submit it as a regular port. If it were just one port and/or just a port maintainers tool I'd agree. But this is something that affects MULTIPLE ports. Surely the whole value/purpose of the ports build infrastructure is to present a consistent way of doing things rather than different maintainers doing their own thing and solving problems in different, and quite possibly sub-optimal ways and/or bloating multiple individual port Makefiles with what could be kept in a single bsd.*.mk file. And if the file were (say) bsd.vcs.mk and were pulled in only if one of USE_SVNFETCH USE_CVSFETCH USE_GITFETCH etc were defined then the impact of the bloat on other ports is minimal. A quick scan of ports reveals that the following contain the string "svn export" in their Makefile ports/cad/kicad-devel ports/comms/wsjt ports/comms/wspr ports/devel/compiler-rt ports/devel/llvm-devel ports/devel/thrift ports/games/evq3 ports/games/q2pro ports/games/freeorion ports/games/worldofpadman ports/net-im/cjc ports/www/twiki ports/www/foswiki ports/x11-toolkits/gigi This thread has revealed at least two further ports that use a svn distribution but where this is hidden from the user. That's 15 ports already that could be sharing common code instead of doing it themselves. How many do you need? -- Thomas Sandford ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Fri, Nov 13, 2009 at 10:00:08AM +0200, Eitan Adler wrote: > Actually I was thinking of eventually adding non-svn support as well I don't think bloating bsd.*.mk for the most common VCS out there is a good idea, not to mention what happens when someone wants support for some oddball VCS that is not normally used? > The reason I started on this project is because the version of mplayer in > ports is severely out of date. When I tried to update to port I noticed that > the project wants you to compile and install from svn. I also noticed a few > other ports that have hacks to let the maintainers "use his/her custom > scripts" stuck into the port's Makefile. I think it would be good if there > was some standardized way of solving both of these problems... Sure, but it doesn't belong in bsd.*.mk. Turn it into a script and submit it as a regular port. -- WXS ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
Actually I was thinking of eventually adding non-svn support as well The reason I started on this project is because the version of mplayer in ports is severely out of date. When I tried to update to port I noticed that the project wants you to compile and install from svn. I also noticed a few other ports that have hacks to let the maintainers "use his/her custom scripts" stuck into the port's Makefile. I think it would be good if there was some standardized way of solving both of these problems... On Fri, Nov 13, 2009 at 3:10 AM, Wesley Shields wrote: > On Tue, Nov 10, 2009 at 10:28:17PM +0200, Eitan Adler wrote: > > Alright - I updated the wiki page to summarize the thread so far. > > I'd appreciate if people could comment on the "spec" part > > specifically. I'd like to see that section become much more specific > > (so that I could use it to unambiguously write something based off of > > it) > > > > I'm holding off on writing any new implementations at the moment to > > see where things head in terms of how things should be done. > > http://wiki.freebsd.org/EitanAdler/ports-svn > > Why only SVN? I'm not trying to be picky but if we start supporting this > for SVN it's very easy to argue we should support it for some other VCS. > Personally I think this kind of thing is best left up to the maintainer > to use his/her custom scripts and not stuff support for it into the > ports infrastructure. > > -- WXS > ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Tue, Nov 10, 2009 at 10:28:17PM +0200, Eitan Adler wrote: > Alright - I updated the wiki page to summarize the thread so far. > I'd appreciate if people could comment on the "spec" part > specifically. I'd like to see that section become much more specific > (so that I could use it to unambiguously write something based off of > it) > > I'm holding off on writing any new implementations at the moment to > see where things head in terms of how things should be done. > http://wiki.freebsd.org/EitanAdler/ports-svn Why only SVN? I'm not trying to be picky but if we start supporting this for SVN it's very easy to argue we should support it for some other VCS. Personally I think this kind of thing is best left up to the maintainer to use his/her custom scripts and not stuff support for it into the ports infrastructure. -- WXS ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
Any problems with something like this in bsd.svn.mk
Comments and suggestions welcome...
x-svn-export:
svn export -r${SVN_REV} ${SVN_URL} ${WRKSRC}
x-svn-tar:
${TAR} -cjvf ${DISTNAME}.tar.bz2 ${WRKSRC}
${RM} -rf ${WRKSRC}
x-svn-head:
SVN_REV != svn info ${SVN_URL} | grep "^Last Changed Rev:"|awk
'${print $$4}'
x-svn-all: .ORDER x-svn-head x-svn-export x-svn-tar makesum
x-svn-prebuild: x-svn-export x-svn-tar checksum
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Tue, Nov 10, 2009 at 06:12:40PM +, RW wrote: > On Tue, 10 Nov 2009 12:32:28 +0200 > Peter Pentchev wrote: > > > > The Ports Collection's distfile checksums make sure that you get > > exactly the same files *as the port maintainer examined at some > > previous moment in time*. > > More importantly it guards against maliciously modified source code. > Someone might break into a legitimate mirror or use dns poisoning to > distribute malware. That's the whole point :) That's also why the maintainer is supposed to examine the files before submitting (or committing) a port update - to guard against source code that has been maliciously modified on the master sites (or on fake master sites that the maintainer has been redirected to). G'luck, Peter -- Peter Pentchev r...@ringlet.netr...@space.bgr...@freebsd.org PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13 If wishes were fishes, the antecedent of this conditional would be true. pgpIONgN43NT0.pgp Description: PGP signature
Re: RFC: svn for make fetch
Alright - I updated the wiki page to summarize the thread so far. I'd appreciate if people could comment on the "spec" part specifically. I'd like to see that section become much more specific (so that I could use it to unambiguously write something based off of it) I'm holding off on writing any new implementations at the moment to see where things head in terms of how things should be done. http://wiki.freebsd.org/EitanAdler/ports-svn ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Eitan Adler wrote: > Correct me if I'm wrong but I thought that svn did its own checksumming. > If so why do we need to our own? "In God we trust, everyone else must have an X.509 certificate." Well, that's not necessarily be a X.509 certificate but it must be some form of signature, as it's not too hard to replace a specific revision in svn if the server gets compromised. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.13 (FreeBSD) iEYEARECAAYFAkr5tssACgkQi+vbBBjt66BdpACdH5+RSlwKN10x8MiGFYiuX0dL L94An1N1uYCFUYJo0f0U2jZgqRK7emo1 =jFoS -END PGP SIGNATURE- ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Tue, 10 Nov 2009 12:32:28 +0200 Peter Pentchev wrote: > The Ports Collection's distfile checksums make sure that you get > exactly the same files *as the port maintainer examined at some > previous moment in time*. More importantly it guards against maliciously modified source code. Someone might break into a legitimate mirror or use dns poisoning to distribute malware. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On 11/8/2009 10:31 AM, Eitan Adler wrote: I was hoping to get a bit more of a response to a recent posting of mine with regard to using svn to fetch files for ports My proposal: http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html A summary of what has been going on: http://wiki.freebsd.org/EitanAdler/ports-svn This is something that more than 2 people should have an input on I would learn toward having something available to create tarballs for maintainers but I don't think having a port fetch directly from svn would be such a good idea. Also what about the load on the repository? There is a lot of infrastructure in place for mirroring tarballs (FreeBSD and otherwise) but very little for svn and I doubt many projects have mirrored repositories. Going with the mplayer example there may not be a lot of FreeBSD users installing mplayer at any given time but it may be enough to be a concern as far as load on the repository. Jonathan ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Tue, Nov 10, 2009 at 08:51:25AM +0200, Eitan Adler wrote: > Correct me if I'm wrong but I thought that svn did its own checksumming. > If so why do we need to our own? Subversion's checksumming makes sure that you get exactly the same files that are on the Subversion server at this particular moment in time. The Ports Collection's distfile checksums make sure that you get exactly the same files *as the port maintainer examined at some previous moment in time*. The difference is crucial. svnadmin create /home/svn/foo svn import http://.../foo/trunk/mycoolproject rm -rf /home/svn/foo svnadmin create /home/svn/foo svn import http://.../foo/trunk/mycoolproject ...and suddenly the port fetches something completely different. G'luck, Peter -- Peter Pentchev r...@ringlet.netr...@space.bgr...@freebsd.org PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 This would easier understand fewer had omitted. pgpgo8opOFsxg.pgp Description: PGP signature
Re: RFC: svn for make fetch
Correct me if I'm wrong but I thought that svn did its own checksumming. If so why do we need to our own? ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Mon, Nov 09, 2009 at 02:28:52PM -0800, Xin LI wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Eitan Adler wrote: > > I was hoping to get a bit more of a response to a recent posting of > > mine with regard to using svn to fetch files for ports > > My proposal: > > http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html > > A summary of what has been going on: > > http://wiki.freebsd.org/EitanAdler/ports-svn > > > > This is something that more than 2 people should have an input on > > Just my $0.02 but I think it would be great if we can do: > > - "make fetch" would prefer using a pre-packaged tarball, but fallback > to a svn/whatever checkout with a specific revision number, then > generate a tarball from the export. > - "make checksum" would check the tarball's checksum. It's probalby not practical to generate an archive with consistently identical checksums due to the various timestamps (at least without adding a tar writer to svn which would be kind of cool. :) > Maybe we can also have some variables to control that we actually want > the 'HEAD' revision without checking any checksum. However, I think it > would be nice if we can do a checksum'ed checkout for specific SCM > revision, especially if we want to have ports to work not only for > *-devel ports where we would prefer signed source code. While this wasn't the intent of the -DBOOTSTRAP stuff I added to the llvm port, people are finding it useful so if we can find I clean way to support this I think it would be cool. -- Brooks pgpJG8wxgpti5.pgp Description: PGP signature
Re: RFC: svn for make fetch
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Eitan Adler wrote: > I was hoping to get a bit more of a response to a recent posting of > mine with regard to using svn to fetch files for ports > My proposal: > http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html > A summary of what has been going on: > http://wiki.freebsd.org/EitanAdler/ports-svn > > This is something that more than 2 people should have an input on Just my $0.02 but I think it would be great if we can do: - "make fetch" would prefer using a pre-packaged tarball, but fallback to a svn/whatever checkout with a specific revision number, then generate a tarball from the export. - "make checksum" would check the tarball's checksum. Maybe we can also have some variables to control that we actually want the 'HEAD' revision without checking any checksum. However, I think it would be nice if we can do a checksum'ed checkout for specific SCM revision, especially if we want to have ports to work not only for *-devel ports where we would prefer signed source code. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.13 (FreeBSD) iEYEARECAAYFAkr4l6QACgkQi+vbBBjt66CD/wCbBcIFcfbK4a0533PAcNxbZuV5 WXAAnjQfl07w/vcHicVS0s+FOrOs5CMS =1Hly -END PGP SIGNATURE- ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Sun, Nov 08, 2009 at 07:07:25PM +, Marcin Wisnicki wrote:
> On Sun, 08 Nov 2009 17:31:57 +0200, Eitan Adler wrote:
>
> > I was hoping to get a bit more of a response to a recent posting of mine
> > with regard to using svn to fetch files for ports My proposal:
> > http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html A
> > summary of what has been going on:
> > http://wiki.freebsd.org/EitanAdler/ports-svn
> >
> > This is something that more than 2 people should have an input on
>
> Unless you solve plist problem (and completely automated plist generation
> would be a fantastic thing to have!), such functionality should not be
> available (or at least advertised) to end-users.
> You may also consider moving it to separate file (bsd.maintainer.mk).
>
> I don't quite get the logic behind ${USER} == ${SVN_USER} conditional.
> Why do you assume that if my username is the same as username for svn
> checkout then I want to upload snapshot to freefall ? In addition not
> every maintainer has @freebsd.org account. Uploading should be
> customizable (maybe UPLOAD_CMD - like FETCH_CMD).
It's a generalization of an ugly hack I put in my llvm-devel port. I
don't really think it should be part of the base.
-- Brooks
pgp9vsVWHpiAN.pgp
Description: PGP signature
Re: RFC: svn for make fetch
Doug Barton wrote:
Eitan Adler wrote:
I was hoping to get a bit more of a response to a recent posting of
mine with regard to using svn to fetch files for ports
My proposal: http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
A summary of what has been going on:
http://wiki.freebsd.org/EitanAdler/ports-svn
I was really hoping that others would have more to say as well. My
chief concern is that unless I'm missing something there is no way to
provide checksums for the source files, correct? If that's true my gut
reaction is "no freakin' way" but I'm willing to listen to arguments
as to why this should be ok.
I tend to agree with the other sentiments already expressed that in
cases where svn is the only way a project distributes its source that
maintainers ought to be putting together tarballs of specific
versions. I don't really see that this is an overwhelming burden, but
again I'm open to arguments as to why I might be wrong about this.
OK - "I think you might be wrong about this"
A classic example is the net/freeswitch port where the porter has done
exactly what you say.
(IMHO they've not done it in an optimal way but...)
1) It is not clear from the Makefile what version they are actually using.
2) It is indeed unclear from the Makefile what the provenance of the
fetched tarball is at all.
3) It is very difficult for anyone other than the maintainer to create
an update of the port either for local use or for submission of a patch.
(and writing this I note that (4) the distfile location has dropped off
the net)
(1) and (2) could be _improved_ by more documentation in the Makefile
and better choice of versioning scheme but 3 is fairly fundamental.
What _I'd_ like to see is a development of a combination of "method 1" &
"method 2" from the wiki page referenced above.
Running "make fetch" would perform an svn export, and would generate a
tarball from this in ${DISTDIR} named as ${PORTNAME}-${PORTVERSION}.t[gb]z.
A maintainer can then upload this (or the ports distfile handling system
at freebsd.org could even be modified to do this automagically) BUT it
now exists on the users system which means that all the rest of the
ports system including
* _not_ refetching every time the port is built _unless_ the distfile
has changed
* the ability to checksum the (generated) distfile
* (depending on the exact implementation of the new fetch target) the
ability to fallback on fetching a distfile copy from
ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/[distfile]
works as for a "normal" port.
However it is now:
* transparent exactly what is being fetched.
* easy to update the port to fetch a later version by a one line
Makefile edit followed by "make makesum" as is the case for a "standard"
port.
(Obviously the editor of the Makefile needs to carry out checks to
ensure that the PLIST doesn't need to be updated, or other changes made,
to work with the updated revision - as for any other port update).
--
Thomas Sandford
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
Eitan Adler wrote: > I was hoping to get a bit more of a response to a recent posting of > mine with regard to using svn to fetch files for ports > My proposal: > http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html > A summary of what has been going on: > http://wiki.freebsd.org/EitanAdler/ports-svn I was really hoping that others would have more to say as well. My chief concern is that unless I'm missing something there is no way to provide checksums for the source files, correct? If that's true my gut reaction is "no freakin' way" but I'm willing to listen to arguments as to why this should be ok. I tend to agree with the other sentiments already expressed that in cases where svn is the only way a project distributes its source that maintainers ought to be putting together tarballs of specific versions. I don't really see that this is an overwhelming burden, but again I'm open to arguments as to why I might be wrong about this. Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: RFC: svn for make fetch
On Sun, 08 Nov 2009 17:31:57 +0200, Eitan Adler wrote:
> I was hoping to get a bit more of a response to a recent posting of mine
> with regard to using svn to fetch files for ports My proposal:
> http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html A
> summary of what has been going on:
> http://wiki.freebsd.org/EitanAdler/ports-svn
>
> This is something that more than 2 people should have an input on
Unless you solve plist problem (and completely automated plist generation
would be a fantastic thing to have!), such functionality should not be
available (or at least advertised) to end-users.
You may also consider moving it to separate file (bsd.maintainer.mk).
I don't quite get the logic behind ${USER} == ${SVN_USER} conditional.
Why do you assume that if my username is the same as username for svn
checkout then I want to upload snapshot to freefall ? In addition not
every maintainer has @freebsd.org account. Uploading should be
customizable (maybe UPLOAD_CMD - like FETCH_CMD).
Other than that I really like the idea (maintainer part) since I had to
do something similar recently with smartmontools and having a
standardised way to prepare ports for svn snapshots would have saved me
some time. FWIW here is how I did it (in port's Makefile):
PORTVERSION=5.38.r${SVNREVISION}
SVNREVISION=2924
# no prebuilt files in svn
USE_AUTOTOOLS= aclocal:110 autoheader:262 automake:110 autoconf:262
# skip...
.if defined(MAINTAINER_MODE)
DISTFILES=
SVN_URL=https://path/to/trunk
x-maintainer-make-snapshot:
svn export -r${SVNREVISION} ${SVN_URL} ${DISTNAME}
${TAR} -cjvf ${DISTNAME}.tar.bz2 ${DISTNAME}
${RM} -rf ${DISTNAME}
post-extract:
svn co -r${SVNREVISION} ${SVN_URL} ${WRKSRC}
.if defined(HEAD_REVISION)
SVNREVISION!= svn info ${SVN_URL} | grep "^Last Changed Rev:" \
| awk '{print $$4}'
.endif
# TODO generate plist
.endif # MAINTAINER_MODE
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
RFC: svn for make fetch
I was hoping to get a bit more of a response to a recent posting of mine with regard to using svn to fetch files for ports My proposal: http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html A summary of what has been going on: http://wiki.freebsd.org/EitanAdler/ports-svn This is something that more than 2 people should have an input on ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
