Re: RFC: svn for make fetch

2009-11-18 Thread Eitan Adler
On Wed, Nov 18, 2009 at 2:19 AM, Scot Hetzel swhet...@gmail.com wrote:
 On Tue, Nov 17, 2009 at 3:59 PM, Eitan Adler eitanadlerl...@gmail.com wrote:
 Actually, I *had* a patch that got the source from svn, tarred it
 and checksummed it, with little modification to the do-fetch target
 and abusing FETCH_* variables.
 The unsolvable problem I ran in to, is that svn doesn't adjust
 timestamps for directories on export, so the checksum for the tar
 was always different. Hacking svn export was outside my timeframe
 and hacking tar to grow an option that sets all created dirs to
 a fixed time stamp, seemed too hackish, so I let it go.

 Creating deterministic tars (ignoring metadeta) sounds like it
 should be a solved problem by now. If it isn't then I will have to
 make it my next project ;)

 Instead of creating tar files, create zip files and then run them
 through torrentzip

 http://www.freebsd.org/cgi/cvsweb.cgi/ports/archivers/torrentzip/

 Torrentzip resets the date/time on the files and directories in the
 zip archive so that the checksum of the file will match, no matter who
 builds the zip file using the same set of files.

 Scot


Does such a tool exist for tar archives?
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-18 Thread Thomas Sandford

(Sorry, Ive lost track of the higher level attributions, but)
Eitan Adler wrote:

Creating deterministic tars (ignoring metadeta) sounds like it
should be a solved problem by now. If it isn't then I will have to
make it my next project ;)


Instead of creating tar files, create zip files and then run them
through torrentzip

http://www.freebsd.org/cgi/cvsweb.cgi/ports/archivers/torrentzip/

Torrentzip resets the date/time on the files and directories in the
zip archive so that the checksum of the file will match, no matter who
builds the zip file using the same set of files.

Scot



Does such a tool exist for tar archives?


Couldn't you achieve the same thing by the use of touch(1)
--
Thomas Sandford
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-18 Thread Mel Flynn
On Wed, 18 Nov 2009 11:43:07 +, Thomas Sandford
freebsdu...@paradisegreen.co.uk wrote:
 (Sorry, Ive lost track of the higher level attributions, but)
 Eitan Adler wrote:
 Creating deterministic tars (ignoring metadeta) sounds like it
 should be a solved problem by now. If it isn't then I will have to
 make it my next project ;)

 Instead of creating tar files, create zip files and then run them
 through torrentzip

 http://www.freebsd.org/cgi/cvsweb.cgi/ports/archivers/torrentzip/

 Torrentzip resets the date/time on the files and directories in the
 zip archive so that the checksum of the file will match, no matter who
 builds the zip file using the same set of files.

 Scot

 
 Does such a tool exist for tar archives?
 
 Couldn't you achieve the same thing by the use of touch(1)

Probably, but my idea was at the time to provide a simple way to support
non-releases (like mplayer) and devel snapshots for which a tarball is
not or rarely rolled, with minimal change to bsd.port.mk.

Hence the abuse of FETCH_*. I found it again, I'll clean it up when I have
more time - there's more stuff in my bsd.local.mk so not a clean diff, but
the crux is this:
FETCH_BINARY=${SVN_CMD}
FETCH_ARGS=${SVN_QUIET} export
FETCH_BEFORE_ARGS=-r${SVN_REVISION}
DISABLE_SIZE=yes # screws with FETCH_BEFORE_ARGS
# cwd at this point is ${_DISTDIR}
FETCH_AFTER_ARGS=${_SVN_STAGE}  ${TAR} -C ${SVN_STAGE} -czf \
${PORTNAME}-${PORTVERSION}.tar.gz ${PORTNAME}-${PORTVERSION} \
WRKSRC?=${WRKDIR}/${SVN_ROOT}

FETCH_NO_APPEND_FILE=yes

pre-fetch:
@${MKDIR} ${SVN_STAGE}

# Once bsdtar supports remove after archiving, we don't need this anymore.
post-fetch:
@${RM} -rf ${_SVN_STAGE}

(Yes, I realize I should inject my own targets and not abuse
pre|post-fetch).

bsd.port.mk diff snippet (long line):
--- Mk/bsd.port.mk   21 Sep 2009 19:13:51 - 1.629
+++ Mk/bsd.port.mk   22 Sep 2009 01:21:26 -
@@ -3490,11 +3490,15 @@
 for site in `eval $$SORTED_MASTER_SITES_CMD_TMP
${_RANDOMIZE_SITES}`; do \
 ${ECHO_MSG} = Attempting to fetch from $${site}.; \
CKSIZE=`alg=SIZE; ${DISTINFO_DATA}`; \
-   case $${file} in \
-   */*)  ${MKDIR} $${file%/*}; \
- args=-o $${file} $${site}$${file};; \
-   *)args=$${site}$${file};; \
-   esac; \
+   if test x${FETCH_NO_APPEND_FILE} != xyes; then \
+  case $${file} in \
+  */*)  ${MKDIR} $${file%/*}; \
+args=-o $${file} $${site}$${file};; \
+  *)args=$${site}$${file};; \
+  esac; \
+   else \
+  args=$${site}; \
+   fi; \
if ${SETENV} ${FETCH_ENV} ${FETCH_CMD} ${FETCH_BEFORE_ARGS}
$${args} ${FETCH_AFTER_ARGS}; then \
   continue 2; \
fi; \

-- 
--
Mel

___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-17 Thread Eitan Adler
 Actually, I *had* a patch that got the source from svn, tarred it
 and checksummed it, with little modification to the do-fetch target
 and abusing FETCH_* variables.
 The unsolvable problem I ran in to, is that svn doesn't adjust
 timestamps for directories on export, so the checksum for the tar
 was always different. Hacking svn export was outside my timeframe
 and hacking tar to grow an option that sets all created dirs to
 a fixed time stamp, seemed too hackish, so I let it go.

Creating deterministic tars (ignoring metadeta) sounds like it
should be a solved problem by now. If it isn't then I will have to
make it my next project ;)

Anyway lets take your script above modify it so that it uploads the
tarball to freebsd mirrors and includes the checksum as of the time
the maintainer created the tar. This would be much closer to my
proposal.
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-17 Thread Scot Hetzel
On Tue, Nov 17, 2009 at 3:59 PM, Eitan Adler eitanadlerl...@gmail.com wrote:
 Actually, I *had* a patch that got the source from svn, tarred it
 and checksummed it, with little modification to the do-fetch target
 and abusing FETCH_* variables.
 The unsolvable problem I ran in to, is that svn doesn't adjust
 timestamps for directories on export, so the checksum for the tar
 was always different. Hacking svn export was outside my timeframe
 and hacking tar to grow an option that sets all created dirs to
 a fixed time stamp, seemed too hackish, so I let it go.

 Creating deterministic tars (ignoring metadeta) sounds like it
 should be a solved problem by now. If it isn't then I will have to
 make it my next project ;)

Instead of creating tar files, create zip files and then run them
through torrentzip

http://www.freebsd.org/cgi/cvsweb.cgi/ports/archivers/torrentzip/

Torrentzip resets the date/time on the files and directories in the
zip archive so that the checksum of the file will match, no matter who
builds the zip file using the same set of files.

Scot
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-16 Thread Thomas Sandford

Peter Jeremy wrote:

On 2009-Nov-14 15:46:27 +, Thomas Sandford 
freebsdu...@paradisegreen.co.uk wrote:

Wesley Shields wrote:

Sure, but it doesn't belong in bsd.*.mk. Turn it into a script and
submit it as a regular port.

...
Surely the whole value/purpose of the ports build infrastructure is to 
present a consistent way of doing things rather than different 
maintainers doing their own thing and solving problems in different, and 
quite possibly sub-optimal ways and/or bloating multiple individual port 
Makefiles with what could be kept in a single bsd.*.mk file.


If it affected several hundred ports and/or was visible to the end
user then this might be justification for embedding it into bsd.*.mk.

The ports build infrastructure is already quite large (20K LOC) and
difficult to follow.  The overheads associated with loading bsd.*.mk
files also makes operations like make index very time-consuming.
IMHO, bloating it further to marginally simplify life for the
maintainers of ~15 ports is not a good tradeoff.

Wesley's suggestion above sounds like the best solution.


And if the file were (say) bsd.vcs.mk and were pulled in only if one of

USE_SVNFETCH
USE_CVSFETCH
USE_GITFETCH

etc were defined then the impact of the bloat on other ports is minimal.


If you still want to go this way, I'd suggest writing a stand-alone
bsd.vcs.mk that can be .include'd by the port when it needs the
functionality.


Personally I think the performance difference between

.ifdef USE_SVNFETCH
.include ${PORTSDIR}/Mk/bsd.vcs.mk
.endif

in bsd.port.mk

and only explicitly including it in the ports that use it is marginal, 
but I could certainly live with this as a compromise.


--
Thomas Sandford
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-16 Thread Dmitry Marakasov
* Eitan Adler (eitanadlerl...@gmail.com) wrote:

 I was hoping to get a bit more of a response to a recent posting of
 mine with regard to using svn to fetch files for ports
 My proposal: 
 http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
 A summary of what has been going on:
 http://wiki.freebsd.org/EitanAdler/ports-svn

This was discussed many times before, and this is a bad idea.
- No checksums - security issue
- No distfile caching/mirroring - excess traffic consumption and
  horrible availability
and I hope you're not even thinking of fetching from VCS without
specifying exact revision.

Some standard way of generating distfiles from VCS may be nice though,
if it goes into separate mk-file and supports all major VCS.

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amd...@amdmi3.ru  ..:  jabber: amd...@jabber.ruhttp://www.amdmi3.ru
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-16 Thread b. f.
 I was hoping to get a bit more of a response to a recent posting of
 mine with regard to using svn to fetch files for ports
 My proposal: 
 http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
 A summary of what has been going on:
 http://wiki.freebsd.org/EitanAdler/ports-svn

This was discussed many times before, and this is a bad idea.
- No checksums - security issue
- No distfile caching/mirroring - excess traffic consumption and
  horrible availability
and I hope you're not even thinking of fetching from VCS without
specifying exact revision.

Some standard way of generating distfiles from VCS may be nice though,
if it goes into separate mk-file and supports all major VCS.

I also think this is a bad idea, even as a separate mk-file, for many
of the same reasons.  But it could be useful as a script in
ports/Tools.
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-16 Thread Doug Barton
Thomas Sandford wrote:
 If it were just one port and/or just a port maintainers tool I'd agree.
 But this is something that affects MULTIPLE ports.

You still seem to be advocating the idea of making this a mechanism
for users, in spite of the fact that numerous people have said that
idea is not likely to ever be approved. Given the universe where this
feature is only used by maintainers, a tool script is a good solution.


Doug

-- 

Improve the effectiveness of your Internet presence with
a domain name makeover!http://SupersetSolutions.com/

___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-16 Thread Mel Flynn
On Mon, 16 Nov 2009 15:16:22 +0300, Dmitry Marakasov amd...@amdmi3.ru
wrote:
 * Eitan Adler (eitanadlerl...@gmail.com) wrote:
 
 I was hoping to get a bit more of a response to a recent posting of
 mine with regard to using svn to fetch files for ports
 My proposal:
 http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
 A summary of what has been going on:
 http://wiki.freebsd.org/EitanAdler/ports-svn
 
 This was discussed many times before, and this is a bad idea.
 - No checksums - security issue
 - No distfile caching/mirroring - excess traffic consumption and
   horrible availability

Actually, I *had* a patch that got the source from svn, tarred it
and checksummed it, with little modification to the do-fetch target
and abusing FETCH_* variables.
The unsolvable problem I ran in to, is that svn doesn't adjust
timestamps for directories on export, so the checksum for the tar
was always different. Hacking svn export was outside my timeframe
and hacking tar to grow an option that sets all created dirs to
a fixed time stamp, seemed too hackish, so I let it go.

--
Mel

___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-15 Thread Peter Jeremy
On 2009-Nov-14 15:46:27 +, Thomas Sandford 
freebsdu...@paradisegreen.co.uk wrote:
Wesley Shields wrote:
 Sure, but it doesn't belong in bsd.*.mk. Turn it into a script and
 submit it as a regular port.

If it were just one port and/or just a port maintainers tool I'd agree. 
But this is something that affects MULTIPLE ports.

Based on your numbers, 15 ports - less that 0.1% of the ports tree.

Surely the whole value/purpose of the ports build infrastructure is to 
present a consistent way of doing things rather than different 
maintainers doing their own thing and solving problems in different, and 
quite possibly sub-optimal ways and/or bloating multiple individual port 
Makefiles with what could be kept in a single bsd.*.mk file.

If it affected several hundred ports and/or was visible to the end
user then this might be justification for embedding it into bsd.*.mk.

The ports build infrastructure is already quite large (20K LOC) and
difficult to follow.  The overheads associated with loading bsd.*.mk
files also makes operations like make index very time-consuming.
IMHO, bloating it further to marginally simplify life for the
maintainers of ~15 ports is not a good tradeoff.

Wesley's suggestion above sounds like the best solution.

And if the file were (say) bsd.vcs.mk and were pulled in only if one of

USE_SVNFETCH
USE_CVSFETCH
USE_GITFETCH

etc were defined then the impact of the bloat on other ports is minimal.

If you still want to go this way, I'd suggest writing a stand-alone
bsd.vcs.mk that can be .include'd by the port when it needs the
functionality.

-- 
Peter Jeremy


pgpEvU8kYZsOq.pgp
Description: PGP signature


Re: RFC: svn for make fetch

2009-11-14 Thread Thomas Sandford

Wesley Shields wrote:

On Fri, Nov 13, 2009 at 10:00:08AM +0200, Eitan Adler wrote:

Actually I was thinking of eventually adding non-svn support as well


I don't think bloating bsd.*.mk for the most common VCS out there is a
good idea, not to mention what happens when someone wants support for
some oddball VCS that is not normally used?


The reason I started on this project is because the version of mplayer in
ports is severely out of date. When I tried to update to port I noticed that
the project wants you to compile and install from svn. I also noticed a few
other ports that have hacks to let the maintainers use his/her custom
scripts stuck into the port's Makefile. I think it would be good if there
was some standardized way of solving both of these problems...


Sure, but it doesn't belong in bsd.*.mk. Turn it into a script and
submit it as a regular port.


If it were just one port and/or just a port maintainers tool I'd agree. 
But this is something that affects MULTIPLE ports.


Surely the whole value/purpose of the ports build infrastructure is to 
present a consistent way of doing things rather than different 
maintainers doing their own thing and solving problems in different, and 
quite possibly sub-optimal ways and/or bloating multiple individual port 
Makefiles with what could be kept in a single bsd.*.mk file.


And if the file were (say) bsd.vcs.mk and were pulled in only if one of

USE_SVNFETCH
USE_CVSFETCH
USE_GITFETCH

etc were defined then the impact of the bloat on other ports is minimal.

A quick scan of ports reveals that the following contain the string svn 
export in their Makefile


ports/cad/kicad-devel
ports/comms/wsjt
ports/comms/wspr
ports/devel/compiler-rt
ports/devel/llvm-devel
ports/devel/thrift
ports/games/evq3
ports/games/q2pro
ports/games/freeorion
ports/games/worldofpadman
ports/net-im/cjc
ports/www/twiki
ports/www/foswiki
ports/x11-toolkits/gigi

This thread has revealed at least two further ports that use a svn 
distribution but where this is hidden from the user. That's 15 ports 
already that could be sharing common code instead of doing it 
themselves. How many do you need?


--
Thomas Sandford
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-13 Thread Eitan Adler
Actually I was thinking of eventually adding non-svn support as well

The reason I started on this project is because the version of mplayer in
ports is severely out of date. When I tried to update to port I noticed that
the project wants you to compile and install from svn. I also noticed a few
other ports that have hacks to let the maintainers use his/her custom
scripts stuck into the port's Makefile. I think it would be good if there
was some standardized way of solving both of these problems...

On Fri, Nov 13, 2009 at 3:10 AM, Wesley Shields w...@freebsd.org wrote:

 On Tue, Nov 10, 2009 at 10:28:17PM +0200, Eitan Adler wrote:
  Alright - I updated the wiki page to summarize the thread so far.
  I'd appreciate if people could comment on the spec part
  specifically. I'd like to see that section become much more specific
  (so that I could use it to unambiguously write something based off of
  it)
 
  I'm holding off on writing any new implementations at the moment to
  see where things head in terms of how things should be done.
  http://wiki.freebsd.org/EitanAdler/ports-svn

 Why only SVN? I'm not trying to be picky but if we start supporting this
 for SVN it's very easy to argue we should support it for some other VCS.
 Personally I think this kind of thing is best left up to the maintainer
 to use his/her custom scripts and not stuff support for it into the
 ports infrastructure.

 -- WXS

___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-13 Thread Wesley Shields
On Fri, Nov 13, 2009 at 10:00:08AM +0200, Eitan Adler wrote:
 Actually I was thinking of eventually adding non-svn support as well

I don't think bloating bsd.*.mk for the most common VCS out there is a
good idea, not to mention what happens when someone wants support for
some oddball VCS that is not normally used?

 The reason I started on this project is because the version of mplayer in
 ports is severely out of date. When I tried to update to port I noticed that
 the project wants you to compile and install from svn. I also noticed a few
 other ports that have hacks to let the maintainers use his/her custom
 scripts stuck into the port's Makefile. I think it would be good if there
 was some standardized way of solving both of these problems...

Sure, but it doesn't belong in bsd.*.mk. Turn it into a script and
submit it as a regular port.

-- WXS
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-12 Thread Eitan Adler
Any problems with something like this in bsd.svn.mk

Comments and suggestions welcome...

x-svn-export:
svn export -r${SVN_REV} ${SVN_URL} ${WRKSRC}
x-svn-tar:
${TAR} -cjvf ${DISTNAME}.tar.bz2 ${WRKSRC}
${RM} -rf ${WRKSRC}
x-svn-head:
SVN_REV !=  svn info ${SVN_URL} | grep ^Last Changed Rev:|awk
'${print $$4}'
x-svn-all: .ORDER x-svn-head x-svn-export x-svn-tar makesum
x-svn-prebuild: x-svn-export x-svn-tar checksum
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-12 Thread Wesley Shields
On Tue, Nov 10, 2009 at 10:28:17PM +0200, Eitan Adler wrote:
 Alright - I updated the wiki page to summarize the thread so far.
 I'd appreciate if people could comment on the spec part
 specifically. I'd like to see that section become much more specific
 (so that I could use it to unambiguously write something based off of
 it)
 
 I'm holding off on writing any new implementations at the moment to
 see where things head in terms of how things should be done.
 http://wiki.freebsd.org/EitanAdler/ports-svn

Why only SVN? I'm not trying to be picky but if we start supporting this
for SVN it's very easy to argue we should support it for some other VCS.
Personally I think this kind of thing is best left up to the maintainer
to use his/her custom scripts and not stuff support for it into the
ports infrastructure.

-- WXS
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-10 Thread Peter Pentchev
On Tue, Nov 10, 2009 at 08:51:25AM +0200, Eitan Adler wrote:
 Correct me if I'm wrong but I thought that svn did its own checksumming.
 If so why do we need to our own?

Subversion's checksumming makes sure that you get exactly the same files
that are on the Subversion server at this particular moment in time.

The Ports Collection's distfile checksums make sure that you get exactly
the same files *as the port maintainer examined at some previous moment
in time*.

The difference is crucial.

  svnadmin create /home/svn/foo

  svn import http://.../foo/trunk/mycoolproject
  
  create a port of mycoolproject

  rm -rf /home/svn/foo

  svnadmin create /home/svn/foo

  svn import http://.../foo/trunk/mycoolproject

...and suddenly the port fetches something completely different.

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.netr...@space.bgr...@freebsd.org
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint FDBA FD79 C26F 3C51 C95E  DF9E ED18 B68D 1619 4553
This would easier understand fewer had omitted.


pgpgo8opOFsxg.pgp
Description: PGP signature


Re: RFC: svn for make fetch

2009-11-10 Thread Jonathan

On 11/8/2009 10:31 AM, Eitan Adler wrote:

I was hoping to get a bit more of a response to a recent posting of
mine with regard to using svn to fetch files for ports
My proposal: http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
A summary of what has been going on:
http://wiki.freebsd.org/EitanAdler/ports-svn

This is something that more than 2 people should have an input on


I would learn toward having something available to create tarballs for 
maintainers but I don't think having a port fetch directly from svn 
would be such a good idea.


Also what about the load on the repository?  There is a lot of 
infrastructure in place for mirroring tarballs (FreeBSD and otherwise) 
but very little for svn and I doubt many projects have mirrored 
repositories.  Going with the mplayer example there may not be a lot of 
FreeBSD users installing mplayer at any given time but it may be enough 
to be a concern as far as load on the repository.


Jonathan
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-10 Thread RW
On Tue, 10 Nov 2009 12:32:28 +0200
Peter Pentchev r...@ringlet.net wrote:


 The Ports Collection's distfile checksums make sure that you get
 exactly the same files *as the port maintainer examined at some
 previous moment in time*.

More importantly it guards against maliciously modified source code.
Someone might break into a legitimate mirror or use dns poisoning to
distribute malware.
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-10 Thread Xin LI
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Eitan Adler wrote:
 Correct me if I'm wrong but I thought that svn did its own checksumming.
 If so why do we need to our own?

In God we trust, everyone else must have an X.509 certificate.

Well, that's not necessarily be a X.509 certificate but it must be some
form of signature, as it's not too hard to replace a specific revision
in svn if the server gets compromised.

Cheers,
- --
Xin LI delp...@delphij.nethttp://www.delphij.net/
FreeBSD - The Power to Serve!  Live free or die
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.13 (FreeBSD)

iEYEARECAAYFAkr5tssACgkQi+vbBBjt66BdpACdH5+RSlwKN10x8MiGFYiuX0dL
L94An1N1uYCFUYJo0f0U2jZgqRK7emo1
=jFoS
-END PGP SIGNATURE-
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-10 Thread Eitan Adler
Alright - I updated the wiki page to summarize the thread so far.
I'd appreciate if people could comment on the spec part
specifically. I'd like to see that section become much more specific
(so that I could use it to unambiguously write something based off of
it)

I'm holding off on writing any new implementations at the moment to
see where things head in terms of how things should be done.
http://wiki.freebsd.org/EitanAdler/ports-svn
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-10 Thread Peter Pentchev
On Tue, Nov 10, 2009 at 06:12:40PM +, RW wrote:
 On Tue, 10 Nov 2009 12:32:28 +0200
 Peter Pentchev r...@ringlet.net wrote:
 
 
  The Ports Collection's distfile checksums make sure that you get
  exactly the same files *as the port maintainer examined at some
  previous moment in time*.
 
 More importantly it guards against maliciously modified source code.
 Someone might break into a legitimate mirror or use dns poisoning to
 distribute malware.

That's the whole point :)  That's also why the maintainer is supposed to
examine the files before submitting (or committing) a port update -
to guard against source code that has been maliciously modified on
the master sites (or on fake master sites that the maintainer has been
redirected to).

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.netr...@space.bgr...@freebsd.org
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13
If wishes were fishes, the antecedent of this conditional would be true.


pgpIONgN43NT0.pgp
Description: PGP signature


Re: RFC: svn for make fetch

2009-11-09 Thread Doug Barton
Eitan Adler wrote:
 I was hoping to get a bit more of a response to a recent posting of
 mine with regard to using svn to fetch files for ports
 My proposal: 
 http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
 A summary of what has been going on:
 http://wiki.freebsd.org/EitanAdler/ports-svn

I was really hoping that others would have more to say as well. My
chief concern is that unless I'm missing something there is no way to
provide checksums for the source files, correct? If that's true my gut
reaction is no freakin' way but I'm willing to listen to arguments
as to why this should be ok.

I tend to agree with the other sentiments already expressed that in
cases where svn is the only way a project distributes its source that
maintainers ought to be putting together tarballs of specific
versions. I don't really see that this is an overwhelming burden, but
again I'm open to arguments as to why I might be wrong about this.


Doug

-- 

Improve the effectiveness of your Internet presence with
a domain name makeover!http://SupersetSolutions.com/

___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-09 Thread Thomas Sandford

Doug Barton wrote:

Eitan Adler wrote:

I was hoping to get a bit more of a response to a recent posting of
mine with regard to using svn to fetch files for ports
My proposal: http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
A summary of what has been going on:
http://wiki.freebsd.org/EitanAdler/ports-svn


I was really hoping that others would have more to say as well. My
chief concern is that unless I'm missing something there is no way to
provide checksums for the source files, correct? If that's true my gut
reaction is no freakin' way but I'm willing to listen to arguments
as to why this should be ok.

I tend to agree with the other sentiments already expressed that in
cases where svn is the only way a project distributes its source that
maintainers ought to be putting together tarballs of specific
versions. I don't really see that this is an overwhelming burden, but
again I'm open to arguments as to why I might be wrong about this.


OK - I think you might be wrong about this

A classic example is the net/freeswitch port where the porter has done 
exactly what you say.


(IMHO they've not done it in an optimal way but...)

1) It is not clear from the Makefile what version they are actually using.

2) It is indeed unclear from the Makefile what the provenance of the 
fetched tarball is at all.


3) It is very difficult for anyone other than the maintainer to create 
an update of the port either for local use or for submission of a patch.


(and writing this I note that (4) the distfile location has dropped off 
the net)


(1) and (2) could be _improved_ by more documentation in the Makefile 
and better choice of versioning scheme but 3 is fairly fundamental.


What _I'd_ like to see is a development of a combination of method 1  
method 2 from the wiki page referenced above.


Running make fetch would perform an svn export, and would generate a 
tarball from this in ${DISTDIR} named as ${PORTNAME}-${PORTVERSION}.t[gb]z.


A maintainer can then upload this (or the ports distfile handling system 
at freebsd.org could even be modified to do this automagically) BUT it 
now exists on the users system which means that all the rest of the 
ports system including


* _not_ refetching every time the port is built _unless_ the distfile 
has changed

* the ability to checksum the (generated) distfile
* (depending on the exact implementation of the new fetch target) the 
ability to fallback on fetching a distfile copy from 
ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/[distfile]


works as for a normal port.

However it is now:
* transparent exactly what is being fetched.
* easy to update the port to fetch a later version by a one line 
Makefile edit followed by make makesum as is the case for a standard 
port.


(Obviously the editor of the Makefile needs to carry out checks to 
ensure that the PLIST doesn't need to be updated, or other changes made, 
to work with the updated revision - as for any other port update).


--
Thomas Sandford
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-09 Thread Brooks Davis
On Sun, Nov 08, 2009 at 07:07:25PM +, Marcin Wisnicki wrote:
 On Sun, 08 Nov 2009 17:31:57 +0200, Eitan Adler wrote:
 
  I was hoping to get a bit more of a response to a recent posting of mine
  with regard to using svn to fetch files for ports My proposal:
  http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html A
  summary of what has been going on:
  http://wiki.freebsd.org/EitanAdler/ports-svn
  
  This is something that more than 2 people should have an input on
 
 Unless you solve plist problem (and completely automated plist generation 
 would be a fantastic thing to have!), such functionality should not be 
 available (or at least advertised) to end-users.
 You may also consider moving it to separate file (bsd.maintainer.mk).
 
 I don't quite get the logic behind ${USER} == ${SVN_USER} conditional.
 Why do you assume that if my username is the same as username for svn 
 checkout then I want to upload snapshot to freefall ? In addition not 
 every maintainer has @freebsd.org account. Uploading should be 
 customizable (maybe UPLOAD_CMD - like FETCH_CMD).

It's a generalization of an ugly hack I put in my llvm-devel port.  I
don't really think it should be part of the base.

-- Brooks


pgp9vsVWHpiAN.pgp
Description: PGP signature


Re: RFC: svn for make fetch

2009-11-09 Thread Xin LI
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Eitan Adler wrote:
 I was hoping to get a bit more of a response to a recent posting of
 mine with regard to using svn to fetch files for ports
 My proposal: 
 http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
 A summary of what has been going on:
 http://wiki.freebsd.org/EitanAdler/ports-svn
 
 This is something that more than 2 people should have an input on

Just my $0.02 but I think it would be great if we can do:

 - make fetch would prefer using a pre-packaged tarball, but fallback
to a svn/whatever checkout with a specific revision number, then
generate a tarball from the export.
 - make checksum would check the tarball's checksum.

Maybe we can also have some variables to control that we actually want
the 'HEAD' revision without checking any checksum.  However, I think it
would be nice if we can do a checksum'ed checkout for specific SCM
revision, especially if we want to have ports to work not only for
*-devel ports where we would prefer signed source code.

Cheers,
- --
Xin LI delp...@delphij.nethttp://www.delphij.net/
FreeBSD - The Power to Serve!  Live free or die
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.13 (FreeBSD)

iEYEARECAAYFAkr4l6QACgkQi+vbBBjt66CD/wCbBcIFcfbK4a0533PAcNxbZuV5
WXAAnjQfl07w/vcHicVS0s+FOrOs5CMS
=1Hly
-END PGP SIGNATURE-
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-09 Thread Brooks Davis
On Mon, Nov 09, 2009 at 02:28:52PM -0800, Xin LI wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Eitan Adler wrote:
  I was hoping to get a bit more of a response to a recent posting of
  mine with regard to using svn to fetch files for ports
  My proposal: 
  http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html
  A summary of what has been going on:
  http://wiki.freebsd.org/EitanAdler/ports-svn
  
  This is something that more than 2 people should have an input on
 
 Just my $0.02 but I think it would be great if we can do:
 
  - make fetch would prefer using a pre-packaged tarball, but fallback
 to a svn/whatever checkout with a specific revision number, then
 generate a tarball from the export.
  - make checksum would check the tarball's checksum.

It's probalby not practical to generate an archive with consistently
identical checksums due to the various timestamps (at least without
adding a tar writer to svn which would be kind of cool. :)

 Maybe we can also have some variables to control that we actually want
 the 'HEAD' revision without checking any checksum.  However, I think it
 would be nice if we can do a checksum'ed checkout for specific SCM
 revision, especially if we want to have ports to work not only for
 *-devel ports where we would prefer signed source code.

While this wasn't the intent of the -DBOOTSTRAP stuff I added to the
llvm port, people are finding it useful so if we can find I clean way to
support this I think it would be cool.

-- Brooks


pgpJG8wxgpti5.pgp
Description: PGP signature


Re: RFC: svn for make fetch

2009-11-09 Thread Eitan Adler
Correct me if I'm wrong but I thought that svn did its own checksumming.
If so why do we need to our own?
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


Re: RFC: svn for make fetch

2009-11-08 Thread Marcin Wisnicki
On Sun, 08 Nov 2009 17:31:57 +0200, Eitan Adler wrote:

 I was hoping to get a bit more of a response to a recent posting of mine
 with regard to using svn to fetch files for ports My proposal:
 http://www.mail-archive.com/freebsd-ports@freebsd.org/msg23776.html A
 summary of what has been going on:
 http://wiki.freebsd.org/EitanAdler/ports-svn
 
 This is something that more than 2 people should have an input on

Unless you solve plist problem (and completely automated plist generation 
would be a fantastic thing to have!), such functionality should not be 
available (or at least advertised) to end-users.
You may also consider moving it to separate file (bsd.maintainer.mk).

I don't quite get the logic behind ${USER} == ${SVN_USER} conditional.
Why do you assume that if my username is the same as username for svn 
checkout then I want to upload snapshot to freefall ? In addition not 
every maintainer has @freebsd.org account. Uploading should be 
customizable (maybe UPLOAD_CMD - like FETCH_CMD).

Other than that I really like the idea (maintainer part) since I had to 
do something similar recently with smartmontools and having a 
standardised way to prepare ports for svn snapshots would have saved me 
some time. FWIW here is how I did it (in port's Makefile):

PORTVERSION=5.38.r${SVNREVISION}
SVNREVISION=2924
# no prebuilt files in svn
USE_AUTOTOOLS=  aclocal:110 autoheader:262 automake:110 autoconf:262
# skip...
.if defined(MAINTAINER_MODE)
DISTFILES=
SVN_URL=https://path/to/trunk

x-maintainer-make-snapshot:
svn export -r${SVNREVISION} ${SVN_URL} ${DISTNAME}
${TAR} -cjvf ${DISTNAME}.tar.bz2 ${DISTNAME}
${RM} -rf ${DISTNAME}

post-extract:
svn co -r${SVNREVISION} ${SVN_URL} ${WRKSRC}

.if defined(HEAD_REVISION)
SVNREVISION!=   svn info ${SVN_URL} | grep ^Last Changed Rev: \
| awk '{print $$4}'
.endif

# TODO generate plist
.endif # MAINTAINER_MODE

___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org