Re: comparing cvsup vs. rsync
:Matt, : :something is weird with the permissions: : :%rsync crater.dragonflybsd.org::dragonfly_cvs/src/crypto/heimdal/Attic/ :drwxrwxr-x1024 2005/03/28 05:35:43 . :-r--rw-r-- 20313 2005/03/28 05:35:43 ChangeLog,v :[..] :-r-xrwxr-x3242 2005/03/28 05:35:43 compile,v : :why are they u-w but g+w? my cvsup gets a little bit confused with that = :and resets permissions back, so there is always a back-and-forth between = :rsync and cvsup. setting "preserved" and not "umask=3D002" hopefully fix= :es that locally. : :cheers : simon This is a bug in cvs's SGID handling on commits. Access is by group and it doesn't pay much attention to the user, so sometimes the user gets out of whack. Many, many cvs files will not have user-write perms but will have group-write perms because of this. In fact, the user will be whoever last committed the file, which is probably not what we want rsync to deliver. -Matt
Re: comparing cvsup vs. rsync
Simon 'corecode' Schubert wrote: > Matthew Dillon wrote: >> My only worry is figuring out how to run the rsync daemon safely. >> I'm a bit paranoid about running things on crater but I do agree >> that we would have to run the master rsync daemon there. > > You can run rsyncd from inetd or standalone. If you're really[tm] > paranoid, a jail with a static rsync binary might be the best. Yes, I'm way off topic here, and I apologize -- but I've seen your posts in the 'git' mail-list and you've experimented with Hg, so I know you have your own opinions on version control systems. Linus avoided rsync in favor of http in 'git' because he thinks rsync is inherently unreliable. I have *no* idea if he is right or wrong in his opinions, but I figure you guys will favor me with your own opinions on the subject. Thanks!
Re: comparing cvsup vs. rsync
Simon 'corecode' Schubert wrote: I am now running an rsync server on crater.dragonflybsd.org, serving the cvs repository as 'dragonfly_cvs'. rsync -a rsync://crater.dragonflybsd.org/dragonfly_cvs blahblah very nice! will immediatelly switch to rsync operation. however, i'll have to find out how to generate the supfile for cvsupd serving... okay, that doesn't work. cvsup changes the RCS files, so I can't simply cvsup over them to generate a scan file :/ operation without scan file then. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
Matthew Dillon wrote: I am now running an rsync server on crater.dragonflybsd.org, serving the cvs repository as 'dragonfly_cvs'. rsync -a rsync://crater.dragonflybsd.org/dragonfly_cvs blahblah Matt, something is weird with the permissions: %rsync crater.dragonflybsd.org::dragonfly_cvs/src/crypto/heimdal/Attic/ drwxrwxr-x1024 2005/03/28 05:35:43 . -r--rw-r-- 20313 2005/03/28 05:35:43 ChangeLog,v [..] -r-xrwxr-x3242 2005/03/28 05:35:43 compile,v why are they u-w but g+w? my cvsup gets a little bit confused with that and resets permissions back, so there is always a back-and-forth between rsync and cvsup. setting "preserved" and not "umask=002" hopefully fixes that locally. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
Matthew Dillon wrote: I am now running an rsync server on crater.dragonflybsd.org, serving the cvs repository as 'dragonfly_cvs'. rsync -a rsync://crater.dragonflybsd.org/dragonfly_cvs blahblah very nice! will immediatelly switch to rsync operation. however, i'll have to find out how to generate the supfile for cvsupd serving... Minor issue, but it would be nice if I could get statistics on downloads. I have this in my rsyncd.conf: syslog facility = local5 transfer logging = yes and in syslog.conf local5.*/var/log/rsyncd.log (obviously could be made better) cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
I am now running an rsync server on crater.dragonflybsd.org, serving the cvs repository as 'dragonfly_cvs'. rsync -a rsync://crater.dragonflybsd.org/dragonfly_cvs blahblah -- Unfortunately I don't know how to get rsyncd to just log statistics on completion, and it insists on logging a warning '/etc/pwd.db: No such file or directory' every time due to the chroot option in the config file, which I can't seem to get rid of. Minor issue, but it would be nice if I could get statistics on downloads. -Matt
Re: Chlamydia lost php-5.2.0?
On Tue, April 10, 2007 3:18 pm, Mischa Peters wrote: >> http://chlamydia.fs.ei.tum.de/pub/DragonFly/packages/RELEASE/i386/ >> vulnerable/ > > What is the reason that it's available on packages.stura.uni-rostock.de? > > Just trying to understand. I don't know - could be an artifact of the mirroring process, where the package was mirrored while it was in /vulnerable, and a later build on Joerg's system moved it.
Re: [HEADS UP] Binary package changes
On Thu, Apr 05, 2007 at 05:32:59PM +0200, Grzegorz B?ach wrote:
> I can't install any packages from pkgsrc-box.org, but from
> pkgsrc.dragonflybsd.org it works OK.
Can you rebuild pkg_install with the attached patch and see if that
works?
Joerg
Index: ftpio.c
===
RCS file: /home/joerg/repo/netbsd/src/usr.sbin/pkg_install/lib/ftpio.c,v
retrieving revision 1.73
diff -u -r1.73 ftpio.c
--- ftpio.c 11 May 2006 23:50:15 - 1.73
+++ ftpio.c 10 Apr 2007 13:32:34 -
@@ -655,7 +655,10 @@
char *s, buf[MaxPathSize];
char tmpname[MaxPathSize];
char best[MaxPathSize];
- int rc, tfd;
+ int rc, tfd, retry_tbz;
+
+ retry_tbz = 0;
+ best[0]='\0';
rc = ftp_start(base);
if (rc == -1) {
@@ -684,8 +687,16 @@
* we can't use the pkg wildcards here as dewey compare
* and alternates won't be handled by ftp(1); sort
* out later, using pmatch() */
- (void) snprintf(buf, sizeof(buf), "nlist %.*s*.t[bg]z %s\n",
- (int)(s-pattern), pattern, tmpname);
+ if (retry_tbz) {
+retry_with_tbz:
+ (void) snprintf(buf, sizeof(buf), "nlist %.*s*.tbz
%s\n",
+ (int)(s-pattern), pattern, tmpname);
+ retry_tbz = 0;
+ } else {
+ (void) snprintf(buf, sizeof(buf), "nlist %.*s*.tgz
%s\n",
+ (int)(s-pattern), pattern, tmpname);
+ retry_tbz = 1;
+ }
}
rc = ftp_cmd(buf, "\n(550|450|226).*\n"); /* catch errors */
@@ -704,7 +715,6 @@
return NULL;
}
- best[0]='\0';
if (access(tmpname, R_OK)==0) {
int matches;
FILE *f;
@@ -747,6 +757,9 @@
warnx("nothing appropriate found");
}
+ if (retry_tbz)
+ goto retry_with_tbz;
+
unlink(tmpname);
if (best[0] == '\0')
Re: Chlamydia lost php-5.2.0?
Trying to install php5 from pkgsrc (chlamydia), which failed. It seems that php-5.2.0.tgz is no longer there. Any package that has at least 1 open security advisory against it at the time of building gets put in /vulnerable instead of /All, so that you can't accidentally install software that may have a security problem because it's a dependency of something else. http://chlamydia.fs.ei.tum.de/pub/DragonFly/packages/RELEASE/i386/ vulnerable/ What is the reason that it's available on packages.stura.uni-rostock.de? Just trying to understand. Mischa
Re: comparing cvsup vs. rsync
Peter Hessler wrote: On 2007 Apr 10 (Tue) at 18:03:49 +0200 (+0200), Simon 'corecode' Schubert wrote: :cvsync does cvs-only (so, :like rsync), but it has a bug which breaks the RCS files in some cases. :The author didn't want to fix it though :/ Can you describe the bug? I've been using cvsync for local copies of the openbsd source tree and haven't seen any RCS corruption. It missorders revisions on branches. CVS then fails to perform a annotate on those revisions (try something like 1.X.Y.2 or so). cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
On Tue, April 10, 2007 11:32 am, Matthew Dillon wrote: > We could just distribute the CVS tree and write a front-end utility > in csh or sh that we distribute along with the rest of the system > to do the nitty gritty work of actually checking something out into > /usr/src. In fact, I think that would be preferable. I am all for simple upgrades - the closer we can get it to a single button-push for obvious upgrades like 1.8->1.8.1, the happier I'd be. > My only worry is figuring out how to run the rsync daemon safely. > I'm a bit paranoid about running things on crater but I do agree > that we would have to run the master rsync daemon there. Is it feasible to run it in a vkernel? External bandwidth, rather than CPU or disk, is probably the choke point, so I would think there wouldn't be much of a tradeoff of performance vs. security.
Re: Chlamydia lost php-5.2.0?
On Tue, April 10, 2007 11:18 am, Mischa Peters wrote: > Hi All, > > Trying to install php5 from pkgsrc (chlamydia), which failed. > It seems that php-5.2.0.tgz is no longer there. Any package that has at least 1 open security advisory against it at the time of building gets put in /vulnerable instead of /All, so that you can't accidentally install software that may have a security problem because it's a dependency of something else. http://chlamydia.fs.ei.tum.de/pub/DragonFly/packages/RELEASE/i386/vulnerable/
Re: comparing cvsup vs. rsync
On 2007 Apr 10 (Tue) at 18:03:49 +0200 (+0200), Simon 'corecode' Schubert wrote: :cvsync does cvs-only (so, :like rsync), but it has a bug which breaks the RCS files in some cases. :The author didn't want to fix it though :/ Can you describe the bug? I've been using cvsync for local copies of the openbsd source tree and haven't seen any RCS corruption. -- The only real way to look younger is not to be born so soon. -- Charles Schulz, "Things I've Had to Learn Over and Over and Over"
Re: comparing cvsup vs. rsync
Dmitri Nikulin wrote: We could just distribute the CVS tree and write a front-end utility in csh or sh that we distribute along with the rest of the system to do the nitty gritty work of actually checking something out into /usr/src. In fact, I think that would be preferable. NetBSD is distributed on pure CVS, anything else is a convenience. However, because CVS is so impressively bad for initial checkouts, they recommend downloading the rather small source and pkgsrc tarballs, and only using CVS for actual updates. This could work for DragonFly too, with the added bonus of pre-distributed cvs dotfiles or shell aliases. Maybe 'make update' (or whatever it is) could be wired to work that way too. CVS is a beat concerning performance and CPU load, both for server and client. I'd say rsync is the best deal for now. cvsync does cvs-only (so, like rsync), but it has a bug which breaks the RCS files in some cases. The author didn't want to fix it though :/ cvsync with checkout mode and the bug fixed would be definitely better than rsync. Maybe we could do that on a weekend sprint, it doesn't seem too complicated to me. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
Matthew Dillon wrote: My only worry is figuring out how to run the rsync daemon safely. I'm a bit paranoid about running things on crater but I do agree that we would have to run the master rsync daemon there. You can run rsyncd from inetd or standalone. If you're really[tm] paranoid, a jail with a static rsync binary might be the best. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
On 4/11/07, Matthew Dillon <[EMAIL PROTECTED]> wrote: We could just distribute the CVS tree and write a front-end utility in csh or sh that we distribute along with the rest of the system to do the nitty gritty work of actually checking something out into /usr/src. In fact, I think that would be preferable. NetBSD is distributed on pure CVS, anything else is a convenience. However, because CVS is so impressively bad for initial checkouts, they recommend downloading the rather small source and pkgsrc tarballs, and only using CVS for actual updates. This could work for DragonFly too, with the added bonus of pre-distributed cvs dotfiles or shell aliases. Maybe 'make update' (or whatever it is) could be wired to work that way too. This would of course be unecessary if the system could be binary packaged and updated like Debian is. If that can be harmonised with the "base system" nature of BSD and the flexibility of source compilation, so much the better. I suppose that discussion has been up in the air for ~ever and nobody's actually done it well. My only worry is figuring out how to run the rsync daemon safely. I'm a bit paranoid about running things on crater but I do agree that we would have to run the master rsync daemon there. From what I've read around, it seems GNU CVS is a real threat as it is. OpenBSD guys have made some effort in replacing it, but I don't know much about that either. Besides, as Joerg said, you can limit the privileges of rsync. --- Dmitri Nikulin Centre for Synchrotron Science Monash University Victoria 3800, Australia
Re: comparing cvsup vs. rsync
On Tue, Apr 10, 2007 at 08:32:55AM -0700, Matthew Dillon wrote: > My only worry is figuring out how to run the rsync daemon safely. > I'm a bit paranoid about running things on crater but I do agree > that we would have to run the master rsync daemon there. rsync allows running chrooted and under unprivileged uid. You specify the root of the subtrees in the config files, e.g. cvsroot and a cvs checkout updated via crontab. Joerg
Re: comparing cvsup vs. rsync
:I timed repeated retrievals of src from theshell.com over the past few :weeks, and here's the result. : :Retrieving all of src: :cvsup averaged about 11.5 minutes :rsync averaged about 19 minutes : :Retrieving only the last 24 hours of changes: :cvsup averaged about 18 seconds :rsync averaged about 25 seconds : :Caveats: I didn't test CPU usage. Also, this was with rsync 2.x - there's :a new version 3 on the way that is supposed to have improvments. : :So, it looks like rsync runs somewhat slower than cvsup, but not :catastrophically so. Also, rsync can't do checkouts of particular :revisions, so we'd have to have a certain checked out version of the :source to allow people to retrieve given releases of DragonFly. No big :surprises here, and no clear indicator we should change. At this point in time I think I would actually like to switch to a more mainstream distribution system, and rsync seems to be that system. I'm getting a bit tired of going through loops to support cvsup. We could just distribute the CVS tree and write a front-end utility in csh or sh that we distribute along with the rest of the system to do the nitty gritty work of actually checking something out into /usr/src. In fact, I think that would be preferable. My only worry is figuring out how to run the rsync daemon safely. I'm a bit paranoid about running things on crater but I do agree that we would have to run the master rsync daemon there. -Matt Matthew Dillon <[EMAIL PROTECTED]>
Chlamydia lost php-5.2.0?
Hi All, Trying to install php5 from pkgsrc (chlamydia), which failed. It seems that php-5.2.0.tgz is no longer there. ftp> cd pub/DragonFly/packages/RELEASE/i386/lang 250 CWD command successful. ftp> dir php-5.2.0.tgz 229 Entering Extended Passive Mode (|||50030|) 150 Opening ASCII mode data connection for '/bin/ls'. lrwxr-xr-x 1 505 505 20 Dec 22 19:14 php-5.2.0.tgz -> ../All/php-5.2.0.tgz 226 Transfer complete. ftp> dir ../All/php5-5.2.0.tgz 229 Entering Extended Passive Mode (|||50032|) 150 Opening ASCII mode data connection for '/bin/ls'. 226 Transfer complete. ftp> quit Mischa
comparing cvsup vs. rsync
Since cvsup is somewhat of a hassle to build, discussion comes up from time to time about switching to rsync. Rsync is generally accepted as slower/more resource intensive, but how much hasn't been quantified. I wanted to look into this in as un-bikesheddy a way as possible... I timed repeated retrievals of src from theshell.com over the past few weeks, and here's the result. Retrieving all of src: cvsup averaged about 11.5 minutes rsync averaged about 19 minutes Retrieving only the last 24 hours of changes: cvsup averaged about 18 seconds rsync averaged about 25 seconds Caveats: I didn't test CPU usage. Also, this was with rsync 2.x - there's a new version 3 on the way that is supposed to have improvments. So, it looks like rsync runs somewhat slower than cvsup, but not catastrophically so. Also, rsync can't do checkouts of particular revisions, so we'd have to have a certain checked out version of the source to allow people to retrieve given releases of DragonFly. No big surprises here, and no clear indicator we should change.
Re: [HEADS UP] Binary package changes
On Thu, Apr 05, 2007 at 05:32:59PM +0200, Grzegorz B?ach wrote: > I have set PKG_PATH to > "ftp://pkgsrc-box.org/packages/current/DragonFly-1.8/All"; > and exec "pkg_add -v bcrypt", after that I get this messages: Try http for now. I'll fix the FTP code in pkg_install to avoid the use of [] independently, but that takes a bit longer. Joerg
