Crater being upgraded - cvsup will be down for an hour or two
cvsup access will be down for the next hour or two while crater is being upgraded. I'm moving the cvs repository around a little to make room for a GIT and SVN mirror. -Matt
Re: New mirroring tool / cvsup alternative
On Tue, Jun 10, 2008 at 07:24:38AM +0200, Nicolas Thery wrote: > 2008/6/10 Vincent Stemen <[EMAIL PROTECTED]>: > > On Fri, Feb 01, 2008 at 02:20:55AM -0600, Vincent Stemen wrote: > >> Hi. > >> > >> I just thought I would let you guys know, I went ahead and put up > >> a simple web page for *mirror*. > >> > >> http://hightek.org/mirror > > > > In the process of updating our dcvs repository today with *mirror* and > > thinking about how nice it is to not be dependent on cvsup, I checked > > our web site today and discovered that the site for *mirror* was missing. > > I had not checked it in a while. The only thing I can think of is that > > it must have gotten overlooked when moving our data across when we > > changed hosting services a few months ago. > > > > Anyway, my apologies if anybody tried to download it and found it > > missing. It's back up now. > > By the way, I've switched to mirror for a few months now and I'm very > happy with it. Thanks > for writing it. Thanks for the feedback. I wondered if anybody was using it since nobody told me the web page was missing :-).
Re: Cvsup kernel source only?
Sascha Wildner wrote: Replace 'dragonfly-cvs-src' with 'dragonfly-src-sys' in the supfile you're using. Should have been: Replace 'dragonfly-cvs-src' with 'dragonfly-cvs-sys'... Sascha -- http://yoyodyne.ath.cx
Re: Cvsup kernel source only?
Andre LeClaire wrote: Hello, everyone! With the idea of creating a custom router, I've installed DragonFly on an older laptop with limited resources (1GB flash drive), and wonder if it's possible to build a custom kernel by cvsup-ing the kernel source only? This was possible with FreeBSD 4*, but all the info I've found describes getting the entire source tree for Dragonfly, which wouldn't fit in the space available. Replace 'dragonfly-cvs-src' with 'dragonfly-src-sys' in the supfile you're using. Seems to work (as I just learned recently). Although you'll also need the toplevel Makefiles in src/. Another option is to take the kernel src tarball which is on the LiveCD, /usr/src-sys.tar.bz2. Sascha -- http://yoyodyne.ath.cx
Re: Cvsup kernel source only?
Andre LeClaire wrote: cvsup-ing the kernel source only? This was possible with FreeBSD 4*, but all the info I've found describes getting the entire source tree for Dragonfly, which wouldn't fit in the space available. Thanks! Andre just do a regular CVS checkout.. e.g. 'co src/sys' good luck.
Re: rsync vs. cvsup benchmarks
Chris Turner wrote: I have some benchmark test results comparing rsync to cvsup. okay.. so like: you'd think with all of these repository copies flying around, there'd be a lot less flaming and a lot more coding going on.. enough! sheesh.. You people are making me want to write this email IN EMACS and, WE ALL KNOW HOW HORRIBLE EMACS IS !!! especially WHEN COMPARED TO VI !!! and also, just to finish this off (for now:) People on each side are LIKE NAZIS etc, etc, etc. LOL! I think I'd rather have polio that either EMACS or VI But you are right - try for new and better solutions. On which I say again HAMMER fs to HAMMER fs will probably have neat syncing features inbuilt. Pulling from HAMMER fs to OTOH, needs new tools if best use of HAMMER features is to be used advantageously. Eiffel should be good for coding that... (ducks and waddles away) ;-) Bill
Re: rsync vs. cvsup benchmarks
I have some benchmark test results comparing rsync to cvsup. ... okay.. so like: you'd think with all of these repository copies flying around, there'd be a lot less flaming and a lot more coding going on.. enough! sheesh.. You people are making me want to write this email IN EMACS and, WE ALL KNOW HOW HORRIBLE EMACS IS !!! especially WHEN COMPARED TO VI !!! and also, just to finish this off (for now:) People on each side are LIKE NAZIS etc, etc, etc.
Re: rsync vs. cvsup benchmarks
Simon 'corecode' Schubert wrote: Garance A Drosihn wrote: Just use rsync, and shut up about it already. What are you people blabering about? Fair question, simple answer. Not wanting to throw a useful tool out on specious grounds. > cvsup SUCKS. Not my field of expertise. Google 'Escort Services'. > not the idea, but the language it is implemented in. and cvsup inherits the suckage. as simple as that. if it was written in a portable language, nobody would bother using rsync. vince's benchmarks were just to establish one realisation: that rsync is not significantly worse than cvsup. end of story. move on. ?? the *language* [1]? 'inherits the suckage'? (so much for *that* view of birth control. ;-) '..nobody would *bother* using rsync.' [2] ?? '..establish one realisation' ??? 'realisation' indeed... Snort enough dried horeshit up your nose and you could come to believe that the whole damn WORLD stinks! But that's only from the observation point of your *own nose*. It doesn't make it so. Give this a think instead: HAMMER fs is expected to deliver capabilities that can reduce the workload required to ascertain what is/was 'of interest' as-at a specified tag and/or point in time. Built-in to the fs. *But neither cvsup/csup nor rsync as they presently stand are aware of that, nor equipped to take advantage of it.* So the bottom line is that a new 'none of the above' utility could be a very good thing to have. Especially if the client fs is other-than HAMMER fs. Until such time as that animal is coded, it just *might* be easier to adapt cvsup/csup than rsync. Embarassing? Or an inspiration to go off and code that tool? There is precedent in Plan9's specialized fs'en. No CVS repository needed per se. But please - keep the feet on the matching legs.. :-) Bill [1] See csup. In C. [2] cvsup/csup affecteth not rsync's utility one wit. Nor the reverse. Each is good at what it does best.
Re: rsync vs. cvsup benchmarks
Garance A Drosihn wrote: Just use rsync, and shut up about it already. What are you people blabering about? cvsup SUCKS. not the idea, but the language it is implemented in. and cvsup inherits the suckage. as simple as that. if it was written in a portable language, nobody would bother using rsync. vince's benchmarks were just to establish one realisation: that rsync is not significantly worse than cvsup. end of story. move on. -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: rsync vs. cvsup benchmarks
At 4:38 PM -0600 1/30/08, Vincent Stemen wrote: That's a good point. It is possible that cvsup would fair better with a matching sup directory. I actually forgot about cvsup keeping that separate state directory when I ran the benchmarks. However, from my viewpoint that does not invalidate the test results or convince me that there is any reason I would want to use cvsup for mirroring because of several reasons. 2 Rsync did not have the benefit of a local state directory either, so it was a one on one fair comparison. Based on all the cvsup claims, I would have expected it to at least come close to matching rsync's performance. Then I would expect a higher possibility of it being faster than rsync with the state directory available. Geez. Just use rsync if you want to use it. But the above reasoning is absurd. "CVSUP must live up to it's performance claims, even though I refuse to run CVSUP the way it is designed to run". "Rsync does not use this method to optimize its performance, so I refuse to let CVSUP use this method to optimize its performance. And look, CVSUP is slower than rsync as long as I make sure cvsup cannot use the performance enhancements which were designed into it!" Just use rsync, and shut up about it already. No one is asking you to use cvsup. But stop trying to defend a obviously incomplete benchmark by pulling out such bizarre reasoning. If you don't want to do a real benchmark, then just don't bother doing one. I can't blame you for that, as I also don't want to do the amount of work it would take to do a really useful benchmark. -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED]
Re: rsync vs. cvsup benchmarks
On Wed, Jan 30, 2008 at 09:54:29AM +, Rahul Siddharthan wrote: > As I understand, cvsup maintains state between updates using checkout > files in a separate "sup" directory. If you are missing that > directory, or it does not correspond to your "aged" tree, cvsup won't > do very well. You should test cvsup by checking out the tree, waiting > a week (or whatever the time is) to age it, and then updating it. > Likewise for rsync (though it doesn't require a checkouts directory). > > Rahul That's a good point. It is possible that cvsup would fair better with a matching sup directory. I actually forgot about cvsup keeping that separate state directory when I ran the benchmarks. However, from my viewpoint that does not invalidate the test results or convince me that there is any reason I would want to use cvsup for mirroring because of several reasons. 1 The state directory would have to make cvsup almost 4 times as fast to even match rsync's average performance, which I think is unlikely, considering it was only half as fast on a full download starting with an empty repository, and did not even match rsync on any of the update tests. 2 Rsync did not have the benefit of a local state directory either, so it was a one on one fair comparison. Based on all the cvsup claims, I would have expected it to at least come close to matching rsync's performance. Then I would expect a higher possibility of it being faster than rsync with the state directory available. 3 I think doing the comparison under robust conditions, and not under fragile or undependable circumstances is more realistic. For example, if you do an rsync update in between, your state directory is no longer going to match. I would not even trust the state directory not to break if you have to do an update using cvsup from a different server because the one you used last time is down. This is also the reason I did not include the '-s' switch to cvsup (which is also supposed to increase performance), because in the manual, it states, "If the -s option is used when one or more files have been modified locally, the results are undefined. Local file dam- age may remain uncorrected, updates may be missed, or cvsup may abort prematurely." I consider that to fragile. 4 All these fragile conditions to get cvsup up to maximum performance only applies to cvs repositores. If you ever decide to use a different revision control system... Considering the fragile nature and non-portability of cvsup, I am not very interested in using it even if it was a little faster than rsync. So I don't really have the time or motivation to generate benchmarks with the cvsup state directory. The way that would have to be tested would require weeks, and a lot more headache, generating and keeping all the matching state directories for each test. Vince
Re: rsync vs. cvsup benchmarks
At 6:38 AM + 1/30/08, Vincent Stemen wrote: My conclusions: === The results are dramatic, with rsync performing hundreds of percent faster on average while only loading the processor on the client side a little over a third as much as cvsup. Either the performance claims about cvsup being faster than rsync are based on theory without real world testing or cvsup has gotten a lot slower or rsync has gotten a lot faster than in the past. For those who are concerned about the validity of these results without including server side load tests and tests under bandwidth congestion conditions, here are my thoughts on the matter. No matter where a bottleneck exists in the transfer, whether it is server side load, client side load, or bandwidth limits, you are going to experience similar loss of throughput. The additional testing is nice to see, but you're not thinking the issues through far enough when it comes to scaling up a service like this. It's good to benchmark something which hasn't been tested in some time, but you have to do pretty extensive benchmarks if you're going to come to any sweeping conclusions for *all* uses of a program. Let's say rsync takes 10% of the cpu on the client, and 10% of the cpu on a server. Let's say cvsup for the same update takes 15% CPU on the client, and 7% on the server. If your benchmarks ignore the load on the server, then they can not possibly see problems which could occur when scaling up to more clients. With a single client connecting to the server, *neither* side is the bottleneck. It might be the disk-speed is the main bottleneck at that point. The update might take longer with cvsup due to the 15% CPU on the client, but the CPU isn't much of a *bottleneck* at that point. But with 10 connections in my fake scenario, rsync could be using 100% of the CPU on the server. It's at this point that rsync will see some bottleneck, while cvsup would only be using 70% of a CPU. Yes, cvsup will be using much more on each client, but then each client shows up with it's own CPU(s) to take up whatever load is thrown at that client. The server does not receive additional CPU's or network-cards for each connection that it accepts. Again, my feeling is that rsync is almost certainly fine for using with dragonfly's repository, given how much faster machines and networks have gotten, and how many simultaneous connections are seen for dragonfly repo-servers. It makes plenty of sense to stick with rsync if your servers are not overloaded. But if you want to prove rsync is better than cvsup for what the loads that cvsup was *MEANT* to solve, then your tests are not extensive enough. Benchmarking a client/server setup like cvsup is a lot of work to get a complete picture. Also note that you don't need to "prove" that rsync is "better". If it is good-enough for what Dragonfly needs at this point, then use it. It would be the people running the servers who will care about the load there, and if you have enough servers then dragonfly may never notice any problems from using rsync. Maybe a dragonfly repo-server will never see more than 10 simultaneous connections, so you'll never even hit the situations that freebsd had to deal with when cvsup was written. (I suspect there's a lot fewer people on dial-up connections now, for instance, so each individual connection will take a lot less wall-clock time than it used to, so you're much less likely to see 100 simultaneous connections). -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED]
Re: rsync considered superior (was: Re: rsync vs. cvsup benchmarks)
"Simon 'corecode' Schubert" <[EMAIL PROTECTED]> wrote: >Thank you for these thorough tests! We finally have some hard numbers to >work with. I think it is obvious that rsync should be the preferred >update mechanism if you want to download the cvs repository. To download, yes, to update, that's not so clear. To repeat my earlier mail: Vincent appears only to have installed a tarball of recent (but not current) sources and used cvsup / rsync to update them. But to operate efficiently, cvsup needs checkout files, which it would have only if it was run previously on those sources. See the FAQ: http://www.cvsup.org/faq.html in particular, numbers 37 and 38: In order to update your files efficiently, CVSup needs to know what you've already got. It stores this information in files called "checkouts" files... If CVSup can't find a checkouts file that it needs, it falls back on other methods of determining which files you have. One such method is to compute checksums (MD5 file signatures) for each of your files, and use those to figure out which file revisions you have. This is perfectly safe, but it is inefficient. It slows down your update and also puts a heavier load on the server. To compare cvsup minus checkout-files to rsync seems quite misplaced. Most people will never use cvsup merely to download sources: they will use it to keep them regularly up-to-date. Rahul
Re: rsync vs. cvsup benchmarks
Justin C. Sherrill wrote: The only minor thing I'd bring up is that I recall one reason for cvsup is that rsync placed a relatively higher load per client on the server. That needs to be established. We already heard that cvsup - contrary to claims - is not competitive with rsync, on the client side. So I can very well believe that this is also true for the server side. I for myself always notice that when syncing from chlamydia, the server basically traverses all 60k files *instantly*, while it takes quite some time on my desktop. So the load doesn't seem to be a problem once the directory structure is in the buffer cache. Of course, that may complaint may date from when people only had 400Mhz CPUs and older versions of rsync, so I doubt it's a strong reason to stay with cvsup any more. Quick test on chlamydia: rsync of already synced repo: % time rsync --delete -aH chlamydia.fs.ei.tum.de::dragonfly-cvs . rsync --delete -aH chlamydia.fs.ei.tum.de::dragonfly-cvs . 0.72s user 1.43s system 36% cpu 5.865 total considering that rsync spends half of the time on the local side, that's < 3s of load on the server: 53331 nobody 161 0 4536K 3888K select 0:00 355.68% 33.89% rsync nobody cares about that. it might take some more cycles when transfering, but so what. seriously. I don't care, this is peanuts. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: rsync vs. cvsup benchmarks
On Wed, January 30, 2008 1:38 am, Vincent Stemen wrote: > > I have some benchmark test results comparing rsync to cvsup. I did 12 > client side tests over the last week. 5 against TheShell.com, 3 against > AllBSD.org, and 4 against chlamydia.fs.ei.tum.de. All tests were > mirroring the DragonFly BSD source repository. The tests were done > with various aged repositories at different times of the day and > night, some with compression on and some with it off. Each test was > done by unpacking two identical copies of a given aged > repository, one to run the cvsup test on and one to run the rsync test on. > Then the rsync and cvsup test was run back to back. The only minor thing I'd bring up is that I recall one reason for cvsup is that rsync placed a relatively higher load per client on the server. Of course, that may complaint may date from when people only had 400Mhz CPUs and older versions of rsync, so I doubt it's a strong reason to stay with cvsup any more.
rsync considered superior (was: Re: rsync vs. cvsup benchmarks)
Hello Vincent, Vincent Stemen wrote: The results are dramatic, with rsync performing hundreds of percent faster on average while only loading the processor on the client side a little over a third as much as cvsup. Either the performance claims about cvsup being faster than rsync are based on theory without real world testing or cvsup has gotten a lot slower or rsync has gotten a lot faster than in the past. Thank you for these thorough tests! We finally have some hard numbers to work with. I think it is obvious that rsync should be the preferred update mechanism if you want to download the cvs repository. Cvsup might still be better suited when only downloading the checked out sources. To state it clearly for everybody: = Use rsync to sync your repos! It is faster and can even be compiled! = cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: rsync vs. cvsup benchmarks
Vincent Stemen <[EMAIL PROTECTED]> wrote: >I have some benchmark test results comparing rsync to cvsup. I did 12 client >side tests over the last week. 5 against TheShell.com, 3 against AllBSD.org, >and 4 against chlamydia.fs.ei.tum.de. All tests were mirroring the DragonFly >BSD source repository. The tests were done with various aged repositories at >different times of the day and night, some with compression on and some with it >off. Each test was done by unpacking two identical copies of a given aged >repository, one to run the cvsup test on and one to run the rsync test on. As I understand, cvsup maintains state between updates using checkout files in a separate "sup" directory. If you are missing that directory, or it does not correspond to your "aged" tree, cvsup won't do very well. You should test cvsup by checking out the tree, waiting a week (or whatever the time is) to age it, and then updating it. Likewise for rsync (though it doesn't require a checkouts directory). Rahul
rsync vs. cvsup benchmarks
I have some benchmark test results comparing rsync to cvsup. I did 12 client side tests over the last week. 5 against TheShell.com, 3 against AllBSD.org, and 4 against chlamydia.fs.ei.tum.de. All tests were mirroring the DragonFly BSD source repository. The tests were done with various aged repositories at different times of the day and night, some with compression on and some with it off. Each test was done by unpacking two identical copies of a given aged repository, one to run the cvsup test on and one to run the rsync test on. Then the rsync and cvsup test was run back to back. === * Environment * === All tests were done in the following environment: DragonFly 1.10.1-RELEASE system with a 1.11.0-DEVELOPMENT kernel. CPU:Intel(R) Pentium(R) 4 CPU 1.60GHz Inernet connection: Cox cable rsync version: 2.6.9 rsync directories: CVSROOT, doc, and src rsync flags:--archive --hard-links --delete --verbose + --compress (on tests with compression on) cvsup version: SNAP_16_1h cvsup supfile: DragonFly-cvs-supfile that comes with DragonFly BSD cvsup file collections: dragonfly-cvs-root dragonfly-cvs-src dragonfly-cvs-doc === * Summary * === On average: === cvsup took 3.76 times as long as rsync making rsync 276% faster. rsync consumed 13.5% of the cpu time on updates to an existing repository. cvsup consumed 33.7% of the cpu time on updates to an existing repository. Minimum performance difference: cvsup took from 1.34 times as long as rsync. Maximum performance difference: cvsup took 9.1 times as long as rsync rsync's best performance was on repositories that are less than a week old, where there was not a large number of updates. In these cases it was usually from 4 to 7 times faster than cvsup, except for one test from allbsd.org where rsync was only about 40% faster. On a Full download of the entire repository, where the tools are not having to check for file differences, the CPU loads were so low and similar enough that I would consider the differences between rsync and cvsup insignificant, so I excluded those two tests from the cpu load average calculation above. However, rsync was still more then twice as fast at completing the entire download. cvsup, on average, consumed 250% of the load that rsync consumed but did not outperform rsync on one single test. Of course, this is client side only. I will leave it up to somebody else to do server side load tests. My conclusions: === The results are dramatic, with rsync performing hundreds of percent faster on average while only loading the processor on the client side a little over a third as much as cvsup. Either the performance claims about cvsup being faster than rsync are based on theory without real world testing or cvsup has gotten a lot slower or rsync has gotten a lot faster than in the past. For those who are concerned about the validity of these results without including server side load tests and tests under bandwidth congestion conditions, here are my thoughts on the matter. No matter where a bottleneck exists in the transfer, whether it is server side load, client side load, or bandwidth limits, you are going to experience similar loss of throughput. So, I would consider client side only tests reasonably valid at determining overall average performance differences when both tools are run back to back under the same conditions. If rsync is shifting more of the processing load to the server than cvsup or consuming more bandwidth, well, it can bog the server down or consume more bandwidth by at least an *extra* 150% or more before it even slows down enough to match cvsup's performance on the average. And, of course, that is only if the bandwidth or server load is hitting saturation. * Test Results * repository age: about 1.5 hours site: TheShell.com compression:off (Probably made no difference since there were apparently no updates) date: Thu Jan 17 20:29:40 CST 2008 rsync time: 1.34s user 3.41s system 13% cpu 34.846 total date: Thu Jan 17 20:17:12 CST 2008 cvsup time: 54.17s user 26.03s system 36% cpu 3:40.77 total = cvsup took 6.33 times as long as rsync rsync was 533% faster cvsup consumed 2.77 times the cpu load of rsync = repository age: about 2 days site: TheShell.com compression:on date: Thu Jan 17 18:50:11 CST 2008 rsync time: 8.29s user 13.31s system 17% cpu 2:03.07 total date: Thu Jan 17 19:13:10 CST 2008 cvsup time: 155.08s user 69.40s system 40% cpu 9:14.73 total = cvsup took 4.5 times as long as rsync rsync was 3
Re: cvsup: theshell.com missing dragonfly-cvs-doc
On 2008-01-23, Peter Avalos <[EMAIL PROTECTED]> wrote: > On Tue, Jan 22, 2008 at 11:41:20PM +, Vincent Stemen wrote: >> Any idea why cvsup.theshell.com is missing the dragonfly-cvs-doc collecti= > on? >>=20 >> I get >> Server message: Unknown collection "dragonfly-cvs-doc" >>=20 >> The doc directory is there via rsync. It is not missing from other >> servers such as cvsup.dragonflybsd.org and cvsup.allbsd.org. >>=20 > > Probably because the server admin screwed up. ;) > > Try it now and let me know. > > --Peter Yep. It's working now :-)
Re: cvsup: theshell.com missing dragonfly-cvs-doc
On Tue, Jan 22, 2008 at 11:41:20PM +, Vincent Stemen wrote: > Any idea why cvsup.theshell.com is missing the dragonfly-cvs-doc collection? > > I get > Server message: Unknown collection "dragonfly-cvs-doc" > > The doc directory is there via rsync. It is not missing from other > servers such as cvsup.dragonflybsd.org and cvsup.allbsd.org. > Probably because the server admin screwed up. ;) Try it now and let me know. --Peter pgpVO8LMJ18BX.pgp Description: PGP signature
cvsup: theshell.com missing dragonfly-cvs-doc
Any idea why cvsup.theshell.com is missing the dragonfly-cvs-doc collection? I get Server message: Unknown collection "dragonfly-cvs-doc" The doc directory is there via rsync. It is not missing from other servers such as cvsup.dragonflybsd.org and cvsup.allbsd.org.
Re: rsync and new mirroring tool (was cvsup)
:Yah. Please let's do a round robin entry to all cvsup mirrors. : :cheers : simon Unfortunately it isn't safe to do that because the mirrors will be slightly out of sync with each other. You would confuse the hell out of cvsup if you ran it at the wrong time. It's better to select one manually. Again, I'm not really all that worried about my meager bandwidth. My router does fair-share scheduling and my servers all have net.inet.tcp.inflight_enable turned on. The net effect is that the worse that happens is you get a slow download. Shell responsiveness shouldn't go down much. -Matt Matthew Dillon <[EMAIL PROTECTED]>
Re: rsync and new mirroring tool (was cvsup)
Dave Hayes wrote: host cvsup.dragonflybsd.org cvsup.dragonflybsd.org is a nickname for crater.dragonflybsd.org crater.dragonflybsd.org has address 216.240.41.25 crater.dragonflybsd.org mail is handled (pri=10) by crater.dragonflybsd.org Given your desired policy above, I'd say moving that CNAME to a higher bandwidth mirror would take some of the load off of your link. The CNAME is, of course, what I use in my automatic cron job...and the principle of least surprise suggests that I shouldn't be overloading anyone's link using a provided example. Yah. Please let's do a round robin entry to all cvsup mirrors. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: rsync and new mirroring tool (was cvsup)
Matthew Dillon <[EMAIL PROTECTED]> writes: > #2. It's ok for crater to be listed, but it should be commented out > for now. Manual use of crater is fine, its the automatic cron jobs > that I'd like to avoid :-) Hrm... > cd /usr/share/examples/cvsup > grep "default host" DragonFly* DragonFly-cvs-supfile:*default host=cvsup.dragonflybsd.org DragonFly-preview-supfile:*default host=cvsup.dragonflybsd.org DragonFly-release1_2-supfile:*default host=cvsup.dragonflybsd.org DragonFly-release1_4-supfile:*default host=cvsup.dragonflybsd.org DragonFly-release1_6-supfile:*default host=cvsup.dragonflybsd.org DragonFly-release1_8-supfile:*default host=cvsup.dragonflybsd.org DragonFly-src-supfile:*default host=cvsup.dragonflybsd.org > host cvsup.dragonflybsd.org cvsup.dragonflybsd.org is a nickname for crater.dragonflybsd.org crater.dragonflybsd.org has address 216.240.41.25 crater.dragonflybsd.org mail is handled (pri=10) by crater.dragonflybsd.org Given your desired policy above, I'd say moving that CNAME to a higher bandwidth mirror would take some of the load off of your link. The CNAME is, of course, what I use in my automatic cron job...and the principle of least surprise suggests that I shouldn't be overloading anyone's link using a provided example. Perhaps I was mistaken to presume that the example would point to something that would tolerate cron jobs? :) -- Dave Hayes - Consultant - Altadena CA, USA - [EMAIL PROTECTED] >>> The opinions expressed above are entirely my own <<< Nasrudin was driving a friend in his car at a spanking pace. Suddenly, glimpsing a signpost, the friend called out "Mulla, we're going in the wrong direction!" "Why don't you ever think of something good?" came the reply. "Just look, for instance, at the speed we are going at."
Re: rsync and new mirroring tool (was cvsup)
:Understood. I suspected something like that might be the case since it :was not in the download site list on dragonflybsd.org. That is why :I put it last :-). : :Mirror will use the first site in the list by default unless you specify :one of the others on the command line. For the default config file, :I could do one of three options: : :1. Leave it last in the list and add what you said as a comment to : make it convenient for developers and mirrors who might need to : use it. :2. Same as 1 but have crater commented out by default. :3. Remove crater altogether from the default configuration file. : :What would you guys prefer? #2. It's ok for crater to be listed, but it should be commented out for now. Manual use of crater is fine, its the automatic cron jobs that I'd like to avoid :-) -Matt Matthew Dillon <[EMAIL PROTECTED]>
Re: cvsup
Simon 'corecode' Schubert wrote: Bill Hacker wrote: *trimmed* The bottom line - viewing it not a a coder, which I haven't been for around 30 years - but as a Manager of scarce resources - primarily *time*, and not even my own in this case - is that: - Apparent: No readily available 'one size' fits all needs well enough to justify excluding others. - Apparent: There is enough gain to justify running parallel options - Suspected: There is NOT PRESENTLY enough 'assured' gain to attract the support needed to run more than one (or even *any*) as well as they need to be run order to secure value. See the comment about the existing git repository not being current... A good idea, bed-ridden, is less mobile a bandaged-up idea on crutches. Even so - it should NOT be necessary to chose 'one and only' one, e.g. move off CVS to . The missing resource is not addressed by taking any one coder away from coding to maintain a repository - nor creating a new toolset from scratch. It *may* be solvable by enlisting several coders to digress long enough to invest in the automation that makes it possible for less effort in future to keep at least one alternative repository system in top form. I do not know if that should be git or some other CVS alternative. But the payback has to become apparent in months if not weeks - no more than that - ELSE it is 'faster' to continue to deal with the Devil one knows than retrain - and *still* struggle. Even so - someone with admin skills AND understanding of what matters AND how it has to work, AND daily availability to stay on top of it, i.e. almost by definition NOT a coder - would be needed to keep it sorted. That's the issue as I see it - need for what we 'Politically Incorrect' old farts used to (be allowed to) call a 'Gal Friday'. - but analysing a barrier isn't the same as removing it, and I am in no better position to actually fix it than anyone else that has yet spoken. That said - while one cannot 'herd cats' or 'direct' the efforts of volunteers... perhaps 'will' and 'need' can enlist enough consensus to move a solution out of 'wish for' and into 'useful' land. pkgsrc revanche. It has paid-off. ELSE - do the best that can be done with existing tools for a while longer. Bill
Re: cvsup
walt wrote: Anyway, you created a DragonFly-git repo at http://repo.or.cz but it is out of date now. They do offer an automatic update service, which sounds very good: "In the mirror mode, we will check the remote repository at the URL you give us every hour and if we spot any changes, we will grab them, mirror them and show them in our gitweb interface." Is there any reason not to take advantage of their very kind offer? It is running in mirror mode, but for that I'd have to keep updating the git repo I used to generate upstream. Now my repo converter is fairly complete, but it missed one or the other manual repo activity :/, so the git repo is unhappy right now. Plus there is a bug somewhere which makes it eat memory (when converting freebsd), like homer is eating donuts. So either somebody will give me a hand or it'll have to wait until I have enough time to fix it myself. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: cvsup
Simon 'corecode' Schubert wrote: .. Oh yes, everybody who EVER tried adding third party software to the repo, or rather kept maintaining it, has been swearing on CVS... Heh. That's ambiguous in colloquial English, which admittedly makes no sense and therefore is difficult to learn. You wanted to say 'swearing at', as in hurling epithets 'at' someone. There is also 'swearing by' something, which means exactly the opposite, e.g. 'I swear by DragonFly -- it's the best OS I've used'. There is also 'swearing on' a Bible to take an oath -- I suppose because you put your right hand 'on' the Bible during the oath. Anyway, you created a DragonFly-git repo at http://repo.or.cz but it is out of date now. They do offer an automatic update service, which sounds very good: "In the mirror mode, we will check the remote repository at the URL you give us every hour and if we spot any changes, we will grab them, mirror them and show them in our gitweb interface." Is there any reason not to take advantage of their very kind offer?
Re: rsync and new mirroring tool (was cvsup)
On Mon, Jan 21, 2008 at 09:00:31PM +, Vincent Stemen wrote: > > 1. Leave it last in the list and add what you said as a comment to >make it convenient for developers and mirrors who might need to >use it. > 2. Same as 1 but have crater commented out by default. > 3. Remove crater altogether from the default configuration file. > > What would you guys prefer? > #3 pgpSPegmJK3t1.pgp Description: PGP signature
Re: rsync and new mirroring tool (was cvsup)
On 2008-01-21, Simon 'corecode' Schubert <[EMAIL PROTECTED]> wrote: > Vincent Stemen wrote: >> sites += crater.dragonflybsd.org::dragonfly_cvs > > could you please not mirror off crater? Matt's link is quite resource > constrained and should mainly be used for feeding mirrors and > developers. Typing in a shell with high latency sucks :) > > cheers > simon Understood. I suspected something like that might be the case since it was not in the download site list on dragonflybsd.org. That is why I put it last :-). Mirror will use the first site in the list by default unless you specify one of the others on the command line. For the default config file, I could do one of three options: 1. Leave it last in the list and add what you said as a comment to make it convenient for developers and mirrors who might need to use it. 2. Same as 1 but have crater commented out by default. 3. Remove crater altogether from the default configuration file. What would you guys prefer?
Re: cvsup
:... :> Hmm, I haven't used mergemaster since the upgrade target went :> into /usr/src/Makefile to replace it. I'm mildly surprised to find it's :> still in the system, is it of any actual use ? : :Absolutely. There is no other way to merge config files, for instance :ssh, sshd, inetd, etc. : :cheers : simon Er. Well, I haven't kept it up to date for years. 'make upgrade' (after an installworld) is all I use. -Matt Matthew Dillon <[EMAIL PROTECTED]>
Re: cvsup
Bill Hacker wrote: >> I can't run a patch and work on a different issue >> myself - I'll mix both. Or I'll have to check out into another tree and >> lose the patch. > Indeed. And have to go find it and manually re-apply, and/or alter and > re-apply 'coz it no longer fits quite tha same on code that has moved on > in other details. And once you've applied it, it is mixed with your existing changes. That's the main problem. > But unless and until the patch is vetted and accepted into the > mainstream that is exactly what a 'production' user wants. A production user doesn't want patches or something else. He wants working binaries. > As said - developers and production users have different priorities and > CVS is a fairly effective compromise - ELSE it would have been scrapped > long ago by a lot more projects than have done so to date. No, that's not true. It's just that developers are lazy and don't want to learn new, potentially better tools. They are also conservative and don't want to change one tool for the other without real benefit. Several projects did the mistake to convert from cvs to svn, which didn't really improve productivity (I guess). >>> Rather than 'nag' - set up what you want and see who joins or lends a >>> hand. >> It is really cumbersome to keep any repo synchronized, especially if you >> want to have a nice repo which reflects vendor branches correctly. > Understood. But *any* repo and any toolset needs effort. No doubt. The differences, however, are orders of magnitude. >> Basically all manual CVS interference has to be dealt with either in the >> tool or manually. > Alternatives may provide more comfortable knobs and buttons - but AFAIK, > none of them yet read minds, let alone cover the sharp edges. Oh yes, they do cover sharp edges. They might have other, different ones, but not as many and not as sharp (I'm obviously biased). > But if your peer contributors - who must have nearly identical concerns > - don't yet see CVS as a target for 'real soon now' replacement, there > must not (yet) be an overwhelming case for change. Oh yes, everybody who EVER tried adding third party software to the repo, or rather kept maintaining it, has been swearing on CVS. Even Matt decided that CVS was evil and started the whole patch thing. The patches helped on one side but are hurting on the other side - much more. The real salvation is a version control system which will help the developers in their tasks - not stand them in the way. > Change can't take place until it is possible for those involved to eval > both tools side-by-side - on DragonFly, not 'other project x' - until > they are convinced the advantage is worthwhile AND will not make it > harder in general to use. No doubt. Bear in mind that every new tool needs some effort to learn. This means that everybody involved has to accept the fact that they need to put some effort in in order to be able to fairly evaluate the tool. So a learning curve has to be expected. > I'm not defending CVS in particular. I'm just saying if *most* folks > don't see as broken a fix won't get a lot of followers. People still program in C. People keep writing shell scripts. *Most* people don't realize the shortcomings of the tools they are using because they a) they don't reflect on their workflows and they are b) too lazy to check out alternatives to realize there is help. > .or we would have left 'C' for something else about fifteen years ago, > let alone the x86 archeologitecture. You exactly nailed the point. People don't want to move. They don't want to learn. Even if the alternative would be orders of magnitude better. cheers simon
Re: cvsup
Steve O'Hara-Smith wrote: >>> Maybe you need to install from a CD to get this makefile? >> I guess so. Nothing does a make distribution except for mergemaster, >> and mergemaster doesn't merge /usr, I think. > Hmm, I haven't used mergemaster since the upgrade target went > into /usr/src/Makefile to replace it. I'm mildly surprised to find it's > still in the system, is it of any actual use ? Absolutely. There is no other way to merge config files, for instance ssh, sshd, inetd, etc. cheers simon
Re: cvsup
On Mon, 21 Jan 2008 13:59:12 +0100 "Simon 'corecode' Schubert" <[EMAIL PROTECTED]> wrote: > Nicolas Thery wrote: > > Maybe you need to install from a CD to get this makefile? > > I guess so. Nothing does a make distribution except for mergemaster, > and mergemaster doesn't merge /usr, I think. Hmm, I haven't used mergemaster since the upgrade target went into /usr/src/Makefile to replace it. I'm mildly surprised to find it's still in the system, is it of any actual use ? -- C:>WIN | Directable Mirror Arrays The computer obeys and wins.| A better way to focus the sun You lose and Bill collects. |licences available see |http://www.sohara.org/
Re: cvsup
Simon 'corecode' Schubert wrote: Bill Hacker wrote: CVS has been the 'compromise' that is at least not harmful or overly demanding. CVS *is* harmful. To you, and other running experimental differences, perhaps so.. I can't run a patch and work on a different issue myself - I'll mix both. Or I'll have to check out into another tree and lose the patch. Indeed. And have to go find it and manually re-apply, and/or alter and re-apply 'coz it no longer fits quite tha same on code that has moved on in other details. But unless and until the patch is vetted and accepted into the mainstream that is exactly what a 'production' user wants. As few surprises as possible. As said - developers and production users have different priorities and CVS is a fairly effective compromise - ELSE it would have been scrapped long ago by a lot more projects than have done so to date. The alternatives are no longer new, nor are they without their own set of irritants. Rather than 'nag' - set up what you want and see who joins or lends a hand. It is really cumbersome to keep any repo synchronized, especially if you want to have a nice repo which reflects vendor branches correctly. Understood. But *any* repo and any toolset needs effort. Basically all manual CVS interference has to be dealt with either in the tool or manually. Alternatives may provide more comfortable knobs and buttons - but AFAIK, none of them yet read minds, let alone cover the sharp edges. If it adds enough value to enough people, bandwidth and storage will be attracted to the solution. Bandwidth and storage isn't the issue. I can develop forever in my git repo, and nobody might ever notice. And it won't magically make the project switch from CVS. cheers simon From tracking your work, I fully appreciate that you have made a great deal of effort, committed a lot of time and resources, and delivered much valuable code - so yes = anything that removes barriers to that gets some support. But if your peer contributors - who must have nearly identical concerns - don't yet see CVS as a target for 'real soon now' replacement, there must not (yet) be an overwhelming case for change. Change can't take place until it is possible for those involved to eval both tools side-by-side - on DragonFly, not 'other project x' - until they are convinced the advantage is worthwhile AND will not make it harder in general to use. I'm not defending CVS in particular. I'm just saying if *most* folks don't see as broken a fix won't get a lot of followers. .or we would have left 'C' for something else about fifteen years ago, let alone the x86 archeologitecture. ;-) Bill
Re: cvsup
Nicolas Thery wrote: >>>>> What, people didn't know we install a Makefile in /usr? Well, >>>>> now you do! >>>> Er...maybe it's because I'm running 1.8.2 that I don't see one in /usr? >>>> When did you folks start doing this? >>> 1.10 IIRC. >> Eh? I thought that was a joke. There's no /usr/Makefile on this box >> (1.11.0-PREVIEW). > I'm pretty sure I used /usr/Makefile yesterday to run cvsup on a > 1.10.1 box. I don't have access to that box right now. I'll check > tonight. > > Maybe you need to install from a CD to get this makefile? I guess so. Nothing does a make distribution except for mergemaster, and mergemaster doesn't merge /usr, I think. cheers simon
Re: cvsup
Bill Hacker wrote: > CVS has been the 'compromise' that is at least not harmful or overly > demanding. CVS *is* harmful. I can't run a patch and work on a different issue myself - I'll mix both. Or I'll have to check out into another tree and lose the patch. > Rather than 'nag' - set up what you want and see who joins or lends a hand. It is really cumbersome to keep any repo synchronized, especially if you want to have a nice repo which reflects vendor branches correctly. Basically all manual CVS interference has to be dealt with either in the tool or manually. > If it adds enough value to enough people, bandwidth and storage will be > attracted to the solution. Bandwidth and storage isn't the issue. I can develop forever in my git repo, and nobody might ever notice. And it won't magically make the project switch from CVS. cheers simon
Re: rsync and new mirroring tool (was cvsup)
Vincent Stemen wrote: > sites += crater.dragonflybsd.org::dragonfly_cvs could you please not mirror off crater? Matt's link is quite resource constrained and should mainly be used for feeding mirrors and developers. Typing in a shell with high latency sucks :) cheers simon
Re: cvsup
Simon 'corecode' Schubert wrote: Matthew Dillon wrote: One of the original reasons for using cvsup was so people could maintain local branches of the repository. I don't think people do this much anymore, if they do it all. Disk space is so cheap these days that keeping a master sync copy and a separate one for local work is not a big deal. People use git or hg nowadays :) I just can't stop nagging, it is unbelievably useful, especially for team work. cheers simon Several such options can co-exist. But developers and 'deployer/maintainer' admins have different needs. - The devoloper needs fine-grain control of diverse options - The end-user / admin needs protection from breakage or stupidity, doesn't want to know anything about granularity beyond CPU family and release number. CVS has been the 'compromise' that is at least not harmful or overly demanding. Rather than 'nag' - set up what you want and see who joins or lends a hand. If it adds enough value to enough people, bandwidth and storage will be attracted to the solution. If not, not. Yet. ;-) Bill
Re: cvsup
Matthew Dillon wrote: People shouldn't worry about server side overhead all that much. Cpu cycles are cheap and the cvs tree is completely cached in memory anyway. And the only effect that extra network bandwidth has is that it takes a little longer to run the operation. Now, granted, we don't want people to be downloading a whole copy's worth of network bandwidth every day, but rsync is close enough that I just don't care. One of the original reasons for using cvsup was so people could maintain local branches of the repository. I don't think people do this much anymore, if they do it all. Disk space is so cheap these days that keeping a master sync copy and a separate one for local work is not a big deal. -Matt Not that DragonFly is not already usable - but the time is approaching - perhaps Q3 or Q4 2008 - when enough 'good stuff' will converge to give it a distinct 'edge'. Especially if the 'major' *BSD doesn't soon find its way back to a more predictable and less fragile model. *Then* the bandwidth and all could very much matter. OTOH, enlisting more mirrors and managing them appropriately can probably happen rather rapidly if/as/when that sort of avalanche begins. PP Bill
Re: cvsup
2008/1/21, Steve O'Hara-Smith <[EMAIL PROTECTED]>: > On Mon, 21 Jan 2008 09:24:02 +0100 > "Nicolas Thery" <[EMAIL PROTECTED]> wrote: > > > 2008/1/21, Dave Hayes <[EMAIL PROTECTED]>: > > > Matthew Dillon <[EMAIL PROTECTED]> writes: > > > > What, people didn't know we install a Makefile in /usr? Well, > > > > now you do! > > > > > > Er...maybe it's because I'm running 1.8.2 that I don't see one in /usr? > > > When did you folks start doing this? > > > > 1.10 IIRC. > > Eh? I thought that was a joke. There's no /usr/Makefile on this box > (1.11.0-PREVIEW). I'm pretty sure I used /usr/Makefile yesterday to run cvsup on a 1.10.1 box. I don't have access to that box right now. I'll check tonight. Maybe you need to install from a CD to get this makefile? Cheers, Nicolas
Re: cvsup
* Steve O'Hara-Smith wrote: > On Mon, 21 Jan 2008 09:24:02 +0100 > "Nicolas Thery" <[EMAIL PROTECTED]> wrote: > > > 2008/1/21, Dave Hayes <[EMAIL PROTECTED]>: > > > Matthew Dillon <[EMAIL PROTECTED]> writes: > > > > What, people didn't know we install a Makefile in /usr? Well, > > > > now you do! > > > > > > Er...maybe it's because I'm running 1.8.2 that I don't see one in /usr? > > > When did you folks start doing this? > > > > 1.10 IIRC. > > Eh? I thought that was a joke. There's no /usr/Makefile on this box > (1.11.0-PREVIEW). Nope, its not :) Look at the cvs log of src/etc/Makefile.usr: revision 1.1 date: 2007-08-02 08:53:14 +0200; author: dillon; state: Exp; commitid: 9C2uflYNvgKK29ss; The distribution installs a Makefile in /usr with easy-to-use targets to create a pkgsrc distribution and to obtain the DragonFly CVS repository and checkout/update /usr/src. A make in /usr with no arguments will list available options. Matthias
Re: cvsup
On Mon, 21 Jan 2008 09:24:02 +0100 "Nicolas Thery" <[EMAIL PROTECTED]> wrote: > 2008/1/21, Dave Hayes <[EMAIL PROTECTED]>: > > Matthew Dillon <[EMAIL PROTECTED]> writes: > > > What, people didn't know we install a Makefile in /usr? Well, > > > now you do! > > > > Er...maybe it's because I'm running 1.8.2 that I don't see one in /usr? > > When did you folks start doing this? > > 1.10 IIRC. Eh? I thought that was a joke. There's no /usr/Makefile on this box (1.11.0-PREVIEW). -- C:>WIN | Directable Mirror Arrays The computer obeys and wins.| A better way to focus the sun You lose and Bill collects. |licences available see |http://www.sohara.org/
Re: cvsup
> > People use git or hg nowadays :) I just can't stop nagging, it is > unbelievably useful, especially for team work. It's been now many times I heared great things about git, while I never personally used it, as I use rsync to keep my development projects syncronized. Why don't we switch to to git now? Think we would be the first *BSD OS project to dump CVS and not that it would be a bad thing by the sounds of it :) Petr
Re: rsync and new mirroring tool (was cvsup)
On 2008-01-21, Matthew Dillon <[EMAIL PROTECTED]> wrote: > We could just add rsync targets in our /usr/Makefile in addition to > all the cvsup/pkgsrc targets already in there. > > What, people didn't know we install a Makefile in /usr? Well, now you > do! > > -Matt Also, before you decide anything, I should let you know that I decided to go ahead and write a general purpose rsync based mirroring tool, both for keeping our own dcvs copy updated and for doing convenient rsync vs cvsup benchmark testing. I have been working on it for two nights now. It is written in Perl and, so far is working pretty nice. I call it *mirror*. I have a configuration file for it that is pre-configured for all the DragonFly rsync repository mirroring sites that I know of so far. It is even more convenient than cvsup. I plan to make it available, hopefully in the next day or two. I was going to set it up as a standard installable source package (i.e. make install) and write a manual first. I was considering hosting it as a project on my own site or one of the project hosting sites or see if you guys are interested in putting it up on the DragonFly site. I would be interested in feedback. It has some pretty nice features so far. Just to give you a little preview, I have included just the configuration file below for DragonFly dcvs mirroring. === # mirror-dragonfly-cvs.conf # # Settings are specified as # variable = value # or # variable += value # to append to a variable. # # This mirror file will maintain a copy of the DragonFly BSD CVS tree. # # $Id$ # # $sites # Source URLs or paths for retrieving file updates. # Example: rsync.TheShell.com::DragonFly/dcvs # or: rsync://rsync.TheShell.com/DragonFly/dcvs # sites = rsync.TheShell.com::DragonFly/dcvs sites += rsync.AllBSD.org::dragonfly-cvs sites += chlamydia.fs.ei.tum.de::dragonfly-cvs sites += crater.dragonflybsd.org::dragonfly_cvs # $destination # Specifies where to place the requested files. # destination = /home/dcvs # $rsync_opts # Options to pass to rsync # rsync_opts += --archive --hard-links --delete rsync_opts += --compress rsync_opts += --verbose #rsync_opts += --progress # $files # Files or directories to mirror # files = CVSROOT# Basic CVS data. Required to use cvs. files += src# The DragonFly source code files += doc# Documentation #files += site # The DragonFly website code
Re: cvsup
2008/1/21, Dave Hayes <[EMAIL PROTECTED]>: > Matthew Dillon <[EMAIL PROTECTED]> writes: > > What, people didn't know we install a Makefile in /usr? Well, now you > > do! > > Er...maybe it's because I'm running 1.8.2 that I don't see one in /usr? > When did you folks start doing this? 1.10 IIRC.
Re: cvsup
Matthew Dillon wrote: One of the original reasons for using cvsup was so people could maintain local branches of the repository. I don't think people do this much anymore, if they do it all. Disk space is so cheap these days that keeping a master sync copy and a separate one for local work is not a big deal. People use git or hg nowadays :) I just can't stop nagging, it is unbelievably useful, especially for team work. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: cvsup
Matthew Dillon <[EMAIL PROTECTED]> writes: > What, people didn't know we install a Makefile in /usr? Well, now you > do! Er...maybe it's because I'm running 1.8.2 that I don't see one in /usr? When did you folks start doing this? -- Dave Hayes - Consultant - Altadena CA, USA - [EMAIL PROTECTED] >>> The opinions expressed above are entirely my own <<< A king who feared wasps once decreed that they would be abolished. As it happened, they did him no harm. But he was eventually stung to death by scorpions.
Re: cvsup
People shouldn't worry about server side overhead all that much. Cpu cycles are cheap and the cvs tree is completely cached in memory anyway. And the only effect that extra network bandwidth has is that it takes a little longer to run the operation. Now, granted, we don't want people to be downloading a whole copy's worth of network bandwidth every day, but rsync is close enough that I just don't care. One of the original reasons for using cvsup was so people could maintain local branches of the repository. I don't think people do this much anymore, if they do it all. Disk space is so cheap these days that keeping a master sync copy and a separate one for local work is not a big deal. -Matt
Re: cvsup
We could just add rsync targets in our /usr/Makefile in addition to all the cvsup/pkgsrc targets already in there. What, people didn't know we install a Makefile in /usr? Well, now you do! -Matt
Re: cvsup
On 2008-01-19, Simon 'corecode' Schubert <[EMAIL PROTECTED]> wrote: > This is a multi-part message in MIME format. > --020108070307000905080907 > Content-Type: text/plain; charset=UTF-8; format=flowed > Content-Transfer-Encoding: quoted-printable > > Vincent Stemen wrote: >> Also, if nobody has written one or is working on one, I am considering = > writing >> a script to provide basic cvsup like features/functionality for reposit= > ory >> updates via rsync. > > I'm not sure what you mean with that. Isn't calling rsync enough? Maybe= >=20 It wasn't for me. I had research who the rsync mirrors are, look up and determine the appropriate command line switches, post on the mailing list for help because I did not know the repository directory paths on the rsync servers, was not yet familiar with the rsync commands to get that information directly from the server (each one is different) and the instructions were not documented on the DragonFly web site like they are for cvsup, and write a custom shell script loop to run rsync to update each directory (CVSROOT, doc, src, etc). Contrast that with having pre-configured cvsup files that come with the base system and just running "cvsup -f path/to/cvsup-file". The updatecvs script you posted looks like it is heading in the direction that I am talking about. Although I was considering making it for just rsync, since we already have cvsup, and setting up a pre-configured configuration file[s] for each of the available rsync servers. It looks like your script is using config files. I will study it more closely to see how close it is to what I am thinking about doing. Thanks.
Re: cvsup
On Sat, January 19, 2008 3:24 am, Vincent Stemen wrote: > The only actual comparison test results I have found so far, other than > what I posted, are from Justin Sherril, who posted some test > results on the dragonfly.users mailing list back in April of 2007 > indicating cvsup was moderately faster at the time. So far as I can > tell, he only tested the client side as well. I was actually surprised to see cvsup faster at the time. Speed isn't of paramount importance here - one doesn't exclude the other from being used. The 'best' choice is probably the one that is easiest to set up, most widely available, and the most well-documented. That will probably be rsync in the near future. Interestingly, I was just looking at a tool at my workplace that uses rsync for daily backups from a good number of systems (~350 or so). Its biggest impact was in terms of processing power, especially on the server. As Garance said, the effect on the mirroring host is also important.
Re: cvsup
At 8:24 AM + 1/19/08, Vincent Stemen wrote: Also, if nobody has written one or is working on one, I am considering writing a script to provide basic cvsup like features/functionality for repository updates via rsync. You might want to wait a bit. In freebsd-hackers, there's a thread on the progress of adding cvsmode support to 'csup' (the re-write of cvsup in C). It is actively being worked on and tested. Of course, 'csup' probably needs some benchmarks done on it, too. -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED]
Re: cvsup
Vincent Stemen wrote:
Also, if nobody has written one or is working on one, I am considering writing
a script to provide basic cvsup like features/functionality for repository
updates via rsync.
I'm not sure what you mean with that. Isn't calling rsync enough? Maybe
people could be interested in my script to download + update various
repos. See attached. Maybe you can figure out the file formats yourself :)
cheers
simon
--
Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\
Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ /
Party Enjoy Relax | http://dragonflybsd.org Against HTML \
Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
#!/bin/sh -e
args=`getopt c:CIs:uv $*`
if [ $? -ne 0 ]
then
printf "usage: updatecvs -cCIsuv\n" >&2
exit 1
fi
set -- $args
unset VERBOSE SKIPSUP RECVS
PREFIX=`head -n 1 $HOME/.updatecvsrc`
INDEX=-f
CVSDIR=$PREFIX/cvs
SRCDIR=$PREFIX/src
for i
do
case $i in
-c)
CVSDIR=$2; shift
shift
;;
-C)
RECVS=yes
shift
;;
-I)
INDEX=
shift
;;
-s)
SRCDIR=$2; shift
shift
;;
-u)
SKIPSUP=yes
shift
;;
-v)
VERBOSE=yes
shift
;;
--)
shift
break
;;
esac
done
if [ -n "$VERBOSE" ]
then
verbose_cvsup="-L 2"
verbose_cvsync=-v
verbose_cvs=""
verbose_glimpse="/dev/stdout"
verbose_rsync="-v --progress"
else
verbose_cvsup="-L 0"
verbose_cvsync=
verbose_cvs="-Q"
verbose_glimpse="/dev/null"
verbose_rsync=
fi
if [ $# -eq 0 ]
then
repos=`cut -f 1 "$CVSDIR/.cvsrepos"`
set -- $repos
fi
cat "$CVSDIR/.cvsrepos" | while read name mode file
do
case "$name" in
"#"*|"")
continue
;;
esac
dorepo=0
for checkrepo
do
if [ "$checkrepo" = "$name" ]
then
dorepo=1
break
fi
done
if [ $dorepo -eq 0 ]
then
continue
fi
docvs=0
cd "$SRCDIR/$name"
case "$mode" in
none|"")
;;
cvsync)
[ -n "$SKIPSUP" ] || cvsync $verbose_cvsync "$file"
docvs=1
;;
cvsup)
[ -n "$SKIPSUP" ] || cvsup $verbose_cvsup "$file"
docvs=1
;;
rsync)
if [ -z "$SKIPSUP" ]
then
for loc in $file
do
rsync -a --delete -z $verbose_rsync "$loc"
"$CVSDIR/$name/"
done
fi
docvs=1
;;
esac
if [ $docvs -ne 0 ]
then
[ -n "$RECVS" ] && rm -rf "$SRCDIR/$name/"*
cat .cvsinfo | while read module dir tag _rem
do
cvs $verbose_cvs -r -R -d "$CVSDIR/$name" \
co -d $dir -PA -r "$tag" $module || true
done
fi
[ -e .glimpse_exclude ] || echo "CVS/" > .glimpse_exclude
glimpseindex -o -n -B -E -f -M 20 -t ${INDEX} -H "$SRCDIR/$name" `cut
-f 2 .cvsinfo` > $verbose_glimpse
done
Re: cvsup
On 2008-01-18, Bill Hacker <[EMAIL PROTECTED]> wrote: > Vincent Stemen wrote: > > *snip* > >> >> Unless I am overlooking something obvious, > > It is not likely so many projects would be using cvsup for as long as > they have if the rsync advantage was that great, or that simple [1]. > > Have you: > > A) compared the loads and bandwidth as well as the time on BOTH end > machines - host as well as client? No. Only client side. > B) tested for the 'more common' case where cvsup/csup are applied to > rather more sparse pulls of just a fraction of older, larger > repositories (older *BSD's) - and by more users simultaneously? > > Unless I am wrong, cvsup/csup places more of the load of determining > what to pull on the client, less on the source server. > > > I think I am going to stick >> with updating our repository via rsync :-). >> >> > > It may be the right answer for now, and for what you are doing. > > It may be less so for general end-user use - or even your own if/as/when > mirror hosts are under heavier load. > > Most older mirror hosts throttle each connection as well as limit the > maximum number permitted simultaneously. The one you are using presently > seems not to do so. > > The key is to include measurement of host and bandwidth as well as > client. TANSTAAFL. > > > Bill > > [1][ subversion, mercurial, et al alternatives are a different type of > issue. Clearly more testing needs to be done. I only posted the results of my initial tests comparing cvsup to rsync because I keep finding postings and documentation about how cvsup is faster than rsync with cvs repositories, yet my initial tests were so dramatically in contrast. Most of the cvsup vs rsync performance claims I have found seem to be based on architecture and/or protocol but I have yet to find any significant real world test results supporting them. The only actual comparison test results I have found so far, other than what I posted, are from Justin Sherril, who posted some test results on the dragonfly.users mailing list back in April of 2007 indicating cvsup was moderately faster at the time. So far as I can tell, he only tested the client side as well. He also added, Caveats: I didn't test CPU usage. Also, this was with rsync 2.x - there's a new version 3 on the way that is supposed to have improvements. If there is interest, I will try to make time to do more client side comparison tests (for different aged repositories, to different servers, etc) and post the results. I don't have time to setup for doing server side testing, but if my results continue to contradict the cvsup vs rsync claims, then perhaps it will spark interest in others in doing additional comparison testing for both client and server. Also, if nobody has written one or is working on one, I am considering writing a script to provide basic cvsup like features/functionality for repository updates via rsync. Is there interest in me posting it here if I write it? It would be something handy to have available on the DragonFly download site.
Re: cvsup
At 9:16 AM + 1/18/08, Vincent Stemen wrote: I realize that everything I read comparing cvsup to rsync indicates that cvsup is faster with mirroring cvs repositories. So I decided to run my own tests this evening. I thought everybody might be interested in the results. My results are not even close to what others are claiming. Rsync was vastly faster. Granted, so far as I know, this was not right after a large number of files have been tagged, but as you mentioned, that does not happen very often. If anybody wants to email me after that does happen, I will try to make time to re-run the tests. This is a very inadequate benchmark. Certainly rsync works very well, and the dragonfly repository's have enough capacity that they can handle whatever the load is. So, I realize that it is perfectly fine to use rsync if that's what works for you. And I realize that there is the (unfortunate) headache due to needing modula-3 when it comes to CVSUP. So, I'm not saying anyone has to use cvsup, and I am sure that rsync will get the job done. I'm just saying that this specific benchmark is so limited that it is meaningless. What was the load on the server? How well does rsync scale when there are thousands of people updating at the same time? (in particular, how well does the *server* handle that?). How big of an update-interval were you testing with? If I'm reading your message right, the largest interval you tested was 2-days-worth of updates. For most larger open-source projects, many end-users are going at least a week between sync's, and many of my friends update their copy of the freebsd repository once every three weeks. Some update their copy only two or three times a year, or after some significant security update is announced. Note that this means the server sees a huge spike right after security updates, because there are connections from people who haven't sync'ed in months, and who probably would not have sync'ed for a few more months if it wasn't for the security update. Tags occur rarely, but they do occur. And in the case of dragonfly, there are also the sliding tags that Matt likes to use. So while he doesn't create a new tag very often, he does move the tag in a group of files. (Admittedly, I have no clue as to how well cvsup does with a moved tag, but it would be worthwhile to know when it comes to benchmarking rsync-vs-cvsup for dragonfly. It is quite possible that cvsup will actually get confused by a moved-tag, and thus not be able to optimize the transfer of those files) The shorter the update-interval, the less likely that all the CVS-specific optimizing code in cvsup will do any good. Note, for instance: For a 1.5 hour old repository: rsync total time: 34.846 cvsup total time: 3:40.77 = cvsup took 6.33 times as long as rsync For a 2 day old repository: rsync total time: 2:03.07 cvsup total time: 9:14.73 = cvsup took 4.5 times as long as rsync Even with just two data points, we see that larger the window, the less-well that rsync does compared to cvsup. In that 1.5 hour old repository, how many files were changed? 10? 100? If there are only 100 files to do *anything* with, then there isn't much for cvsup to optimize on. It's pretty likely that rsync is going to be faster than cvsup at "sync"ing a file which has zero changes which need to be sync'ed. If you have users who are regularly sync-ing their repository every 1.5 hours, 24 hours a day, 7 days a week, then there are some cvsup servers which would block that user's IP address for being such an annoying pest. The only people who need to sync *that* often are people who themselves are running mirrors of the repository. For all other users, syncing that often is an undesirable and unwanted load on the server. The people running a sync-server wouldn't want to optimize for behavior patterns which they don't want to see in the first place. I would say the *smallest* window that you should bother testing is an six-hour window (which would be four updates per day), and that the most interesting window to test would be a 1-week window. It took more than a year to write cvsup, by someone who was working basically full-time at it. (that's what he told me, at least! :-) He wouldn't have put in all that work if there was no point to it, and he would have based his work on a wide range of usage patterns. Unless I am overlooking something obvious, I think I am going to stick with updating our repository via rsync :-). As I said earlier, rsync is certainly a reasonable solution. I'm just commenting on the "benchmark". And I realize I haven't done *any* benchmarks, so I can't claim much of anything either. But you would need a much more elaborate and tedious set of
Re: cvsup
On 2008-01-18, Simon 'corecode' Schubert <[EMAIL PROTECTED]> wrote:
> Vincent Stemen wrote:
>> *** using cvsup
>>
>> cvsup -L 3 ./DragonFly-cvs-supfile 155.08s user 69.40s system 40% cpu
>> 9:14.73 total
>
> I for sure didn't use cvsup for a long time, but that seems quite long.
> This can happen for various reasons (just guesses):
>
> - CVSup not knowing certain tags ("commitid") and thus falling back to a
> less efficient mechanism
>
> - not using the list files which store file information. Be sure to run
> cvsup -s
Ok. I was not aware of the -s switch. That is a good suggestion.
I will see if I can make the time to re-run the comparison tests this
evening with "cvsup -s".
Although, technically it was probably the most fair performance
comparison to rsync without the "-s" because rsync still works properly
if files have been locally modified.
Re: cvsup
Vincent Stemen wrote: *snip* Unless I am overlooking something obvious, It is not likely so many projects would be using cvsup for as long as they have if the rsync advantage was that great, or that simple [1]. Have you: A) compared the loads and bandwidth as well as the time on BOTH end machines - host as well as client? B) tested for the 'more common' case where cvsup/csup are applied to rather more sparse pulls of just a fraction of older, larger repositories (older *BSD's) - and by more users simultaneously? Unless I am wrong, cvsup/csup places more of the load of determining what to pull on the client, less on the source server. > I think I am going to stick with updating our repository via rsync :-). It may be the right answer for now, and for what you are doing. It may be less so for general end-user use - or even your own if/as/when mirror hosts are under heavier load. Most older mirror hosts throttle each connection as well as limit the maximum number permitted simultaneously. The one you are using presently seems not to do so. The key is to include measurement of host and bandwidth as well as client. TANSTAAFL. Bill [1][ subversion, mercurial, et al alternatives are a different type of issue.
Re: cvsup
Vincent Stemen wrote:
*** using cvsup
cvsup -L 3 ./DragonFly-cvs-supfile 155.08s user 69.40s system 40% cpu 9:14.73
total
I for sure didn't use cvsup for a long time, but that seems quite long.
This can happen for various reasons (just guesses):
- CVSup not knowing certain tags ("commitid") and thus falling back to a
less efficient mechanism
- not using the list files which store file information. Be sure to run
cvsup -s
Unless I am overlooking something obvious, I think I am going to stick
with updating our repository via rsync :-).
I do advise to stick to rsync, yes :)
cheers
simon
--
Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\
Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ /
Party Enjoy Relax | http://dragonflybsd.org Against HTML \
Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: cvsup
On 2008-01-16, Simon 'corecode' Schubert <[EMAIL PROTECTED]> wrote:
>
> You know that you can get a listing with rsync?
>
> sweatshorts % rsync chlamydia.fs.ei.tum.de::
No I did not know that. I have been using rsync for transferring files
with ssh but had not studied the features for talking to an rsync daemon
on the other end.
Thank you very much Simon, Joerg, and Peter for the information. It was
all useful and got me going.
>
> Considering the fact that you only changed one line per file, that's quite
> a lot. Plus it doesn't say how many round trips it was needing.
>
>> As you can see, it took 9 seconds for a full download, but only less than 1.5
>> seconds to update them. That seems reasonably fast to me. Is cvsup really
>> faster than that? I am sceptical that it could be much faster.
>
> Yes, cvsup is way faster in such a case. Maybe not necessarily for 8
> files, but for 800 for sure. It pipelines communication, so that there is
> no need to wait for immediate replies, thus saving network roundtrip times.
>
> cheers
> simon
>
I realize that everything I read comparing cvsup to rsync indicates that
cvsup is faster with mirroring cvs repositories. So I decided to run my
own tests this evening. I thought everybody might be interested in the
results.
My results are not even close to what others are claiming. Rsync was
vastly faster. Granted, so far as I know, this was not right after
a large number of files have been tagged, but as you mentioned, that
does not happen very often. If anybody wants to email me after that
does happen, I will try to make time to re-run the tests.
Peter, I decided to take you up on your posting I found from the end of
2006 in a discussion about the same subject, where you said
From: Peter Avalos
Date: 2006-12-26 05:30:30
I'll also extend that if anyone wants to use theshell.com, go for
it. It is well-connected, and I don't mind if people go to town on it.
I hope the offer still stands :-).
My tests were doing identical updates using both rsync and cvsup, back to
back from theshell.com.
Environment
===
DragonFly 1.10.1-RELEASE system with a 1.11.0-DEVELOPMENT kernel.
cvsup version SNAP_16_1h
rsync version 2.6.9
The updates included the CVSROOT, doc, and src directories.
"rsup" is the rsync script I used. It contains:
cd /home/dcvs || exit 1
for d in CVSROOT doc src; do
# rsync -avHz --delete rsync.theshell.com::DragonFly/dcvs/$d .
rsync -avH --delete rsync.theshell.com::DragonFly/dcvs/$d .
done
I did one test with compression (-z) and one without.
For the cvsup tests, I used the standard DragonFly-cvs-supfile that that
comes with DragonFly with the following three file collections
uncommented.
dragonfly-cvs-root
dragonfly-cvs-src
dragonfly-cvs-doc
Here are the results
Updating about a 2 day old repository
With compression
cvsup verbose level 3 (-L 3)
Thu Jan 17 18:50:11 CST 2008
*** using rsync
rsup 8.29s user 13.31s system 17% cpu 2:03.07 total
Thu Jan 17 19:13:10 CST 2008
*** using cvsup
cvsup -L 3 ./DragonFly-cvs-supfile 155.08s user 69.40s system 40% cpu 9:14.73
total
rsync total time: 2:03.07
cvsup total time: 9:14.73
=
cvsup took 4.5 times as long as rsync
=
Updating about a 1.5 hour old repository
Without compression (Probably made no difference since there were
apparently no updates)
cvsup default verbose level 1
Thu Jan 17 20:17:12 CST 2008
*** using cvsup
cvsup ./DragonFly-cvs-supfile 54.17s user 26.03s system 36% cpu 3:40.77 total
Thu Jan 17 20:29:40 CST 2008
*** using rsync
rsup 1.34s user 3.41s system 13% cpu 34.846 total
rsync total time: 34.846
cvsup total time: 3:40.77
=
cvsup took 6.33 times as long as rsync
=
I am seeing rsync perform from 4 to over 6 times faster than cvsup.
Not only that, from the output of "time" rsync took substantially less
user time, system time, and cpu percentage.
As long as the cvsup file collections are the same as the three
directories I rsync'd, then this should be a one on one fair comparison.
Unless I am overlooking something obvious, I think I am going to stick
with updating our repository via rsync :-).
Re: cvsup
On Wed, Jan 16, 2008 at 02:35:48AM +, Vincent Stemen wrote: > I am not clear how to access it. Try something like the attached script. Joerg update.sh Description: Bourne shell script
Re: cvsup
Vincent Stemen wrote: On 2008-01-15, Simon 'corecode' Schubert <[EMAIL PROTECTED]> wrote: Yes. There are a couple of mirrors which serve the cvs repo via rsync. For instance use chlamydia.fs.ei.tum.de. There are more, I guess. Thanks. I just checked the list, but most of the sites say they only mirror _Daily snapshots and official ISOs_. The rsync link to the one you mentioned, *chlamydia.fs.ei.tum.de*, does take you to a web page that says they have the cvsup collections, but they do not provide the full path to access them via rsync. I guess you're expected to find out with rsync yourself :) You know that you can get a listing with rsync? sweatshorts % rsync chlamydia.fs.ei.tum.de:: dflysnapDragonFlyBSD daily snapshots dragonfly-cvs DragonFly CVS repository freebsd-cvs FreeBSD CVS repository netbsd-cvs NetBSD CVS repository openbsd-cvs OpenBSD CVS repository sweatshorts % rsync chlamydia.fs.ei.tum.de::dragonfly-cvs drwxr-xr-x 512 2007/05/07 23:38:22 . -rwxrwxr-x 321 2003/07/15 06:17:23 fixme drwxrwxr-x1536 2008/01/16 12:33:43 CVSROOT drwxrwxr-x 512 2006/06/28 22:24:29 dfports drwxrwxr-x 512 2007/04/06 22:03:43 doc drwxrwxr-x 512 2007/12/21 06:39:21 site drwxrwxr-x 512 2004/06/18 13:23:48 src.sys.alpha.old drwxrwxr-x1024 2008/01/14 13:39:27 src Rsync is very slow when you tag a repo, because all the files change, but only one line. It's not really optimized for this task. But this doesn't happen very often, though. That is interesting. The rsync docs claim to be very fast and say this The rsync remote-update protocol allows rsync to transfer just the dif- ferences between two sets of files across the network connection, using an efficient checksum-search algorithm ... Well, of course in general it is fast. But CVS repos are a special case, consisting of hundreds of thousands of small RCS text files, which only get one line added when the repo is tagged. A general solution is bound to perform worse than a special case solution. I ran a test transfering 8 text files over the internet that are each about 450K and the performance seems pretty good when I changed just one line in each file. In my test I told rsync to use ssh, so the transfer was also encrypted. CVS repos are different: Imagine 10 files with an average size of 4-8KB. I think rsync has to do 10 network roundtrips to negotiate on the changesets to be sent. Total bytes sent: 31162 Total bytes received: 26842 Considering the fact that you only changed one line per file, that's quite a lot. Plus it doesn't say how many round trips it was needing. As you can see, it took 9 seconds for a full download, but only less than 1.5 seconds to update them. That seems reasonably fast to me. Is cvsup really faster than that? I am sceptical that it could be much faster. Yes, cvsup is way faster in such a case. Maybe not necessarily for 8 files, but for 800 for sure. It pipelines communication, so that there is no need to wait for immediate replies, thus saving network roundtrip times. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: cvsup
On Wed, Jan 16, 2008 at 03:22:26AM +, Vincent Stemen wrote: > > _TheShell.com_ is the only other one that lists *Code* under _Mirrored > Data_. Their rsync link takes you to > rsync://rsync.theshell.com/pub/DragonFly but there are no instructions > on the full path to access the repository. I can only guess > rsync://rsync.theshell.com/pub/DragonFly/dragonfly-cvs-src > I will experiment with it. > The cvs repository is dcvs/. The directory structure is the same for http and ftp. If you want the entire DragonFly CVS repo: rsync://rsync.theshell.com/pub/DragonFly/dcvs --Peter pgpRU2nIfdgfP.pgp Description: PGP signature
Re: cvsup
On 2008-01-15, Simon 'corecode' Schubert <[EMAIL PROTECTED]> wrote: > > Yes. There are a couple of mirrors which serve the cvs repo via rsync. > For instance use chlamydia.fs.ei.tum.de. There are more, I guess. Thanks. I just checked the list, but most of the sites say they only mirror _Daily snapshots and official ISOs_. The rsync link to the one you mentioned, *chlamydia.fs.ei.tum.de*, does take you to a web page that says they have the cvsup collections, but they do not provide the full path to access them via rsync. _TheShell.com_ is the only other one that lists *Code* under _Mirrored Data_. Their rsync link takes you to rsync://rsync.theshell.com/pub/DragonFly but there are no instructions on the full path to access the repository. I can only guess rsync://rsync.theshell.com/pub/DragonFly/dragonfly-cvs-src I will experiment with it. > >> Also, has anybody considered writing some front end scripts to rsync to >> emulate >> the necessary features of cvsup? I have found rsync to be very fast since it >> efficiently only transfers what has changed. I also do not see any reason >> for >> the mirroring tool to need any knowledge of a specific repository type (e.g >> cvs, svn, etc). You are just mirroring files. > > Rsync is very slow when you tag a repo, because all the files change, but > only one line. It's not really optimized for this task. But this doesn't > happen very often, though. That is interesting. The rsync docs claim to be very fast and say this The rsync remote-update protocol allows rsync to transfer just the dif- ferences between two sets of files across the network connection, using an efficient checksum-search algorithm ... I ran a test transfering 8 text files over the internet that are each about 450K and the performance seems pretty good when I changed just one line in each file. In my test I told rsync to use ssh, so the transfer was also encrypted. Here are the results. *Initial download* $ time rsync --stats --del --rsh="ssh $wh_ssh_args" -auv $whr:/var/tmp/test . receiving file list ... done test/ test/testfile1 test/testfile2 test/testfile3 test/testfile4 test/testfile5 test/testfile6 test/testfile7 test/testfile8 Number of files: 9 Number of files transferred: 8 Total file size: 3607368 bytes Total transferred file size: 3607368 bytes Literal data: 3607368 bytes Matched data: 0 bytes File list size: 168 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 202 Total bytes received: 3608346 sent 202 bytes received 3608346 bytes 379847.16 bytes/sec total size is 3607368 speedup is 1.00 rsync --stats --del --rsh="ssh $wh_ssh_args" -auv $whr:/var/tmp/test . 0.19s user 0.04s system 2% cpu 9.059 total *After modifying each file* $ time rsync --stats --del --rsh="ssh $wh_ssh_args" -auv $whr:/var/tmp/test . receiving file list ... done test/ test/testfile1 test/testfile2 test/testfile3 test/testfile4 test/testfile5 test/testfile6 test/testfile7 test/testfile8 Number of files: 9 Number of files transferred: 8 Total file size: 3607376 bytes Total transferred file size: 3607376 bytes Literal data: 5608 bytes Matched data: 3601768 bytes File list size: 168 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 31162 Total bytes received: 26842 sent 31162 bytes received 26842 bytes 23201.60 bytes/sec total size is 3607376 speedup is 62.19 rsync --stats --del --rsh="ssh $wh_ssh_args" -auv $whr:/var/tmp/test . 0.08s user 0.02s system 6% cpu 1.469 total As you can see, it took 9 seconds for a full download, but only less than 1.5 seconds to update them. That seems reasonably fast to me. Is cvsup really faster than that? I am sceptical that it could be much faster.
Re: cvsup
On 2008-01-15, Joerg Sonnenberger <[EMAIL PROTECTED]> wrote: > On Tue, Jan 15, 2008 at 10:23:27PM +, Vincent Stemen wrote: >> Is the repository available via rsync and, if not, I am curious as to >> why not? Rsync is robust, popular, portable, and written in C. > > Yes. Check the mirror list, e.g. allbsd has it. > > Joerg I am not clear how to access it. It only says that Daily snapshots and official ISOs are available from ALLBSD.org and *rsync* under _Access methods_ on the Dragonfly site is not a link. I will look into some of the sites. Thanks.
Re: cvsup
Vincent Stemen wrote: I was dismayed to discover that the BSD community (not just DragonFly) has created such as dependency on a tool written in an obscure problematic language such as Modula-3 that are not even easily ported to platforms such as DragonFly, causing us to have to run a binary from different platform after all this time. Yes, not only you are upset about this. I found csup, a cvsup replacement written in C, only to discover it currently only supports what is called CVS mode, which means I cannot use it to download/update the repository. Plus there is do csupd, so for running a cvsupd, you're still stuck with m3. There is cvsync, however. It only supports repo syncing (no checkout). But it is using a broken algorithm, which leads to RCS files being rearranged in a way that CVS can't use them (under certain conditions). The authors are aware of this (I told them), but didn't want to fix it. Is the repository available via rsync and, if not, I am curious as to why not? Rsync is robust, popular, portable, and written in C. Yes. There are a couple of mirrors which serve the cvs repo via rsync. For instance use chlamydia.fs.ei.tum.de. There are more, I guess. Also, has anybody considered writing some front end scripts to rsync to emulate the necessary features of cvsup? I have found rsync to be very fast since it efficiently only transfers what has changed. I also do not see any reason for the mirroring tool to need any knowledge of a specific repository type (e.g cvs, svn, etc). You are just mirroring files. Rsync is very slow when you tag a repo, because all the files change, but only one line. It's not really optimized for this task. But this doesn't happen very often, though. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: cvsup
On Tue, Jan 15, 2008 at 10:23:27PM +, Vincent Stemen wrote: > Is the repository available via rsync and, if not, I am curious as to > why not? Rsync is robust, popular, portable, and written in C. Yes. Check the mirror list, e.g. allbsd has it. Joerg
RE: comparing cvsup vs. rsync
It would be huge! I actually meant to key each file, and version of each file with an integer, and store it out on disk, and store the metadata(checksums) in the DB for speed, but a lack of coffee prevented the conveyance of the full idea. ;-) ...Going away now to read up on git... Nigel Weeks Tech Support and Systems Developer Rural Press Tasmania The Examiner Newspaper Ph. 03 6336 7234 Mob. 0408 133 738 Email. [EMAIL PROTECTED] > -Original Message- > From: [EMAIL PROTECTED] [mailto:users- > [EMAIL PROTECTED] On Behalf Of Simon 'corecode' Schubert > Sent: Thursday, 12 April 2007 9:12 AM > To: users@crater.dragonflybsd.org > Subject: Re: comparing cvsup vs. rsync > > Nigel Weeks wrote: > > Just having an idea about this...are there any files in the source tree > that exceed 32kbytes? > > > > What if a database solution were created to: > > Contain every version of every file of every branch in a nicely indexed > database table > > The md5/sha256 of each entry mentioned above > > 512 byte chunks of each file mentioned above in a nicely indexed table > > The md5/sha256 of every 512 bytes of the above mentioned file. > > uh-ah. i guess that's for checkout, but still it seems very wrong. I > mean, really. Uh, I can't stop shuddering. I just hope you are not > serious :) > > and the database will be really huge... > > actually, it sounds like git, just worse. > > > File checkins could simply be a file upload, or a mime encoded fetch > request, or an email message, or an ftp drop, or an scp copy, or an rsync > push... > > file checkins right now are cvs checkins, and I guess that will stay for > some time. > > cheers > simon > > -- > Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ > Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / > Party Enjoy Relax | http://dragonflybsd.org Against HTML \ > Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: comparing cvsup vs. rsync
Nigel Weeks wrote: Just having an idea about this...are there any files in the source tree that exceed 32kbytes? What if a database solution were created to: Contain every version of every file of every branch in a nicely indexed database table The md5/sha256 of each entry mentioned above 512 byte chunks of each file mentioned above in a nicely indexed table The md5/sha256 of every 512 bytes of the above mentioned file. uh-ah. i guess that's for checkout, but still it seems very wrong. I mean, really. Uh, I can't stop shuddering. I just hope you are not serious :) and the database will be really huge... actually, it sounds like git, just worse. File checkins could simply be a file upload, or a mime encoded fetch request, or an email message, or an ftp drop, or an scp copy, or an rsync push... file checkins right now are cvs checkins, and I guess that will stay for some time. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
RE: comparing cvsup vs. rsync
Just having an idea about this...are there any files in the source tree that exceed 32kbytes? What if a database solution were created to: Contain every version of every file of every branch in a nicely indexed database table The md5/sha256 of each entry mentioned above 512 byte chunks of each file mentioned above in a nicely indexed table The md5/sha256 of every 512 bytes of the above mentioned file. Then, a small client could be written to: Check for the existence of a later version of a file. Calculate the checksum of the local file, and fetch the checksum of the file in the repository database. If the checksums differ, then calculate the checksums for each 512 bytes of the local file, and fetch the differing sections, and cat then back together. You could then do a full sync with the programs find, awk, fetch, and cat. I build file repositories with version control, so the server side's easy. File checkins could simply be a file upload, or a mime encoded fetch request, or an email message, or an ftp drop, or an scp copy, or an rsync push... I have a FreeBSD machine that could be used for prototyping... Nigel Weeks Tech Support and Systems Developer Rural Press Tasmania The Examiner Newspaper Ph. 03 6336 7234 Mob. 0408 133 738 Email. [EMAIL PROTECTED] > -Original Message- > From: [EMAIL PROTECTED] [mailto:users- > [EMAIL PROTECTED] On Behalf Of Simon 'corecode' Schubert > Sent: Wednesday, 11 April 2007 8:29 PM > To: users@crater.dragonflybsd.org > Subject: Re: comparing cvsup vs. rsync > > walt wrote: > > Linus avoided rsync in favor of http in 'git' because he thinks > > rsync is inherently unreliable. I have *no* idea if he is right or > > wrong in his opinions, but I figure you guys will favor me with your > > own opinions on the subject. > > Possibly for transferring the git objects. They never change, so rsync is > not efficient. RCS files do change, so just transferring deltas saves a > lot. Additionally, the http transport in git is quite dumb and needs a > pre-created file to help the download. > > cheers > simon > > -- > Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ > Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / > Party Enjoy Relax | http://dragonflybsd.org Against HTML \ > Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: comparing cvsup vs. rsync
walt wrote: Linus avoided rsync in favor of http in 'git' because he thinks rsync is inherently unreliable. I have *no* idea if he is right or wrong in his opinions, but I figure you guys will favor me with your own opinions on the subject. Possibly for transferring the git objects. They never change, so rsync is not efficient. RCS files do change, so just transferring deltas saves a lot. Additionally, the http transport in git is quite dumb and needs a pre-created file to help the download. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
On Wednesday 11 April 2007 05:14:43 walt wrote: > Yes, I'm way off topic here, and I apologize -- but I've seen your > posts in the 'git' mail-list and you've experimented with Hg, so I > know you have your own opinions on version control systems. > > Linus avoided rsync in favor of http in 'git' because he thinks > rsync is inherently unreliable. I have *no* idea if he is right or > wrong in his opinions, but I figure you guys will favor me with your > own opinions on the subject. Also, http is easier to push through corporate firewalls :) Cheers, Emiel
Re: comparing cvsup vs. rsync
:Matt, : :something is weird with the permissions: : :%rsync crater.dragonflybsd.org::dragonfly_cvs/src/crypto/heimdal/Attic/ :drwxrwxr-x1024 2005/03/28 05:35:43 . :-r--rw-r-- 20313 2005/03/28 05:35:43 ChangeLog,v :[..] :-r-xrwxr-x3242 2005/03/28 05:35:43 compile,v : :why are they u-w but g+w? my cvsup gets a little bit confused with that = :and resets permissions back, so there is always a back-and-forth between = :rsync and cvsup. setting "preserved" and not "umask=3D002" hopefully fix= :es that locally. : :cheers : simon This is a bug in cvs's SGID handling on commits. Access is by group and it doesn't pay much attention to the user, so sometimes the user gets out of whack. Many, many cvs files will not have user-write perms but will have group-write perms because of this. In fact, the user will be whoever last committed the file, which is probably not what we want rsync to deliver. -Matt
Re: comparing cvsup vs. rsync
Simon 'corecode' Schubert wrote: > Matthew Dillon wrote: >> My only worry is figuring out how to run the rsync daemon safely. >> I'm a bit paranoid about running things on crater but I do agree >> that we would have to run the master rsync daemon there. > > You can run rsyncd from inetd or standalone. If you're really[tm] > paranoid, a jail with a static rsync binary might be the best. Yes, I'm way off topic here, and I apologize -- but I've seen your posts in the 'git' mail-list and you've experimented with Hg, so I know you have your own opinions on version control systems. Linus avoided rsync in favor of http in 'git' because he thinks rsync is inherently unreliable. I have *no* idea if he is right or wrong in his opinions, but I figure you guys will favor me with your own opinions on the subject. Thanks!
Re: comparing cvsup vs. rsync
Simon 'corecode' Schubert wrote: I am now running an rsync server on crater.dragonflybsd.org, serving the cvs repository as 'dragonfly_cvs'. rsync -a rsync://crater.dragonflybsd.org/dragonfly_cvs blahblah very nice! will immediatelly switch to rsync operation. however, i'll have to find out how to generate the supfile for cvsupd serving... okay, that doesn't work. cvsup changes the RCS files, so I can't simply cvsup over them to generate a scan file :/ operation without scan file then. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
Matthew Dillon wrote: I am now running an rsync server on crater.dragonflybsd.org, serving the cvs repository as 'dragonfly_cvs'. rsync -a rsync://crater.dragonflybsd.org/dragonfly_cvs blahblah Matt, something is weird with the permissions: %rsync crater.dragonflybsd.org::dragonfly_cvs/src/crypto/heimdal/Attic/ drwxrwxr-x1024 2005/03/28 05:35:43 . -r--rw-r-- 20313 2005/03/28 05:35:43 ChangeLog,v [..] -r-xrwxr-x3242 2005/03/28 05:35:43 compile,v why are they u-w but g+w? my cvsup gets a little bit confused with that and resets permissions back, so there is always a back-and-forth between rsync and cvsup. setting "preserved" and not "umask=002" hopefully fixes that locally. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
Matthew Dillon wrote: I am now running an rsync server on crater.dragonflybsd.org, serving the cvs repository as 'dragonfly_cvs'. rsync -a rsync://crater.dragonflybsd.org/dragonfly_cvs blahblah very nice! will immediatelly switch to rsync operation. however, i'll have to find out how to generate the supfile for cvsupd serving... Minor issue, but it would be nice if I could get statistics on downloads. I have this in my rsyncd.conf: syslog facility = local5 transfer logging = yes and in syslog.conf local5.*/var/log/rsyncd.log (obviously could be made better) cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
I am now running an rsync server on crater.dragonflybsd.org, serving the cvs repository as 'dragonfly_cvs'. rsync -a rsync://crater.dragonflybsd.org/dragonfly_cvs blahblah -- Unfortunately I don't know how to get rsyncd to just log statistics on completion, and it insists on logging a warning '/etc/pwd.db: No such file or directory' every time due to the chroot option in the config file, which I can't seem to get rid of. Minor issue, but it would be nice if I could get statistics on downloads. -Matt
Re: comparing cvsup vs. rsync
Peter Hessler wrote: On 2007 Apr 10 (Tue) at 18:03:49 +0200 (+0200), Simon 'corecode' Schubert wrote: :cvsync does cvs-only (so, :like rsync), but it has a bug which breaks the RCS files in some cases. :The author didn't want to fix it though :/ Can you describe the bug? I've been using cvsync for local copies of the openbsd source tree and haven't seen any RCS corruption. It missorders revisions on branches. CVS then fails to perform a annotate on those revisions (try something like 1.X.Y.2 or so). cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
On Tue, April 10, 2007 11:32 am, Matthew Dillon wrote: > We could just distribute the CVS tree and write a front-end utility > in csh or sh that we distribute along with the rest of the system > to do the nitty gritty work of actually checking something out into > /usr/src. In fact, I think that would be preferable. I am all for simple upgrades - the closer we can get it to a single button-push for obvious upgrades like 1.8->1.8.1, the happier I'd be. > My only worry is figuring out how to run the rsync daemon safely. > I'm a bit paranoid about running things on crater but I do agree > that we would have to run the master rsync daemon there. Is it feasible to run it in a vkernel? External bandwidth, rather than CPU or disk, is probably the choke point, so I would think there wouldn't be much of a tradeoff of performance vs. security.
Re: comparing cvsup vs. rsync
On 2007 Apr 10 (Tue) at 18:03:49 +0200 (+0200), Simon 'corecode' Schubert wrote: :cvsync does cvs-only (so, :like rsync), but it has a bug which breaks the RCS files in some cases. :The author didn't want to fix it though :/ Can you describe the bug? I've been using cvsync for local copies of the openbsd source tree and haven't seen any RCS corruption. -- The only real way to look younger is not to be born so soon. -- Charles Schulz, "Things I've Had to Learn Over and Over and Over"
Re: comparing cvsup vs. rsync
Dmitri Nikulin wrote: We could just distribute the CVS tree and write a front-end utility in csh or sh that we distribute along with the rest of the system to do the nitty gritty work of actually checking something out into /usr/src. In fact, I think that would be preferable. NetBSD is distributed on pure CVS, anything else is a convenience. However, because CVS is so impressively bad for initial checkouts, they recommend downloading the rather small source and pkgsrc tarballs, and only using CVS for actual updates. This could work for DragonFly too, with the added bonus of pre-distributed cvs dotfiles or shell aliases. Maybe 'make update' (or whatever it is) could be wired to work that way too. CVS is a beat concerning performance and CPU load, both for server and client. I'd say rsync is the best deal for now. cvsync does cvs-only (so, like rsync), but it has a bug which breaks the RCS files in some cases. The author didn't want to fix it though :/ cvsync with checkout mode and the bug fixed would be definitely better than rsync. Maybe we could do that on a weekend sprint, it doesn't seem too complicated to me. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
Matthew Dillon wrote: My only worry is figuring out how to run the rsync daemon safely. I'm a bit paranoid about running things on crater but I do agree that we would have to run the master rsync daemon there. You can run rsyncd from inetd or standalone. If you're really[tm] paranoid, a jail with a static rsync binary might be the best. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ signature.asc Description: OpenPGP digital signature
Re: comparing cvsup vs. rsync
On 4/11/07, Matthew Dillon <[EMAIL PROTECTED]> wrote: We could just distribute the CVS tree and write a front-end utility in csh or sh that we distribute along with the rest of the system to do the nitty gritty work of actually checking something out into /usr/src. In fact, I think that would be preferable. NetBSD is distributed on pure CVS, anything else is a convenience. However, because CVS is so impressively bad for initial checkouts, they recommend downloading the rather small source and pkgsrc tarballs, and only using CVS for actual updates. This could work for DragonFly too, with the added bonus of pre-distributed cvs dotfiles or shell aliases. Maybe 'make update' (or whatever it is) could be wired to work that way too. This would of course be unecessary if the system could be binary packaged and updated like Debian is. If that can be harmonised with the "base system" nature of BSD and the flexibility of source compilation, so much the better. I suppose that discussion has been up in the air for ~ever and nobody's actually done it well. My only worry is figuring out how to run the rsync daemon safely. I'm a bit paranoid about running things on crater but I do agree that we would have to run the master rsync daemon there. From what I've read around, it seems GNU CVS is a real threat as it is. OpenBSD guys have made some effort in replacing it, but I don't know much about that either. Besides, as Joerg said, you can limit the privileges of rsync. --- Dmitri Nikulin Centre for Synchrotron Science Monash University Victoria 3800, Australia
Re: comparing cvsup vs. rsync
On Tue, Apr 10, 2007 at 08:32:55AM -0700, Matthew Dillon wrote: > My only worry is figuring out how to run the rsync daemon safely. > I'm a bit paranoid about running things on crater but I do agree > that we would have to run the master rsync daemon there. rsync allows running chrooted and under unprivileged uid. You specify the root of the subtrees in the config files, e.g. cvsroot and a cvs checkout updated via crontab. Joerg
Re: comparing cvsup vs. rsync
:I timed repeated retrievals of src from theshell.com over the past few :weeks, and here's the result. : :Retrieving all of src: :cvsup averaged about 11.5 minutes :rsync averaged about 19 minutes : :Retrieving only the last 24 hours of changes: :cvsup averaged about 18 seconds :rsync averaged about 25 seconds : :Caveats: I didn't test CPU usage. Also, this was with rsync 2.x - there's :a new version 3 on the way that is supposed to have improvments. : :So, it looks like rsync runs somewhat slower than cvsup, but not :catastrophically so. Also, rsync can't do checkouts of particular :revisions, so we'd have to have a certain checked out version of the :source to allow people to retrieve given releases of DragonFly. No big :surprises here, and no clear indicator we should change. At this point in time I think I would actually like to switch to a more mainstream distribution system, and rsync seems to be that system. I'm getting a bit tired of going through loops to support cvsup. We could just distribute the CVS tree and write a front-end utility in csh or sh that we distribute along with the rest of the system to do the nitty gritty work of actually checking something out into /usr/src. In fact, I think that would be preferable. My only worry is figuring out how to run the rsync daemon safely. I'm a bit paranoid about running things on crater but I do agree that we would have to run the master rsync daemon there. -Matt Matthew Dillon <[EMAIL PROTECTED]>
comparing cvsup vs. rsync
Since cvsup is somewhat of a hassle to build, discussion comes up from time to time about switching to rsync. Rsync is generally accepted as slower/more resource intensive, but how much hasn't been quantified. I wanted to look into this in as un-bikesheddy a way as possible... I timed repeated retrievals of src from theshell.com over the past few weeks, and here's the result. Retrieving all of src: cvsup averaged about 11.5 minutes rsync averaged about 19 minutes Retrieving only the last 24 hours of changes: cvsup averaged about 18 seconds rsync averaged about 25 seconds Caveats: I didn't test CPU usage. Also, this was with rsync 2.x - there's a new version 3 on the way that is supposed to have improvments. So, it looks like rsync runs somewhat slower than cvsup, but not catastrophically so. Also, rsync can't do checkouts of particular revisions, so we'd have to have a certain checked out version of the source to allow people to retrieve given releases of DragonFly. No big surprises here, and no clear indicator we should change.
Re: obtaining kernel src with cvsup
On Thu, 8 Feb 2007 12:57:44 -0600 "Mire, John" <[EMAIL PROTECTED]> wrote: > > Thanks, that bit of info seemed to escape me. > Damn, I've been using FBSD so long, I expect everything to be like it :( > By the by, > http://leaf.dragonflybsd.org/~justin/handbook/kernelconfig-building.html > was my reference from both the homepage and the wiki. > http://wiki.dragonflybsd.org/index.cgi/kernelconfig.html was driving me > 'round the bend. > And http://wiki.dragonflybsd.org/index.cgi/CategoryHowTo was informative > on so many nonrelated levels I shall post a HowTo for the sick, lame > and lazy former freebsd administrator. Ah, this would be great, SwitchingFromFreeBSD... > I haven't had this much fun > since I first installed FBSD 2.0.5 and thought it was the greatest thing > since sliced bread. > I hate sliced bread. :-P -- Gergo Szakal <[EMAIL PROTECTED]> University Of Szeged, HU Faculty Of General Medicine /* Please do not CC me with replies, thank you. */
Re: obtaining kernel src with cvsup
/usr/src/sys/config -- Gergo Szakal <[EMAIL PROTECTED]> University Of Szeged, HU Faculty Of General Medicine /* Please do not CC me with replies, thank you. */
RE: obtaining kernel src with cvsup
>On Wed, Feb 07, 2007 at 05:35:35PM -0600, John Mire wrote: >> how do I get the kernel src through cvsup? >> > >... > >> =used the following command to cvsup the most recent release src: >> >> cvsup -g -L 2 -h fred.acm.cs.rpi.edu >> /usr/share/examples/cvsup/DragonFly-release1_8-supfile >> > >Yes. > >> the cvsup completed without errors but no /usr/src/sys/i386/conf >> directory so building a kernel is out of the question. >> > >Try /usr/src/sys/config/. > >--Peter Thanks, that bit of info seemed to escape me. Damn, I've been using FBSD so long, I expect everything to be like it :( By the by, http://leaf.dragonflybsd.org/~justin/handbook/kernelconfig-building.html was my reference from both the homepage and the wiki. http://wiki.dragonflybsd.org/index.cgi/kernelconfig.html was driving me 'round the bend. And http://wiki.dragonflybsd.org/index.cgi/CategoryHowTo was informative on so many nonrelated levels I shall post a HowTo for the sick, lame and lazy former freebsd administrator. I haven't had this much fun since I first installed FBSD 2.0.5 and thought it was the greatest thing since sliced bread.
RE: obtaining kernel src with cvsup
Thanks, guess you didn't read the diary entry for Tue Feb 6 08:19:03 CST 2007 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of walt Sent: Wednesday, February 07, 2007 6:25 PM To: users@crater.dragonflybsd.org Subject: Re: obtaining kernel src with cvsup John Mire wrote: > how do I get the kernel src through cvsup? > > > > here's my diaryfile: > > Thu Feb 1 02:50:54 CST 2007 > > =inital install from iso-image > > =setup the pkgsrc tree as per the Handbook: > > http://leaf.dragonflybsd.org/~justin/handbook/pkgsrc-sourcetree-using.ht ml Welcome! The pkgsrc project is separate from DragonFly. You need the sources for DragonFly: http://www.dragonflybsd.org/community/download.shtml It would be nice if you could use one of the mirror sites listed on the same webpage to preserve Matt's bandwidth.
Re: obtaining kernel src with cvsup
John Mire wrote: > how do I get the kernel src through cvsup? > > > > here's my diaryfile: > > Thu Feb 1 02:50:54 CST 2007 > > =inital install from iso-image > > =setup the pkgsrc tree as per the Handbook: > > http://leaf.dragonflybsd.org/~justin/handbook/pkgsrc-sourcetree-using.html Welcome! The pkgsrc project is separate from DragonFly. You need the sources for DragonFly: http://www.dragonflybsd.org/community/download.shtml It would be nice if you could use one of the mirror sites listed on the same webpage to preserve Matt's bandwidth.
Re: obtaining kernel src with cvsup
On Wed, Feb 07, 2007 at 05:35:35PM -0600, John Mire wrote: > how do I get the kernel src through cvsup? > ... > =used the following command to cvsup the most recent release src: > > cvsup -g -L 2 -h fred.acm.cs.rpi.edu > /usr/share/examples/cvsup/DragonFly-release1_8-supfile > Yes. > the cvsup completed without errors but no /usr/src/sys/i386/conf directory > so building a kernel is out of the question. > Try /usr/src/sys/config/. --Peter pgpOIbdZ1Urc4.pgp Description: PGP signature
obtaining kernel src with cvsup
how do I get the kernel src through cvsup? here's my diaryfile: Thu Feb 1 02:50:54 CST 2007 =inital install from iso-image =setup the pkgsrc tree as per the Handbook: http://leaf.dragonflybsd.org/~justin/handbook/pkgsrc-sourcetree-using.html # cd /usr/ # cvs -d [EMAIL PROTECTED]:/cvsroot co pkgsrc seemed to hang after copying 2 directories, the following got further and finished: # env CVS_RSH=ssh cvs -d [EMAIL PROTECTED]:/cvsroot checkout pkgsrc ++jmire Mon Feb 5 05:54:05 CST 2007 =setup sshd as per standard guidelines: PermitRootLogin without-password =installed perl5.8.8 using pkgsrc ++jmire Tue Feb 6 08:19:03 CST 2007 =used the following command to cvsup the most recent release src: cvsup -g -L 2 -h fred.acm.cs.rpi.edu /usr/share/examples/cvsup/DragonFly-release1_8-supfile ++jmire Wed Feb 7 04:28:48 CST 2007 the cvsup completed without errors but no /usr/src/sys/i386/conf directory so building a kernel is out of the question. dragon# ls -aCF /usr/src/sys ./ compile/ ddb/ net/ platform/ ../ conf/ dev/ netgraph/ sys/ Makefile config/ emulation/ netinet/ tools/ Makefile.modules contrib/ kern/ netinet6/ vfs/ boot/ cpu/ libiconv/ netproto/ vm/ bus/ crypto/ libkern/ opencrypto/ ++jmire
Re: about cvsup and supfile...
On 2006-11-17 18:32, Saverio Iacovelli wrote: I want update DFly with cvsup, but I want to optimize the upgrade operation. I don't want to upgrade all system, but only kernel and some application. 1 - Is it possible to set cvsup for thos operation? 2 - How can I find howto about supfile? I believe that there are a sup-file for only the kernel sources, however if you want to upgrade an application you'll have to upgrade the whole world too. Depending on how old your kernel sources are you might need a new world to be able to use the new kernel too, I'm not sure about that. However, if you already have quite recent sources on your computer than using cvsup will be quite efficient even if you download both world and kernel, since it only download that which has changed. As for learning about cvsup, there really is not much to it. Take a look in /usr/share/examples/cvsup, there you'll find a couple of different sup-files ready for use that will download different things (you can open them with an editor if you want). Then, when you have found one that suits your needs, you run #cvsup -h /path/to/sup-file To find a host near you take a look at www.dragonflybsd.org in the download section. If you don't have any recent sources you might want to download a snapshot of the sources (find a mirror in the download section) and extract that on your computer and then run cvsup to update them to the very latest. More information can be found in the cvsup manual page (man cvsup): http://leaf.dragonflybsd.org/cgi/web-man?command=cvsup§ion=ANY You might also be interested in the Updating section in the Handbook: http://leaf.dragonflybsd.org/~justin/handbook/updating.html -- Erik Wikström
about cvsup and supfile...
I want update DFly with cvsup, but I want to optimize the upgrade operation. I don't want to upgrade all system, but only kernel and some application. 1 - Is it possible to set cvsup for thos operation? 2 - How can I find howto about supfile? Regards, Saverio __ Do You Yahoo!? Poco spazio e tanto spam? Yahoo! Mail ti protegge dallo spam e ti da tanto spazio gratuito per i tuoi file e i messaggi http://mail.yahoo.it
Re: Spurious results from using cvsup?
:I discovered that using cvsup on DF-current will give some :stragely wrong results while updating my pkgsrc tree, but :no error messages of any kind. : :One notable example: : : Checkout pkgsrc/misc/gnome-user-docs/DESCR : Checkout pkgsrc/misc/gnome-user-docs/Makefile : Checkout pkgsrc/misc/gnome-user-docs/PLIST : Checkout pkgsrc/misc/gnome-user-docs/distinfo : :... : :I'm wondering how many other errors may be slipping :past without any error messages to alert me. : :I'm using cvsup from cvsup-bootstrap-20051229.tgz which :(IIRC) I got from Corecode's package repository. : :Can anyone else confirm this result? My guess is that they performed some CVS surgery to rename gnom2-user-docs to gnome-user-docs and it wound up confusing cvsup. I normally do not cvsup source checkouts. Instead I cvsup the CVS repository itself and then do the checkout manually. -Matt Matthew Dillon <[EMAIL PROTECTED]>
Spurious results from using cvsup?
I discovered that using cvsup on DF-current will give some stragely wrong results while updating my pkgsrc tree, but no error messages of any kind. One notable example: Checkout pkgsrc/misc/gnome-user-docs/DESCR Checkout pkgsrc/misc/gnome-user-docs/Makefile Checkout pkgsrc/misc/gnome-user-docs/PLIST Checkout pkgsrc/misc/gnome-user-docs/distinfo The part that worries me is that the gnome-user-docs directory is completely empty afterwards, and this is repeatable using two different cvsup servers. (Note that most other directories update properly -- I have no clue why that directory is peculiar.) I'm wondering how many other errors may be slipping past without any error messages to alert me. I'm using cvsup from cvsup-bootstrap-20051229.tgz which (IIRC) I got from Corecode's package repository. Can anyone else confirm this result?
Re: where is cvsup?
On Thu, Oct 05, 2006 at 08:34:33PM -0700, walt wrote: > > updater.c lacks support for two commands that are needed to deal > > with newphrases(any unknown phrases in the current implementation) > > in RCS file. > > > > Please try this: > > $ cd > > $ fetch http://les.ath.cx/DragonFly/dfly-csup.tar.gz > > $ cd /usr/src > > $ tar zxf ~/dly-csup.tar.gz > > $ cpdup ~freebsd/projects/csup contrib/csup > > $ cd usr.bin/csup > > $ make obj && make depend && make && make install > > > > The csup source in contrib/csup directory in FreeBSD tree > > doesn't have the fix to queue.h, so it produces (unharmful) warnings. > > Now I see this: > > Updating collection dragonfly-cvs-src/cvs > Edit src/etc/defaults/periodic.conf > newphrase commitid ignored > Edit src/share/man/man5/periodic.conf.5 > newphrase commitid ignored > Finished successfully > > When I re-run cvsup I get no new updates -- so I guess that > means it's working in spite of the warnings, right? Yes, it merely means that it doesn't know about a newphrase `commitid'. Since commitid happens to have nothing to do with checkout mode, it can be safely ignored. I'm not 100% sure if *any* newphrase can be ignored, but I doubt one would want to add a new RCS phrase that requires special handling to older implementations without breaking compatibility. When the first time I talked to Maxime in private e-mail about this problem, he didn't seem to be convinced about ignoring newphrase, but rather cvsup and/or its protocol should be adjusted to recognize the commitid keyword to get things done correctly. He also said he was going to take care about it after a few more conversations, but I haven't heard from him or seen a commit since then. Cheers.
Re: where is cvsup?
On 06.10.2006, at 10:37, Steve O'Hara-Smith wrote: Last I checked csup didn't handle repository mode, there are alternatives to cvsup for repository copies - but when I used cvsync I wound up with a repository copy that missed some updates and rsync is slower and harder on the servers than cvsup I think. cvsync is broken as it re-oders the cvs files so that they are incompatible with cvs (in some cases). cvsup does maintain a supfile, so the server doesn't have to scan the repo all the time, so that's less resource consuming that rsync, i agree. nevertheless I don't think it is a problem for servers to serve rsync. when I update my cvs tree from chlamydia, the server scans the tree faster than my local (faster) box, probably because the tree is already cached in memory. I think the times for optimizing every last single bit to transfer or not are quite over (considering amounts of bandwidth wasted by spam). cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \ PGP.sig Description: This is a digitally signed message part
