Re: wiki unbroken
Justin C. Sherrill wrote: > How about now? I think it was missing the language packs that it looks > for when encountering a browser that's using a language different than > the default. (This is not documented in the somewhat unfocused > install/upgrade docs, so I'm taking a wild stab.) Nope. It seems to solve problem with languages moinmoin has translations to (tested with de), but it's not the case for Estonian (et) I happen to use ;). -- Hasso
Re: wiki unbroken
On Tue, January 29, 2008 2:14 am, Hasso Tepper wrote: > Doesn't work for me with any internationalised browser (tested Konqueror > and Firefox): > > AttributeError > 'NoneType' object has no attribute 'startswith' > > /usr/pkg/lib/python2.4/posixpath.py:60 > > > Switching to en_US fixes the problem. How about now? I think it was missing the language packs that it looks for when encountering a browser that's using a language different than the default. (This is not documented in the somewhat unfocused install/upgrade docs, so I'm taking a wild stab.)
Re: wiki unbroken
On Wed, January 30, 2008 5:35 am, Sdävtaker wrote: > When i do a full text search, i got this: > --> --> Fixed - moinmoin was trying to access a file as a directory, and blew up when it couldn't open it as it expected.
Re: rsync vs. cvsup benchmarks
Garance A Drosihn wrote: Just use rsync, and shut up about it already. What are you people blabering about? cvsup SUCKS. not the idea, but the language it is implemented in. and cvsup inherits the suckage. as simple as that. if it was written in a portable language, nobody would bother using rsync. vince's benchmarks were just to establish one realisation: that rsync is not significantly worse than cvsup. end of story. move on. -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: rsync vs. cvsup benchmarks
At 4:38 PM -0600 1/30/08, Vincent Stemen wrote: That's a good point. It is possible that cvsup would fair better with a matching sup directory. I actually forgot about cvsup keeping that separate state directory when I ran the benchmarks. However, from my viewpoint that does not invalidate the test results or convince me that there is any reason I would want to use cvsup for mirroring because of several reasons. 2 Rsync did not have the benefit of a local state directory either, so it was a one on one fair comparison. Based on all the cvsup claims, I would have expected it to at least come close to matching rsync's performance. Then I would expect a higher possibility of it being faster than rsync with the state directory available. Geez. Just use rsync if you want to use it. But the above reasoning is absurd. "CVSUP must live up to it's performance claims, even though I refuse to run CVSUP the way it is designed to run". "Rsync does not use this method to optimize its performance, so I refuse to let CVSUP use this method to optimize its performance. And look, CVSUP is slower than rsync as long as I make sure cvsup cannot use the performance enhancements which were designed into it!" Just use rsync, and shut up about it already. No one is asking you to use cvsup. But stop trying to defend a obviously incomplete benchmark by pulling out such bizarre reasoning. If you don't want to do a real benchmark, then just don't bother doing one. I can't blame you for that, as I also don't want to do the amount of work it would take to do a really useful benchmark. -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED]
Re: rsync vs. cvsup benchmarks
On Wed, Jan 30, 2008 at 09:54:29AM +, Rahul Siddharthan wrote: > As I understand, cvsup maintains state between updates using checkout > files in a separate "sup" directory. If you are missing that > directory, or it does not correspond to your "aged" tree, cvsup won't > do very well. You should test cvsup by checking out the tree, waiting > a week (or whatever the time is) to age it, and then updating it. > Likewise for rsync (though it doesn't require a checkouts directory). > > Rahul That's a good point. It is possible that cvsup would fair better with a matching sup directory. I actually forgot about cvsup keeping that separate state directory when I ran the benchmarks. However, from my viewpoint that does not invalidate the test results or convince me that there is any reason I would want to use cvsup for mirroring because of several reasons. 1 The state directory would have to make cvsup almost 4 times as fast to even match rsync's average performance, which I think is unlikely, considering it was only half as fast on a full download starting with an empty repository, and did not even match rsync on any of the update tests. 2 Rsync did not have the benefit of a local state directory either, so it was a one on one fair comparison. Based on all the cvsup claims, I would have expected it to at least come close to matching rsync's performance. Then I would expect a higher possibility of it being faster than rsync with the state directory available. 3 I think doing the comparison under robust conditions, and not under fragile or undependable circumstances is more realistic. For example, if you do an rsync update in between, your state directory is no longer going to match. I would not even trust the state directory not to break if you have to do an update using cvsup from a different server because the one you used last time is down. This is also the reason I did not include the '-s' switch to cvsup (which is also supposed to increase performance), because in the manual, it states, "If the -s option is used when one or more files have been modified locally, the results are undefined. Local file dam- age may remain uncorrected, updates may be missed, or cvsup may abort prematurely." I consider that to fragile. 4 All these fragile conditions to get cvsup up to maximum performance only applies to cvs repositores. If you ever decide to use a different revision control system... Considering the fragile nature and non-portability of cvsup, I am not very interested in using it even if it was a little faster than rsync. So I don't really have the time or motivation to generate benchmarks with the cvsup state directory. The way that would have to be tested would require weeks, and a lot more headache, generating and keeping all the matching state directories for each test. Vince
Re: rsync vs. cvsup benchmarks
At 6:38 AM + 1/30/08, Vincent Stemen wrote: My conclusions: === The results are dramatic, with rsync performing hundreds of percent faster on average while only loading the processor on the client side a little over a third as much as cvsup. Either the performance claims about cvsup being faster than rsync are based on theory without real world testing or cvsup has gotten a lot slower or rsync has gotten a lot faster than in the past. For those who are concerned about the validity of these results without including server side load tests and tests under bandwidth congestion conditions, here are my thoughts on the matter. No matter where a bottleneck exists in the transfer, whether it is server side load, client side load, or bandwidth limits, you are going to experience similar loss of throughput. The additional testing is nice to see, but you're not thinking the issues through far enough when it comes to scaling up a service like this. It's good to benchmark something which hasn't been tested in some time, but you have to do pretty extensive benchmarks if you're going to come to any sweeping conclusions for *all* uses of a program. Let's say rsync takes 10% of the cpu on the client, and 10% of the cpu on a server. Let's say cvsup for the same update takes 15% CPU on the client, and 7% on the server. If your benchmarks ignore the load on the server, then they can not possibly see problems which could occur when scaling up to more clients. With a single client connecting to the server, *neither* side is the bottleneck. It might be the disk-speed is the main bottleneck at that point. The update might take longer with cvsup due to the 15% CPU on the client, but the CPU isn't much of a *bottleneck* at that point. But with 10 connections in my fake scenario, rsync could be using 100% of the CPU on the server. It's at this point that rsync will see some bottleneck, while cvsup would only be using 70% of a CPU. Yes, cvsup will be using much more on each client, but then each client shows up with it's own CPU(s) to take up whatever load is thrown at that client. The server does not receive additional CPU's or network-cards for each connection that it accepts. Again, my feeling is that rsync is almost certainly fine for using with dragonfly's repository, given how much faster machines and networks have gotten, and how many simultaneous connections are seen for dragonfly repo-servers. It makes plenty of sense to stick with rsync if your servers are not overloaded. But if you want to prove rsync is better than cvsup for what the loads that cvsup was *MEANT* to solve, then your tests are not extensive enough. Benchmarking a client/server setup like cvsup is a lot of work to get a complete picture. Also note that you don't need to "prove" that rsync is "better". If it is good-enough for what Dragonfly needs at this point, then use it. It would be the people running the servers who will care about the load there, and if you have enough servers then dragonfly may never notice any problems from using rsync. Maybe a dragonfly repo-server will never see more than 10 simultaneous connections, so you'll never even hit the situations that freebsd had to deal with when cvsup was written. (I suspect there's a lot fewer people on dial-up connections now, for instance, so each individual connection will take a lot less wall-clock time than it used to, so you're much less likely to see 100 simultaneous connections). -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED]
Re: rsync considered superior
Guys, I just don't care about minor differences in client or server cpu use, or bandwidth. I think the only real issue here is the one Rahul brought up which is, in fact, the original reason why cvsup was written in the first place, so cvs tagging wouldn't require a complete redownload of the repository. I personally do not think its a big deal. We do a major tagging operation twice a year, and a week or two before the release anyhow so there's plenty of time for the servers to work through the update. Slip tags only change a few files throughout the year... it's really just the pre-release tagging that creates the issue. The issue of network bandwidth is more an issue of managing the data rate, and not so much worrying about the actual volume of information or how long it takes to settle down again as long as it isn't too long (like no more then a week or two). If it becomes a problem there are plenty of solutions in the pipeline, particularly once I start using HAMMER on the servers (which won't be soon, but probably some time later this year). At that point doing historical diffs will be trivial and clients can simply supply the last 'sync time' (as in a timestamp) and the server can produce a diff from then to current, or otherwise tell the client that it can't and the client falls back to a normal rsync. Believe me, it isn't something we have to worry about, the solution is just around the corner. -Matt Matthew Dillon <[EMAIL PROTECTED]>
Re: rsync considered superior
Simon 'corecode' Schubert wrote: *snip* Simon, Your command of the *language* is superb. But it isn't about debating skills. Test 100 simultaneous connections. Or Not. IDGASEW Bill
Re: rsync considered superior (was: Re: rsync vs. cvsup benchmarks)
"Simon 'corecode' Schubert" <[EMAIL PROTECTED]> wrote: >Thank you for these thorough tests! We finally have some hard numbers to >work with. I think it is obvious that rsync should be the preferred >update mechanism if you want to download the cvs repository. To download, yes, to update, that's not so clear. To repeat my earlier mail: Vincent appears only to have installed a tarball of recent (but not current) sources and used cvsup / rsync to update them. But to operate efficiently, cvsup needs checkout files, which it would have only if it was run previously on those sources. See the FAQ: http://www.cvsup.org/faq.html in particular, numbers 37 and 38: In order to update your files efficiently, CVSup needs to know what you've already got. It stores this information in files called "checkouts" files... If CVSup can't find a checkouts file that it needs, it falls back on other methods of determining which files you have. One such method is to compute checksums (MD5 file signatures) for each of your files, and use those to figure out which file revisions you have. This is perfectly safe, but it is inefficient. It slows down your update and also puts a heavier load on the server. To compare cvsup minus checkout-files to rsync seems quite misplaced. Most people will never use cvsup merely to download sources: they will use it to keep them regularly up-to-date. Rahul
Re: rsync considered superior
Bill Hacker wrote: To state it clearly for everybody: = Use rsync to sync your repos! It is faster and can even be compiled! To state it even MORE clearly... " ...so long as you do not give a damn about the extra load you are placing on the source server" wrong. there is no extra load. do you have numbers? if not, don't state this like a fact. I believe that cvsup will produce a higher load on the server, but that's up to curious people. rsync predates CVSUP. unix predates windows, yet windows is in wide use. is it good? who cares, we're using bsd. so who predates whom is no argument. - cvsup would never have seen the light of day in the first place. there are thousands of useless and stupid software products out there. existance is no argument for superiority. - NOR been adopted so *very* widely. very? it's seemingly only used by the bsds, and of those only intensively by freebsd. other groups use cvsync or rather don't sync the cvs repos. - NOR have remained in service for so long on so many projects. never change a running system. as long as it works, why bother? - NOR survived challenges from 'Mercurial' and several other similar tools. which challenges? mercurial is orthogonal. if you're stuck with cvs, you gotta use a repo syncer or suffer with anoncvs. A vast supposition about rsync, backed up by half-vast testing doesn't change any of that. Not even with a nicely done write-up. oh come on, seriously. you believe that? how can you pull out a couple of shady arguments and shoot down a well-founded evaluation? that's not even remotely scientific. Set up the repo you have mirrored as a source server. I am running the server. I checked the load while syncing one client. arguably that's not a founded experiment, but it is a data point. Instrument that server's load with 100 simultaneous rsync clients and again with 100 simultaneous cvsup clients. Post the results. go ahead, make my day. I for sure don't have time for such useless experiments. the choice is rsync. cvsup is dead. it is written in an unmaintainable language and it performs slower on the client. 'nuff said, no more data needed. the facts speak clearly, even if they don't cover the server part. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: rsync considered superior
Simon 'corecode' Schubert wrote: Hello Vincent, Vincent Stemen wrote: The results are dramatic, with rsync performing hundreds of percent faster on average while only loading the processor on the client side a little over a third as much as cvsup. Either the performance claims about cvsup being faster than rsync are based on theory without real world testing or cvsup has gotten a lot slower or rsync has gotten a lot faster than in the past. Thank you for these thorough tests! We finally have some hard numbers to work with. I think it is obvious that rsync should be the preferred update mechanism if you want to download the cvs repository. Cvsup might still be better suited when only downloading the checked out sources. To state it clearly for everybody: = Use rsync to sync your repos! It is faster and can even be compiled! To state it even MORE clearly... " ...so long as you do not give a damn about the extra load you are placing on the source server" WBH = cheers simon Think about it. rsync predates CVSUP. If rsync plus a bit of scripting or 'steering' code was better 'all around'? - cvsup would never have seen the light of day in the first place. - NOR been adopted so *very* widely. - NOR have remained in service for so long on so many projects. - NOR survived challenges from 'Mercurial' and several other similar tools. A vast supposition about rsync, backed up by half-vast testing doesn't change any of that. Not even with a nicely done write-up. It is all still one-ended. Set up the repo you have mirrored as a source server. Instrument that server's load with 100 simultaneous rsync clients and again with 100 simultaneous cvsup clients. Post the results. Bill
Re: rsync vs. cvsup benchmarks
Justin C. Sherrill wrote: The only minor thing I'd bring up is that I recall one reason for cvsup is that rsync placed a relatively higher load per client on the server. That needs to be established. We already heard that cvsup - contrary to claims - is not competitive with rsync, on the client side. So I can very well believe that this is also true for the server side. I for myself always notice that when syncing from chlamydia, the server basically traverses all 60k files *instantly*, while it takes quite some time on my desktop. So the load doesn't seem to be a problem once the directory structure is in the buffer cache. Of course, that may complaint may date from when people only had 400Mhz CPUs and older versions of rsync, so I doubt it's a strong reason to stay with cvsup any more. Quick test on chlamydia: rsync of already synced repo: % time rsync --delete -aH chlamydia.fs.ei.tum.de::dragonfly-cvs . rsync --delete -aH chlamydia.fs.ei.tum.de::dragonfly-cvs . 0.72s user 1.43s system 36% cpu 5.865 total considering that rsync spends half of the time on the local side, that's < 3s of load on the server: 53331 nobody 161 0 4536K 3888K select 0:00 355.68% 33.89% rsync nobody cares about that. it might take some more cycles when transfering, but so what. seriously. I don't care, this is peanuts. cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: rsync vs. cvsup benchmarks
On Wed, January 30, 2008 1:38 am, Vincent Stemen wrote: > > I have some benchmark test results comparing rsync to cvsup. I did 12 > client side tests over the last week. 5 against TheShell.com, 3 against > AllBSD.org, and 4 against chlamydia.fs.ei.tum.de. All tests were > mirroring the DragonFly BSD source repository. The tests were done > with various aged repositories at different times of the day and > night, some with compression on and some with it off. Each test was > done by unpacking two identical copies of a given aged > repository, one to run the cvsup test on and one to run the rsync test on. > Then the rsync and cvsup test was run back to back. The only minor thing I'd bring up is that I recall one reason for cvsup is that rsync placed a relatively higher load per client on the server. Of course, that may complaint may date from when people only had 400Mhz CPUs and older versions of rsync, so I doubt it's a strong reason to stay with cvsup any more.
Re: wiki unbroken
On Wed, January 30, 2008 5:35 am, Sdävtaker wrote: > When i do a full text search, i got this: > --> --> > > Traceback (most recent call last): > File I'll work on this, along with the language issues Hasso reported - I didn't get a chance last night.
rsync considered superior (was: Re: rsync vs. cvsup benchmarks)
Hello Vincent, Vincent Stemen wrote: The results are dramatic, with rsync performing hundreds of percent faster on average while only loading the processor on the client side a little over a third as much as cvsup. Either the performance claims about cvsup being faster than rsync are based on theory without real world testing or cvsup has gotten a lot slower or rsync has gotten a lot faster than in the past. Thank you for these thorough tests! We finally have some hard numbers to work with. I think it is obvious that rsync should be the preferred update mechanism if you want to download the cvs repository. Cvsup might still be better suited when only downloading the checked out sources. To state it clearly for everybody: = Use rsync to sync your repos! It is faster and can even be compiled! = cheers simon -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /"\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: wiki unbroken
When i do a full text search, i got this:
--> -->
Traceback (most recent call last):
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/request/__init__.py",
line 1283, in run
handler(self.page.page_name, self)
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/action/fullsearch.py",
line 188, in execute
results = searchPages(request, query, sort, mtime, historysearch)
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/search/__init__.py",
line 33, in searchPages
historysearch=historysearch).run()
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/search/builtin.py",
line 459, in run
hits = self._moinSearch()
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/search/builtin.py",
line 611, in _moinSearch
hits = self._getHits(pages, self._moinMatch)
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/search/builtin.py",
line 660, in _getHits
matches = matchSearchFunction(page=page, uid=uid)
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/search/builtin.py",
line 621, in _moinMatch
return self.query.search(page)
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/search/queryparser.py",
line 323, in search
body = page.get_raw_body()
File "/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/Page.py",
line 266, in get_raw_body
return self.body
File "/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/Page.py",
line 209, in get_body
f = codecs.open(self._text_filename(), 'rb', config.charset)
File "/usr/pkg/lib/python2.4/codecs.py", line 666, in open
file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 20] Not a directory:
'/usr/local/www/wiki/dfwiki/data/pages/meta/revisions/'
Additionally cgitb raised this exception:
Traceback (most recent call last):
File "/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/failure.py",
line 168, in handle
handler.handle()
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/support/cgitb.py",
line 576, in handle
doc = view.format(formatter, self.context)
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/support/cgitb.py",
line 354, in format
return formatter.section(self.formatContent(), {'class': 'cgitb'})
File "/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/failure.py",
line 35, in formatContent
content = (
File "/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/failure.py",
line 96, in formatDebugInfo
info = [self.debugInfoHideScript(),
File "/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/failure.py",
line 110, in formatTraceback
return self.formatAllTracebacks(self.formatOneTraceback)
File "/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/failure.py",
line 122, in formatAllTracebacks
tracebacks.append(formatFuction((ttype, tvalue, tb)))
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/support/cgitb.py",
line 432, in formatOneTraceback
output = [self.formatter.subTitle('Traceback'),
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/support/cgitb.py",
line 445, in tracebackFrames
frames.append(frame.format(self.formatter))
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/support/cgitb.py",
line 201, in format
vars, highlight = self.scan()
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/support/cgitb.py",
line 290, in scan
vars = self.scanVariables(reader)
File
"/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/support/cgitb.py",
line 305, in scanVariables
value = getattr(parent, token, __UNDEF__)
File "/usr/local/www/wiki/lib/python2.4/site-packages/MoinMoin/Page.py",
line 209, in get_body
f = codecs.open(self._text_filename(), 'rb', config.charset)
File "/usr/pkg/lib/python2.4/codecs.py", line 666, in open
file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 20] Not a directory:
'/usr/local/www/wiki/dfwiki/data/pages/meta/revisions/'
The term i was looking for was irc :-(
On Jan 29, 2008 10:52 AM, Justin C. Sherrill <[EMAIL PROTECTED]> wrote:
> On Tue, January 29, 2008 3:58 am, Dario 'Capn Sonic' Banno wrote:
> > Oh my...Fred is gone!
> >
> > I can't see the DragonFly logo on top.
>
> The default layout back to 'modern', the default it ships with. I had the
> 'dfly' layout as default before; I'll re-add it after I take another stab
> at improving it, as it was showing the logo but not the background when I
> tested it last night.
>
> You can switch to the dfly layout in your preferences now if you are
> suffering Fred-based anxiety.
>
>
--
Sdävtaker prays to Rikku goddess for a good treasure.
--
Sdävtaker prays to Rikku goddess for a good treasure.
Re: rsync vs. cvsup benchmarks
Vincent Stemen <[EMAIL PROTECTED]> wrote: >I have some benchmark test results comparing rsync to cvsup. I did 12 client >side tests over the last week. 5 against TheShell.com, 3 against AllBSD.org, >and 4 against chlamydia.fs.ei.tum.de. All tests were mirroring the DragonFly >BSD source repository. The tests were done with various aged repositories at >different times of the day and night, some with compression on and some with it >off. Each test was done by unpacking two identical copies of a given aged >repository, one to run the cvsup test on and one to run the rsync test on. As I understand, cvsup maintains state between updates using checkout files in a separate "sup" directory. If you are missing that directory, or it does not correspond to your "aged" tree, cvsup won't do very well. You should test cvsup by checking out the tree, waiting a week (or whatever the time is) to age it, and then updating it. Likewise for rsync (though it doesn't require a checkouts directory). Rahul
