Re: FreeBSD 7, DragonFly's status
On Mon, Mar 10, 2008 at 9:37 AM, Kris Kennaway [EMAIL PROTECTED] wrote: Bill Hacker wrote: Kris, w/r the http://people.freebsd.org/~kris/scaling/mysql.html page The link to the MySQL config: http://www.freebsd.org/%7Ekris/scaling/my.cnf ...gives me a 404. Thanks, fixed. I don't have even a Quad-core I can spare from duty at the moment, but I'd like to at least see what the relative UMP dual-core results are on one of the OpenSolaris releases we have handy. Solaris-on-x86 subjectively seems relatively faster now than 'SlowLaris' days, but still no great shakes speed-wise. I don't know how well UP will perform, but Solaris have put enormous amounts of work into their SMP implementation. One thing we found in FreeBSD is that SMP optimization work often also ends up improving UP performance at the same time, so the results may be surprising. I hope to rerun my benchmarks on an 8-core system soon (the trick has been getting Solaris netbooted). Hi Kris, Do you think you'd have a chance to load up Windows Server on the same machine and compare its MySQL and PostgreSQL to modern Linux, FreeBSD and Solaris? That's probably asking a lot, but I'm sure I'm not the only person interested in seeing how Windows' latest kernels perform for server roles. I just hope that, in such a benchmark, the userland software implementation is fair to the platform, and not degenerating to low-performance APIs from lack of optimisation. There's a lot of debate about the use of FreeBSD and Linux and others for servers, and personally I'm just happy to have so much choice available to optimise per project. But it'd be really great to know where Windows fits in the performance competition, and it's nice to have some numbers to point to when arguing that free operating systems outperform proprietary ones. -- Dmitri Nikulin Centre for Synchrotron Science Monash University Victoria 3800, Australia
Re: FreeBSD 7, DragonFly's status
Dmitri Nikulin wrote: Hi Kris, Do you think you'd have a chance to load up Windows Server on the same machine and compare its MySQL and PostgreSQL to modern Linux, FreeBSD and Solaris? I dont think there's much chance of that, sorry. I dont have access to a copy, the test machines are remotely hosted, and I doubt windows server can be booted over pxe :) I'd need someone to provide me with access to their own machine running those OSes. Kris
Re: FreeBSD 7, DragonFly's status
Dmitri Nikulin wrote: Hi Kris, Do you think you'd have a chance to load up Windows Server on the same machine and compare its MySQL and PostgreSQL to modern Linux, FreeBSD and Solaris? I dont think there's much chance of that, sorry. I dont have access to a copy, the test machines are remotely hosted, and I doubt windows server can be booted over pxe :) I'd need someone to provide me with access to their own machine running those OSes. Kris
Re: FreeBSD 7, DragonFly's status
On Mon, Mar 10, 2008 at 9:11 PM, Kris Kennaway [EMAIL PROTECTED] wrote: Dmitri Nikulin wrote: Hi Kris, Do you think you'd have a chance to load up Windows Server on the same machine and compare its MySQL and PostgreSQL to modern Linux, FreeBSD and Solaris? I dont think there's much chance of that, sorry. I dont have access to a copy, the test machines are remotely hosted, and I doubt windows server can be booted over pxe :) I'd need someone to provide me with access to their own machine running those OSes. I was afraid you'd say that. I'm a pretty average consumer hardware-wise (I like to think I make up for it with software :P) so all I have is an Athlon64 X2 and a portable Core Duo. By the way, jeffr mentions: http://jeffr-tech.livejournal.com/18706.html Next up, we now have a 16 way xeon and 16 way opteron system to tune and test with. More points of contention are being removed. The code marches on. Perhaps future benchmarks can use those systems, though I don't know the exact details of how those machines are provided and their limitations. -- Dmitri Nikulin Centre for Synchrotron Science Monash University Victoria 3800, Australia
Re: FreeBSD 7, DragonFly's status
Kris Kennaway [EMAIL PROTECTED] writes: The assertion is often made by dragonfly project supporters that dragonfly has much better stability than FreeBSD. It is not clear by what metric this is being objectively evaluated (if at all). ... Obviously one panic does not demonstrate wide-ranging system instability, but it does point to a possible selection bias amongst the project supporters, who may not be looking hard enough for the stability problems that exist. The very definition of supporter implicitly contains selection bias. I daresay one cannot be an unbiased supporter and a human being at the same time. Thus what you say here reads to me like a tautology. Pointing to various events is all well and good (e.g. the stress2 panic for DragonFly, unplugging mounted USB storage devices from FreeBSD), and I understand the need for advocacy and challenge among various supporters of projects...but I am keenly interested in some sort of objective metric for stability beyond stress test panics and unfixable bugs. Does an objective metric of stability actually exist? ( If you say uptime I'll take that as a no ;) ) If it does, I would really like to learn what that metric is. Do you know of any current low-project-bias work that has been done in this area? Thanks in advance. :) -- Dave Hayes - Consultant - Altadena CA, USA - [EMAIL PROTECTED] The opinions expressed above are entirely my own Envy devours good deeds, as a fire devours fuel.
Re: FreeBSD 7, DragonFly's status
Dave Hayes wrote: Does an objective metric of stability actually exist? ( If you say uptime I'll take that as a no ;) ) If it does, I would really like to learn what that metric is. Do you know of any current low-project-bias work that has been done in this area? Thanks in advance. :) It's easiest to define stability by lack of instability, i.e. system does not crash no matter what you do to it. The best method I know to evaluate this is by brute force (other techniques like static code analysis and formal model checking can help). You have to try really hard to put the system through all kinds of bizarre contortions in the workloads you care about (which is everything for a general OS developer) until you find something that breaks. Then fix it and try again. You have to put serious effort into it though, because after you fix all the obvious panics that can be reproduced in a few minutes of testing, you end up trying to cause extremely low probability events that can (and do) nevertheless pop up on real systems given the right combination of circumstances. If you don't put in the work, the bugs will usually not get fixed until they crash a user's system and they bother to report the bug. This doesn't always happen, often they just curse you out. Once you can no longer trigger bugs no matter how hard you try (assuming you are achieving good coverage of the system), I think it's reasonable to provisionally award the label of pretty stable to the aspects of the system you have been testing. There will always be more bugs than those you found (especially with particular hardware configurations), but at least you've made a concerted effort to find them. This is basically what stress2 and other tools try to help automate, although it can never replace human-driven QA. It's literally a full time job to do properly. Kris
Re: FreeBSD 7, DragonFly's status
Kris Kennaway wrote: Adrian Michael Nida wrote: SnipAndRearrange/ The benchmark at http://people.freebsd.org/~kris/scaling/os-mysql.png SnipAndRearrange/ Is measuring 1.8. We're at 1.12 now. I'm sure an updated graph has a different trend. Take it upon yourself to redo the benchmark. Hi Adrian, Per your request I reran the benchmark, and I also took the opportunity to do some additional performance comparisons between Dragonfly 1.12, FreeBSD 4.11 and FreeBSD 7.0. Here is a brief report of my results. Kris Kris, w/r the http://people.freebsd.org/~kris/scaling/mysql.html page The link to the MySQL config: http://www.freebsd.org/%7Ekris/scaling/my.cnf ..gives me a 404. I don't have even a Quad-core I can spare from duty at the moment, but I'd like to at least see what the relative UMP dual-core results are on one of the OpenSolaris releases we have handy. Solaris-on-x86 subjectively seems relatively faster now than 'SlowLaris' days, but still no great shakes speed-wise. Thanks, Bill = In May 2007 I ran some benchmarks of Dragonfly 1.8 to evaluate progress of its SMP implementation, which was the original focus of the project when it launched in 2003 and is still widely believed to be an area in which they had made concrete progress. This was part of a larger cross-OS multiprocessor performance evaluation comparing improvements in FreeBSD to Linux, NetBSD and other operating systems. The 2007 results [1] showed essentially no performance increase from multiple processors on dragonfly 1.8, in contrast to the performance of FreeBSD 7.0 which scaled to 8 CPUs on the benchmark. Recently Dragonfly 1.12 was released, and the question was raised on the dragonfly-users list [2] of how well the OS performs after a further year of development. I performed several benchmarks to study this question. MySQL - This is a good general test of kernel performance and parallelism, as well as performance of the thread library. MySQL performance (together with PostgreSQL performance) has been a driving force in FreeBSD, Linux and NetBSD SMP development over the past year, see http://people.freebsd.org/~kris/scaling/os-mysql.png In this round of testing I compared Dragonfly 1.12-RELEASE, FreeBSD 4.11-STABLE and FreeBSD 7.0-RELEASE, running on the same 8-core Xeon hardware. On Dragonfly and FreeBSD 4.11 the GENERIC kernel configuration was used except for enabling SMP and APIC_IO (for the SMP tests), and removing I486_CPU. Under FreeBSD 7.0 the GENERIC kernel was used except for enabling the SCHED_ULE scheduler, removing I486_CPU and enabling SMP when appropriate. The test applications were compiled from ports/pkgsrc and the same versions and configuration options used for each OS. Other configuration is the same as in my previous test and is also documented here http://people.freebsd.org/~kris/scaling/mysql.html Here are the results: http://people.freebsd.org/~kris/scaling/dfly-mysql.png Dragonfly 1.12 achieves peak SMP performance of only 15% better than UP performance, and drops to about 50% below UP performance at higher loads. Enabling SMP has a 20% performance overhead on this benchmark. UP mode is faster than 4.11 when using the libthread_xu library. With libc_r (not graphed) performance is identical to 4.11 in both UP and SMP mode, so the UP performance increase is most likely due to the thread library. Note: I am using mysql 5.0.51 in the current tests, which has different performance characteristics than the older 5.0.37 tested last year, so the current data cannot directly be compared to the previous dragonfly 1.8 graphs to evaluate whether a small amount of progress was made since 1.8. However, there does not appear to be any significant performance improvement from dragonfly 1.8 to 1.12. FreeBSD 7.0 scales to 8 CPUs on this benchmark. Peak performance is 6.5 times higher than peak dragonfly performance, and 9.0 times higher than FreeBSD 4.11 performance. UP performance is consistent with SMP performance with a single thread. 7.0 UP is 45% faster than 4.11 UP and 10% faster than dragonfly UP. Note that while these benchmarks are on a test system with 8 CPU cores, the results also provide information about performance on systems with fewer than 8 cores, such as dual core systems. If the system does not show appreciable performance gain when 2 threads are running and most CPUs are idle, it is unlikely to perform much better when the system only has 2 CPUs. I could not test this directly because I don't know how to disable CPUs at boot time/run time in dragonfly. For example, this graph shows FreeBSD 7.0 running postgresql on the same system with 1, 2, 4 or 8 CPU cores active, as well as comparing the UP and SMP kernel running with 1 CPU active http://people.freebsd.org/~kris/scaling/pgsql-ncpu.pdf The performance seen with 8 CPUs also scales down to 1, 2 and 4 CPUs. This also shows that there is negligible overhead from
Re: FreeBSD 7, DragonFly's status
Bill Hacker wrote: Kris, w/r the http://people.freebsd.org/~kris/scaling/mysql.html page The link to the MySQL config: http://www.freebsd.org/%7Ekris/scaling/my.cnf ...gives me a 404. Thanks, fixed. I don't have even a Quad-core I can spare from duty at the moment, but I'd like to at least see what the relative UMP dual-core results are on one of the OpenSolaris releases we have handy. Solaris-on-x86 subjectively seems relatively faster now than 'SlowLaris' days, but still no great shakes speed-wise. I don't know how well UP will perform, but Solaris have put enormous amounts of work into their SMP implementation. One thing we found in FreeBSD is that SMP optimization work often also ends up improving UP performance at the same time, so the results may be surprising. I hope to rerun my benchmarks on an 8-core system soon (the trick has been getting Solaris netbooted). Kris
Re: FreeBSD 7, DragonFly's status
Adrian Michael Nida wrote: SnipAndRearrange/ The benchmark at http://people.freebsd.org/~kris/scaling/os-mysql.png SnipAndRearrange/ Is measuring 1.8. We're at 1.12 now. I'm sure an updated graph has a different trend. Take it upon yourself to redo the benchmark. Hi Adrian, Per your request I reran the benchmark, and I also took the opportunity to do some additional performance comparisons between Dragonfly 1.12, FreeBSD 4.11 and FreeBSD 7.0. Here is a brief report of my results. Kris = In May 2007 I ran some benchmarks of Dragonfly 1.8 to evaluate progress of its SMP implementation, which was the original focus of the project when it launched in 2003 and is still widely believed to be an area in which they had made concrete progress. This was part of a larger cross-OS multiprocessor performance evaluation comparing improvements in FreeBSD to Linux, NetBSD and other operating systems. The 2007 results [1] showed essentially no performance increase from multiple processors on dragonfly 1.8, in contrast to the performance of FreeBSD 7.0 which scaled to 8 CPUs on the benchmark. Recently Dragonfly 1.12 was released, and the question was raised on the dragonfly-users list [2] of how well the OS performs after a further year of development. I performed several benchmarks to study this question. MySQL - This is a good general test of kernel performance and parallelism, as well as performance of the thread library. MySQL performance (together with PostgreSQL performance) has been a driving force in FreeBSD, Linux and NetBSD SMP development over the past year, see http://people.freebsd.org/~kris/scaling/os-mysql.png In this round of testing I compared Dragonfly 1.12-RELEASE, FreeBSD 4.11-STABLE and FreeBSD 7.0-RELEASE, running on the same 8-core Xeon hardware. On Dragonfly and FreeBSD 4.11 the GENERIC kernel configuration was used except for enabling SMP and APIC_IO (for the SMP tests), and removing I486_CPU. Under FreeBSD 7.0 the GENERIC kernel was used except for enabling the SCHED_ULE scheduler, removing I486_CPU and enabling SMP when appropriate. The test applications were compiled from ports/pkgsrc and the same versions and configuration options used for each OS. Other configuration is the same as in my previous test and is also documented here http://people.freebsd.org/~kris/scaling/mysql.html Here are the results: http://people.freebsd.org/~kris/scaling/dfly-mysql.png Dragonfly 1.12 achieves peak SMP performance of only 15% better than UP performance, and drops to about 50% below UP performance at higher loads. Enabling SMP has a 20% performance overhead on this benchmark. UP mode is faster than 4.11 when using the libthread_xu library. With libc_r (not graphed) performance is identical to 4.11 in both UP and SMP mode, so the UP performance increase is most likely due to the thread library. Note: I am using mysql 5.0.51 in the current tests, which has different performance characteristics than the older 5.0.37 tested last year, so the current data cannot directly be compared to the previous dragonfly 1.8 graphs to evaluate whether a small amount of progress was made since 1.8. However, there does not appear to be any significant performance improvement from dragonfly 1.8 to 1.12. FreeBSD 7.0 scales to 8 CPUs on this benchmark. Peak performance is 6.5 times higher than peak dragonfly performance, and 9.0 times higher than FreeBSD 4.11 performance. UP performance is consistent with SMP performance with a single thread. 7.0 UP is 45% faster than 4.11 UP and 10% faster than dragonfly UP. Note that while these benchmarks are on a test system with 8 CPU cores, the results also provide information about performance on systems with fewer than 8 cores, such as dual core systems. If the system does not show appreciable performance gain when 2 threads are running and most CPUs are idle, it is unlikely to perform much better when the system only has 2 CPUs. I could not test this directly because I don't know how to disable CPUs at boot time/run time in dragonfly. For example, this graph shows FreeBSD 7.0 running postgresql on the same system with 1, 2, 4 or 8 CPU cores active, as well as comparing the UP and SMP kernel running with 1 CPU active http://people.freebsd.org/~kris/scaling/pgsql-ncpu.pdf The performance seen with 8 CPUs also scales down to 1, 2 and 4 CPUs. This also shows that there is negligible overhead from running the FreeBSD 7.0 SMP kernel on a UP system on this workload. Filesystem I/O -- Currently the major focus of the dragonfly project is the development of a new filesystem, so it's interesting to see how well the dragonfly filesystem layer performs. I created a 500MB memory filesystem (MFS) and used sysbench to perform multi-threaded random write I/O. It seems that MFS cannot create file systems larger than 500MB, which was a limiting factor on dragonfly and 4.11.
Re: FreeBSD 7, DragonFly's status
On Sat, March 8, 2008 6:37 pm, Kris Kennaway wrote: Dragonfly 1.12 UP performance is about 30% lower than FreeBSD 4.11 UP performance. This regression seems strange; I don't think mfs has been touched much; it may be an indirect effect of something else. The first problem was encountered while trying to unpack the stress2 archive to NFS: You wrote first problem - what was the second? Did you ever run this stress2 test? What was the NFS server? I can unscentifically note that I've been reading/writing files on NFS while doing the bulk builds and other stuff for some time, and have yet to encounter an issue.
Re: FreeBSD 7, DragonFly's status
Justin C. Sherrill wrote: On Sat, March 8, 2008 6:37 pm, Kris Kennaway wrote: Dragonfly 1.12 UP performance is about 30% lower than FreeBSD 4.11 UP performance. This regression seems strange; I don't think mfs has been touched much; it may be an indirect effect of something else. Yeah, it's likely to be a side effect of other architectural changes in Dragonfly. The first problem was encountered while trying to unpack the stress2 archive to NFS: You wrote first problem - what was the second? Did you ever run this stress2 test? The second problem was the panic. What was the NFS server? I can unscentifically note that I've been reading/writing files on NFS while doing the bulk builds and other stuff for some time, and have yet to encounter an issue. It's a FreeBSD 7.0 NFS server. Kris
Re: FreeBSD 7, DragonFly's status
Kris Kennaway wrote: giga-snippage Summary --- As with the dragonfly 1.8 kernel, the dragonfly 1.12 kernel does not scale to a second CPU on the benchmarks performed, and the limited SMP implementation can cause a large performance loss at higher loads. There is sometimes a large performance overhead from enabling SMP compared to UP, and performance was sometimes worse than that of 4.11. In all cases measured, FreeBSD 7.0 performs significantly better than both FreeBSD 4.11 and dragonfly 1.12 in both SMP and UP configurations. Hi Kris, I've been running a FBSD home firewall since before DragonFly was born, and I've never changed because it does what I need. Therefore I can hardly be called anti-FBSD. I think you will concede that FBSD started down a very bumpy road with FBSD 5, but the subsequent improvement has been impressive, yes? I give all the FBSD devs great credit for that. Matt forked from FBSD because he had a different road in mind -- a road that is just as long as FBSD's, but with a tiny fraction of the manpower that FBSD has enjoyed in the meantime. If any DF evangelists have been goading or taunting the FBSD people with claims of 'victory' I would hope that the more mature audience would just let them pass as testosterone- fired braggadocio. This project continues to rely on improvements imported from FBSD. I hope that FBSD will likewise benefit from DragonFly. That's what the opensource world is all about (but I'm preaching to the choir). I will continue to follow both projects (and others) with great interest.
Re: FreeBSD 7, DragonFly's status
Rahul Siddharthan [EMAIL PROTECTED] writes: I'm still keeping an eye on DragonFly, and hope to run it again one of these days. I think the biggest problem in FreeBSD that DragonFly fixes is attitude. It's not a completely accurate answer, but attitude is certainly sufficient in my book. I find your comment very appropriate. In fact, a careful observer might find this as a primary reason people are still using this system despite these apparent shortcomings of SMP, 64-bit, etc. In my experience, technical issues are much easier to fix than collective attitudes. I think all DragonFly really needs is more good developers, and this appears to be happening at the correct speed to preserve the fresh attitude this community currently has. As to the original posting, technical comparison is normally a good thing. However: Dmitri Nikulin [EMAIL PROTECTED] writes: I'm gravely concerned about the growth and survival of this brilliant project in the face of increasing pressure from projects with much larger developer communities and software ecosystems. Exactly what pressure are you referring to here? :) -- Dave Hayes - Consultant - Altadena CA, USA - [EMAIL PROTECTED] The opinions expressed above are entirely my own Every stick has two ends.
Re: FreeBSD 7, DragonFly's status
On Thu, Feb 28, 2008 at 4:09 PM, Justin C. Sherrill [EMAIL PROTECTED] wrote: On Wed, February 27, 2008 11:29 pm, Dmitri Nikulin wrote: The benchmark at http://people.freebsd.org/~kris/scaling/os-mysql.png (for the full presentation, see http://www.freedomtc.com/pdf/7.0_Preview.pdf, that plot is on slide 17) indicates that FreeBSD 7 not only competes strongly with current Linux performance and scalability, but that DragonFly has been beaten even by NetBSD which came late to the SMP party. Minor quibble: you're pointing at benchmarks on an 8-core xeon. Relatively uncommon hardware, though I'm sure that'll change within a year or so. Sure, but SMP scalability is one of the key goals of DragonFly, and for it to be beaten by NetBSD (for which this is not a major goal, and for which SMP scalability only started being worked on a year ago) is very confusing. I haven't seen it compared with 1.12, but since no huge scalability work has gone in for a long time, I doubt it would close the gap much. Even so, while 8-core Xeons are not yet common, AMD64 compatible chips are pretty much the only mass consumer chips available now, and DragonFly can still only use the 32 bit portion of that, and even then, doesn't yet scale to two cores fully, let alone the increasingly common quad cores. That falls far short of the goal to scale well for increasingly parallel systems, which continue to be picked up by Linux and now FreeBSD at an increasing rate. That's really disappointing for an operating system I originally believed would humiliate FreeBSD's SMP implementation. At the risk of sounding like a troll, may I ask, if FreeBSD 7 has high performance, high stability and remains reasonably clean and maintainable, doesn't that partly invalidate the reasons DragonFly was created? Being cleaner and more revolutionary doesn't count for much if the product itself doesn't serve as a compelling alternative. Maybe I'm just missing the point of DragonFly's development. There were people attracted to DragonFly early on because they were mistreated by various folks involved with FreeBSD, but liking DragonFly because it's not FreeBSD is not enough to build a project. DragonFly is not a FreeBSD replacement, nor does DragonFly require FreeBSD to suck in order to exist. You could probably make the exact same argument you just did but substitute in the name of another BSD, if you are going to take the tack of fastest benchmark so why use anything else?. (I realize I'm oversimplifying your words.) You're right - you are oversimplifying my words. I'm only saying that, for goals which DragonFly considers groundwork, FreeBSD still matches or wins by a large margin. That includes good performance (FreeBSD wins) and stability (FreeBSD seems to be no worse off). I can't attest to the maintainability, but they seem to be doing alright, and cleanups get done too. So while DragonFly has a lot going for it (which I never denied), it's challenging to find a situation for which DragonFly is actually a better choice than FreeBSD. DragonFly is oriented towards creating a single system image over multiple networked computers on the Internet, which has never been done before. It's years off even for DragonFly. If it does one day achieve that goal, but by then other systems have advanced to the point that DragonFly isn't even compelling for a single node, why will anyone want to run it just to join a cluster? It needs some minimum user base before internet clustering becomes useful at all. That's my biggest fear regarding this project - that by the time its highest goals are achievable, the full potential of those goals will still be out of reach, perhaps forever. FreeBSD - run on x86 OpenBSD - run securely NetBSD - run on anything DragonFly - run once everywhere? For myself, I find DragonFly useful because it's a tight and friendly developer community, with lots of interesting projects to do. FreeBSD having a good mysql benchmark doesn't affect that. neglected. Again, at the risk of sounding like a troll, I'm gravely concerned about the growth and survival of this brilliant project in the face of increasing pressure from projects with much larger developer communities and software ecosystems. We've been doing pretty well, really. The number of changes and new developers in 2007 have been on an upswing relative to 2006. (I should know, given my digest work.) There's been enough happening that I've had a steady 2-day buffer of things to post for a month, which has never happened before. That's really good to hear. It's hard to tell just from mailing lists. See, my problem is, I really care about DragonFly from a very geeky perspective. I want the clever, somewhat revolutionary ideas to win and show the rest of the world how it's done. But on the other hand, Inferno was set to do that, and how relevant is that now? -- Dmitri Nikulin
Re: FreeBSD 7, DragonFly's status
On Thu, 28 Feb 2008 21:53:36 +1100 Dmitri Nikulin [EMAIL PROTECTED] wrote: On Thu, Feb 28, 2008 at 4:09 PM, Justin C. Sherrill [EMAIL PROTECTED] wrote: On Wed, February 27, 2008 11:29 pm, Dmitri Nikulin wrote: The benchmark at http://people.freebsd.org/~kris/scaling/os-mysql.png (for the full presentation, see http://www.freedomtc.com/pdf/7.0_Preview.pdf, that plot is on slide 17) indicates that FreeBSD 7 not only competes strongly with current Linux performance and scalability, but that DragonFly has been beaten even by NetBSD which came late to the SMP party. Minor quibble: you're pointing at benchmarks on an 8-core xeon. Relatively uncommon hardware, though I'm sure that'll change within a year or so. Sure, but SMP scalability is one of the key goals of DragonFly, and for it to be beaten by NetBSD (for which this is not a major goal, and for which SMP scalability only started being worked on a year ago) is very confusing. I haven't seen it compared with 1.12, but since no huge scalability work has gone in for a long time, I doubt it would close the gap much. AIUI not using the approach to SMP that FreeBSD used was a major factor in starting DragonFly on the basis that it hurts single proceesor performance and is inelegant. I'm sure the DragonFly team would welcome a developer determined to push the BGL steadily further inwards until it vanishes. We would then all find out if the DragonFly approach to SMP really is the best. Until that work is done there is no real way of telling. So while DragonFly has a lot going for it (which I never denied), it's challenging to find a situation for which DragonFly is actually a better choice than FreeBSD. I really like the continuous rewindable backups I get from the journal system in DragonFly, AFAIK that's still unique to DragonFly. I look forward to better ones with Hammer. All my boxes are uniprocessor so SMP performance really doesn't matter at all to me, but losing data does :) -- C:WIN | Directable Mirror Arrays The computer obeys and wins.| A better way to focus the sun You lose and Bill collects. |licences available see |http://www.sohara.org/
Re: FreeBSD 7, DragonFly's status
On Thu, February 28, 2008 5:53 am, Dmitri Nikulin wrote: before internet clustering becomes useful at all. That's my biggest fear regarding this project - that by the time its highest goals are achievable, the full potential of those goals will still be out of reach, perhaps forever. Only if someone else figures out an open-source SSI system first. I'm not so worried about this because open source operating systems aren't really important any more. People care about the applications running on them. People are far more concerned about a system's ability to match their application environment than anything else these days. Putting it another way: if we couldn't reliably run Apache, we'd be much more worried than if we're relatively low on a benchmark. We've got a big leg up for having pkgsrc and having so many pkgsrc packages run well on DragonFly. (much credit to Joerg for that.) Anyway, this is an open-source project. We don't have to meet any goals or profit levels unless we want to; the only thing we need to do is enjoy what we are doing. Judging by other people's responses here, that's working. That's really good to hear. It's hard to tell just from mailing lists. See, my problem is, I really care about DragonFly from a very geeky perspective. I want the clever, somewhat revolutionary ideas to win and show the rest of the world how it's done. But on the other hand, Inferno was set to do that, and how relevant is that now? I'm curious to see what the commit frequency is over time for DragonFly; I was hoping Ohloh would have that tracked, but I don't see it. There's other folks out there that have solutions graphing from CVS, but nothing yet that's a drop-in solution. ( http://www.ohloh.net/projects/7261/analyses/latest ) Commit frequency isn't an exact measure, but I love infoporn as much as any other geek.
Re: FreeBSD 7, DragonFly's status
Sure, but SMP scalability is one of the key goals of DragonFly, and for it to be beaten by NetBSD (for which this is not a major goal, and for which SMP scalability only started being worked on a year ago) is very confusing. I haven't seen it compared with 1.12, but since no huge scalability work has gone in for a long time, I doubt it would close the gap much. While a scalable SMP implementation is a goal of DragonFly, so is many other things. There isn't very many people around besides Matt that have the time and the skill to help pushing the big giant lock out. it simple as that. Im sure more developers would be welcome. But if evrything goes as planned with HAMMER, i think you'll be seeing more interest in DragonFly quite soon and probably much higher SMP performance by the end of the year. Petr
Re: FreeBSD 7, DragonFly's status
Well, I'll give you my 5-second opinion. What I am not worried about: * Developer interest has always increased slowly and continues to do so. I'd be interested in commit statistics but my gut feeling, from NOT having to push into subsystems that I used to have to push into, is that more developers are working on more of the critical infrastructure. * More developers are doing the 'harder' aspects of kernel development rather then just me. This is very, very important. * Ports and packages. This was a huge worry of mine at the beginning of the project. I no longer worry about it. What I am worried about: * The big-ticket items that we have traditionally compared ourselves against, SMP primarily, have developed slowly, primarily because up until recently I was the only person working on it. It an issue of man-hours more then anything else. FreeBSD simply has more developers working on SMP then we do. We have the infrastructure and the abstraction, we now need to get down to the brass balls and finish it. * I really want to see someone take up the ball on getting the network stack MP safe. It's important to the project but I can't do that and HAMMER at the same time. It's the easiest thing to make MP safe in the project. * Similarly with AMD64. We need it. I've developed the infrastructure separation required and we even have a fully virtualized kernel (vkernel) which demonstratres the infrastructure separation. Most of the generic kernel code can easily support a 64 bit architecture. The compiler supports it. It's a project waiting for someone to take up the ball. * Our interrupt routing subsystem really needs a major upgrade. (i.e. a major port from FreeBSD). Where I think the future is: * SMP, Storage, and SSI. Real time mirroring at the logical level. HAMMER is a major component for the storage, SSI, and mirroring components, and I believe HAMMER will be a large interest magnet. Our project goals have not changed, but if I had it all to do over again I would have started work on HAMMER much earlier then I did. I spent more time then I should have perfecting the low level infrastructure, trying to build a base upon which all the other work could occur. -Matt
Re: FreeBSD 7, DragonFly's status
Matthew Dillon wrote: * Similarly with AMD64. We need it. I've developed the infrastructure separation required and we even have a fully virtualized kernel (vkernel) which demonstratres the infrastructure separation. Most of the generic kernel code can easily support a 64 bit architecture. The compiler supports it. It's a project waiting for someone to take up the ball. Note that Noah Yan has been working on an AMD64 port. Some stuff is already committed, and last time I checked, he had about 328KB of uncommitted patches in his git repository (http://repo.or.cz/w/dragonfly/port-amd64.git). He doesn't seem to have much time at the moment but said he'll return to it as soon as the stuff he's occupied with has settled down a bit. Sascha -- http://yoyodyne.ath.cx
Re: FreeBSD 7, DragonFly's status
Matthew Dillon wrote: Well, I'll give you my 5-second opinion. *snip* * Our interrupt routing subsystem really needs a major upgrade. (i.e. a major port from FreeBSD). Given that theirs has choked several times on some fairly common hardware that DID work thru 6.2 RELEASE, that would not necessarily be the first or best place to look. Where I think the future is: * SMP, Storage, and SSI. Real time mirroring at the logical level. HAMMER is a major component for the storage, SSI, and mirroring components, and I believe HAMMER will be a large interest magnet. Our project goals have not changed, but if I had it all to do over again I would have started work on HAMMER much earlier then I did. I spent more time then I should have perfecting the low level infrastructure, trying to build a base upon which all the other work could occur. -Matt It may seem so in the rear-view mirror, but had you NOT done the low-level infrastructure, AND the 2+ year code-clean-up of what was adapted from Free (and other) BSD, nothing else would be working as well as it does. That was time that pays back with long-running and ongoing dividends. JM2CW, Bill
Re: FreeBSD 7, DragonFly's status
On Thu, Feb 28, 2008 at 07:13:25PM +, Bill Hacker wrote: Matthew Dillon wrote: I spent more time then I should have perfecting the low level infrastructure, trying to build a base upon which all the other work could occur. -Matt It may seem so in the rear-view mirror, but had you NOT done the low-level infrastructure, AND the 2+ year code-clean-up of what was adapted from Free (and other) BSD, nothing else would be working as well as it does. That was time that pays back with long-running and ongoing dividends. JM2CW, Bill I agree. I really appreciate that you spent the time perfecting the low level infrastructure first. That is one of the reasons we have switched our projects over to DragonFly. I am tired of systems that are like a sky scraper built on top of a rickety foundation. That is one of the reasons Linux became so unstable that we left it after being on it for at least 10 years, and again with FreeBSD. When FreeBSD-5.x became so unstable that we could not even complete NFS backups without the server hanging, and stayed that way for months, we left it and went to NetBSD, which is still a pretty nice system, although not as feature rich in a lot of ways. We did not wait for years for FreeBSD to stabilize like we did with Linux (Hard lesson learned) before discovering it was not going to. Of course, I hope that does not prove true with FreeBSD. FreeBSD still seem to have a more professional development community than Linux. However, we have since switched to DragonFly because of architecture, planning, developer attitude, and emphasis on clean code and stability. I believe a perfected low level infrastructure should pay off in the long run with accelerated development without loosing stability. I have been impressed with the progress of DragonFly so far, especially considering the comparatively modest number of developers. Vince
Re: FreeBSD 7, DragonFly's status
Stability is important to me but I also recognize that even the best project can become stale if one does not choose to develop the right aspects of it. A weakness in DragonFly is that it took a while to get to the more interesting things, like HAMMER, and will take yet longer to get to SSI. One of FreeBSD's strengths is that its brute-force method of development tends to pull in more interest simply by the sheer number of projects being worked on parallel. One of its weaknesses is a lack of stability. I far prefer our low level infrastructure. Our abstractions are an order of magnitude cleaner: VFS, timers, schedulers, cpu messaging, threading, namecache, VM paths, virtualization, PRNG, network drivers, network protocols, route table, and the list goes on. I am also very happy that all that infrastructure work is now basicaly done and I can focus on the more interesting aspects of the project. How do I explain to a lay person why moving the responsibility for namespace and I/O atomicy (range locking) into the kernel was important? It's hard. -Matt
Re: FreeBSD 7, DragonFly's status
On Fri, Feb 29, 2008 at 5:05 AM, Matthew Dillon [EMAIL PROTECTED] wrote: Well, I'll give you my 5-second opinion. What I am not worried about: ... * Ports and packages. This was a huge worry of mine at the beginning of the project. I no longer worry about it. Yeah, pkgsrc is a real life saver. I discovered NetBSD around the same time DragonFly was picking up pkgsrc, so it was very pleasant for me to have a reasonably uniform administrative strategy across the two operating systems. What I am worried about: * The big-ticket items that we have traditionally compared ourselves against, SMP primarily, have developed slowly, primarily because up until recently I was the only person working on it. It an issue of man-hours more then anything else. FreeBSD simply has more developers working on SMP then we do. We have the infrastructure and the abstraction, we now need to get down to the brass balls and finish it. Well yeah, that's my point. FreeBSD's vast developer base lets it get away with relatively unclean code and still remain relevant in performance and functionality. Stability has picked up a lot too, although that's hard to measure with hard evidence. * I really want to see someone take up the ball on getting the network stack MP safe. It's important to the project but I can't do that and HAMMER at the same time. It's the easiest thing to make MP safe in the project. Wasn't it almost done years ago, and then virtually untouched thereafter? I haven't been keeping up with source-level changes like that, but I got the impression the last 10% takes 99.99% of the time which is why it's still not done. * Similarly with AMD64. We need it. I've developed the infrastructure separation required and we even have a fully virtualized kernel (vkernel) which demonstratres the infrastructure separation. Most of the generic kernel code can easily support a 64 bit architecture. The compiler supports it. It's a project waiting for someone to take up the ball. * Our interrupt routing subsystem really needs a major upgrade. (i.e. a major port from FreeBSD). That's pretty much a showstopper for a lot of deployments. It's things like that which I believe bring the project down. Even if it's only a problem for 6 months, that's 6 months worth of new users which may be discouraged from using the system, or use another system temporarily that never gets replaced with DragonFly again. This has brought down Linux and other projects a lot in the past, where a critical bug was left unchecked long enough that hundreds or thousands of users took years to try again. I contend that fixing major problems must come before implementing new technology, especially if that new technology is likely to add new problems. If somebody installs DragonFly just to try HAMMERFS and discovers half of his hardware fails because of an interrupt bug, what good does that do? Where I think the future is: * SMP, Storage, and SSI. Real time mirroring at the logical level. HAMMER is a major component for the storage, SSI, and mirroring components, and I believe HAMMER will be a large interest magnet. I for one don't like to speculate about the future like that, because such gambles often cost a lot even to companies who have a lot of information from which to predict. Although it's been said the best way to know the future is to invent it, a lot of inventors have had their efforts wasted because of the lack of marketability of their inventions. Our project goals have not changed, but if I had it all to do over again I would have started work on HAMMER much earlier then I did. I spent more time then I should have perfecting the low level infrastructure, trying to build a base upon which all the other work could occur. No, I agree with everyone who says the low level stuff is what makes this project great in the first place. I'm just concerned it's nowhere near enough to take enough market share that SSI becomes truly useful. If it's just 1000 hardcore geeks in the whole world split into a few SSIs, what practical benefit is there? They may as well SSH in to a few powerful machines, and even consumer machines are becoming ludicrously powerful lately. The security and efficiency of shell access is a much more mature field than the security and efficiency of SSI in general, saying nothing of any specific implementation. And then, application clustering for web serving, databases, etc. is already largely solved on the application level, often much more optimally than a general approach could ever be. SSI doesn't help there either, unfortunately. -Matt -- Dmitri Nikulin
Re: FreeBSD 7, DragonFly's status
Quoting Dmitri Nikulin [EMAIL PROTECTED]: Hi everyone, I've been busy so I haven't said anything here in a long time, but reading about FreeBSD 7 has raised some thoughts. Snip/ Same here. From one old busy quiet guy to another, welcome back. SnipAndRearrange/ I look forward to being told I'm completely wrong and everything is much better than it seems :) SnipAndRearrange/ You're completely wrong and everything is much better than it seems Happy? Let's now show you why... SnipAndRearrange/ The benchmark at http://people.freebsd.org/~kris/scaling/os-mysql.png SnipAndRearrange/ Is measuring 1.8. We're at 1.12 now. I'm sure an updated graph has a different trend. Take it upon yourself to redo the benchmark. SnipAndRearrange/ At the risk of sounding like a troll, may I ask, if FreeBSD 7 has high performance, high stability and remains reasonably clean and maintainable, doesn't that partly invalidate the reasons DragonFly was created? SnipRearrangeAndClippedForBandwith/ Partly, but there are will always be many more for why it should continue. Focus on those, we still have a long way to fly. Adrian
Re: FreeBSD 7, DragonFly's status
On Wed, February 27, 2008 11:29 pm, Dmitri Nikulin wrote: The benchmark at http://people.freebsd.org/~kris/scaling/os-mysql.png (for the full presentation, see http://www.freedomtc.com/pdf/7.0_Preview.pdf, that plot is on slide 17) indicates that FreeBSD 7 not only competes strongly with current Linux performance and scalability, but that DragonFly has been beaten even by NetBSD which came late to the SMP party. Minor quibble: you're pointing at benchmarks on an 8-core xeon. Relatively uncommon hardware, though I'm sure that'll change within a year or so. At the risk of sounding like a troll, may I ask, if FreeBSD 7 has high performance, high stability and remains reasonably clean and maintainable, doesn't that partly invalidate the reasons DragonFly was created? Being cleaner and more revolutionary doesn't count for much if the product itself doesn't serve as a compelling alternative. Maybe I'm just missing the point of DragonFly's development. There were people attracted to DragonFly early on because they were mistreated by various folks involved with FreeBSD, but liking DragonFly because it's not FreeBSD is not enough to build a project. DragonFly is not a FreeBSD replacement, nor does DragonFly require FreeBSD to suck in order to exist. You could probably make the exact same argument you just did but substitute in the name of another BSD, if you are going to take the tack of fastest benchmark so why use anything else?. (I realize I'm oversimplifying your words.) DragonFly is oriented towards creating a single system image over multiple networked computers on the Internet, which has never been done before. FreeBSD - run on x86 OpenBSD - run securely NetBSD - run on anything DragonFly - run once everywhere? For myself, I find DragonFly useful because it's a tight and friendly developer community, with lots of interesting projects to do. FreeBSD having a good mysql benchmark doesn't affect that. neglected. Again, at the risk of sounding like a troll, I'm gravely concerned about the growth and survival of this brilliant project in the face of increasing pressure from projects with much larger developer communities and software ecosystems. We've been doing pretty well, really. The number of changes and new developers in 2007 have been on an upswing relative to 2006. (I should know, given my digest work.) There's been enough happening that I've had a steady 2-day buffer of things to post for a month, which has never happened before.
Re: FreeBSD 7, DragonFly's status
Dmitri Nikulin wrote: At the risk of sounding like a troll, may I ask, if FreeBSD 7 has high performance, high stability and remains reasonably clean and maintainable, doesn't that partly invalidate the reasons DragonFly was created? Matt disputes the maintainability part of FreeBSD, but that is for the FreeBSD folks to worry about. Speaking for myself, I'd much prefer to use a BSD -- I used FreeBSD and DragonFly for quite a while -- but currently use Linux despite its bloat and other issues. There is one single overriding reason why I don't use FreeBSD: removable media. It is utterly absurd that, in 2008, you can't reliably use USB memory sticks without panicking the system. If you complain on the FreeBSD lists, they'll tell you that it's your own damn fault for not unmounting it first, and not keeping it physically stable so that it doesn't accidentally jiggle, or whatever. They also say the problem runs so deep, and involves so many layers, written over the last 15 years, that it can't be fixed. Says Warner Losh: If it were easy, one of the dozen or so people that have tried to fix it in the past 8 years would have succeeded. http://lists.freebsd.org/pipermail/freebsd-stable/2007-July/036219.html On the same thread, somebody pointed out that the issue had been fixed in DragonFly, and Matt commented on what was involved. http://lists.freebsd.org/pipermail/freebsd-stable/2007-August/036469.html There was NO follow-up. Nobody in FreeBSD-land is interested in sorting out the issue. Well, that's up to them, but I need my USB media. The unmount first advice was ok in floppy days -- it is difficult to accidentally remove a floppy -- but USB is physically much less stable. It's not just USB mass storage media -- I have had problems with USB audio, USB-to-serial converters, etc. The entire USB stack is flaky, and maybe, as Warner says, the flakiness extends deep into the system. Maybe this doesn't apply so much to servers. If you really want to run FreeBSD, you can ensure nobody inserts/removes USB devices or otherwise fiddles with the system. For a desktop/laptop user it's another matter. So to me DragonFly, when it supports your hardware, seems the more stable system. But, as you say, AMD64 is low-priority, and SMP isn't done. (Matt says the infrastructure is all there but someone has to step up and fill in the gaps, ie take subsystems out of the BGL.) That excludes most newer computers, which are at least dual-core (most servers are multi-core now) and nearly always are 64-bit. I'm still keeping an eye on DragonFly, and hope to run it again one of these days. I think the biggest problem in FreeBSD that DragonFly fixes is attitude. Matt, and the others, don't brush aside problem reports saying it can't be done or that's not the true BSD way. Rahul
