I'd say it's probably both. For some reason the hardware (oldish HP DL380 G2 machines, we are on a budget:-)) work much better with Dragonfly then Debian. The network throughput is much higher with Dragonfly than with Debian and we have not touched anything in terms of throttling. Just basic vanilla install. The crawlers are written in a combination of Perl and C. Once data is fetched it is written to a MySQL cluster (also running on Drafongly) and the TPS for MySQL is also much higher (2x-3x) than it was on Debian. So that part could be due to a combination of filesystem and MySQL performance gains on Dragonfly. The flesystem for the boxes running MySQL is the default UFS. Oh, I guess forgot to mention - the MySQL cluster is running on 20 machines also running Dragonfly, so the total is about 70 system running Dragonfly. The frontend for the site is Ubuntu Linus b/c our developers are Ruby guys and they are comfortable with Linux+Ruby environment. I am trying to convince everybody to go Dragonfly-only for the whole setup but there is some pushback for the frontend given that the systems are all multi-cores and Dragonfly currently lags a bit behind Linux in SMP support. The crawler and DB cluster will gradually gros in size so I can keep you updated on the count if you need it. Also, I will keep track of performance changes once we switch to HammerFS. Thanks for good work guys, Dragonfly rocks!
On Fri, Jun 19, 2009 at 12:18 PM, Matthew Dillon < [email protected]> wrote: > > :Hi all, > : > :Just wanted to let you know that we have deployed in production a cluster > of > :about 50 single-CPU machines running Dragonfly. That's probably > Dragonfly's > :largest production install base at a single location:-) The machines are > :used as the crawler layer for a search engine. We are not using hammer yet > :but plan on transition to it in the next 6-12 months. In case you want to > :find out more about the project please visit: > :http://www.euraeka.com/faq > : > :We are very happy with the way Dragonfly handles the load. Our previous > :crawler cluster ran Debian Linux and the crawling speed was horrible. > After > :switching to Dragonfly, we gained 5x-6x in crawling speed. > :I am just forwarding this information in case there are plans to set up a > :page "Sites using Dragonfly" like the ones on the FreeBSD and NetBSD > :projects. > :Thanks. > > Hey, that's cool! Where did the primary increase in performance > come from? Networking? Filesystem? > > -Matt > Matthew Dillon > <[email protected]> >
