[Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
Hello,
Can anybody help me to understand why, on simple performance tests, I see so
different results between .NET and Mono on various Linux platforms? Are my
tests incorrect? Did I have no luck and simply fall on some
methods/functionality better implemented by MS? And why is there such a
difference between Suse and Debian for the IPAddress dictionary?
Any help appreciated (please, no trolling, no political debate - just ideas
about how to improve things or ideas to point out methodology errors I would
have made in my tests)
Regards,
Lionel
A few hints on how I did my tests:
- the OS are: a bare XP SP3, a bare OpenSuse (kernel 2.6.27), a Debian with
a lightened kernel (2.6.30) - bare = the OS is up-to-date but no other
software installed
- all OS were running as vmware guest (in vm workstation 6.5) with two
cpus (dual-core T7200) and 512Mo of RAM
- Mono version was 2.4.2.3 on Suse, and 2.5 (compiled from SVN r140059) on
Debian. .NET version is 3.5
- On Suse and Debian, I both run the tests with the exe compiled with csc
(.net) and with the exe compiled with gmcs - the results were the same.
You'll find attached the sample code I used - I test basically a dictionary
of int, a dictionary of IPAddress and the IPAddress.ToString method. The
duration of the test (in millisec are:
int32 Dictionary test
4812 DotNet
9493 Mono on light Debian
9815 Mono on Suse
IPAddress Dictionary test
13500 DotNet
29533 Mono on light Debian
41732 Mono on Suse
IPAddress.ToString test
2453 DotNet
16665 Mono on light Debian
17747 Mono on Suse
using System;
using System.Collections.Generic;
using System.Net;
namespace Test
{
public class Class1
{
static void Main(string[] args)
{
TestPerfOnDico();
TestPerfOnIpAddress();
}
static void TestPerfOnIpAddress()
{
Console.WriteLine(IPAddress.ToString test);
long sum = 0;
int count = (int)Math.Pow(10, 7);
Console.Write(Iterations (def={0:0,0})? , count);
string sChoice = Console.ReadLine();
if (!string.IsNullOrEmpty(sChoice))
count = int.Parse(sChoice);
IPAddress ip;
for (int i = 0; i 100; i++)
{
ip = new IPAddress(i);
sum += ip.ToString().Length;
}
Console.WriteLine(Dummy: + sum);
sum = 0;
DateTime start = DateTime.UtcNow;
for (int i = 0; i count; i++)
{
ip = new IPAddress(i);
sum += ip.ToString().Length;
}
TimeSpan duration = DateTime.UtcNow.Subtract(start);
Console.WriteLine(Total: + sum);
Console.WriteLine(Duration: +
duration.TotalMilliseconds);
}
static void TestPerfOnDico()
{
int count = (int)Math.Pow(10, 8);
Console.Write(Iterations (def={0:0,0})? , count);
string sChoice = Console.ReadLine();
if (!string.IsNullOrEmpty(sChoice))
count = int.Parse(sChoice);
Console.WriteLine(int32 Dictionary test);
DateTime start = DateTime.UtcNow;
int intDicoSizeMax = 1;
Dictionaryint, int dicoInt = new Dictionaryint,
int();
for (int i = 0; i count; i++)
{
int key = (int)long)i) * 5 + 11) %
intDicoSizeMax);
dicoInt[key] = i;
}
TimeSpan duration = DateTime.UtcNow.Subtract(start);
Console.WriteLine(Duration for int dico (in millisec):
+ duration.TotalMilliseconds);
Console.WriteLine(IPAddress Dictionary test);
start = DateTime.UtcNow;
int iPAddressDicoSizeMax = 1;
DictionaryIPAddress, IPAddress dicoIP = new
DictionaryIPAddress, IPAddress();
for (int i = 0; i count; i++)
{
long key = long)i) * 5 + 11) %
iPAddressDicoSizeMax);
IPAddress ip = new IPAddress(key);
dicoIP[ip] = ip;
}
duration = DateTime.UtcNow.Subtract(start);
Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
On Tue, Oct 6, 2009 at 12:15 PM, Robert Jordan robe...@gmx.net wrote: Lionel Cuir wrote: A few hints on how I did my tests: - the OS are: a bare XP SP3, a bare OpenSuse (kernel 2.6.27), a Debian with a lightened kernel (2.6.30) - bare = the OS is up-to-date but no other software installed - all OS were running as vmware guest (in vm workstation 6.5) with two cpus (dual-core T7200) and 512Mo of RAM - Mono version was 2.4.2.3 on Suse, and 2.5 (compiled from SVN r140059) on Debian. .NET version is 3.5 - On Suse and Debian, I both run the tests with the exe compiled with csc (.net) and with the exe compiled with gmcs - the results were the same. You're benching different Mono versions. Mono 2.4 was branched in January, whereas r140059 is pretty up-to-date. Besides, while measuring DictionaryT, K performance has some relation to actual program performance, I doubt whether anyone will call IPAdress.ToString() often enough to even notice the speed difference... ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
What is the background noise on the CPU load in the VMWare environments? Are all the CPUs running at the same spare capcity, and the same/similar CPU utilisation during the tests, or are tasks running in the background, system updates etc.. on the Linuxes that's eating resources? Cheers, Euan. Date: Tue, 6 Oct 2009 12:27:12 +0200 From: skol...@gmail.com To: mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ? On Tue, Oct 6, 2009 at 12:15 PM, Robert Jordan robe...@gmx.net wrote: Lionel Cuir wrote: A few hints on how I did my tests: - the OS are: a bare XP SP3, a bare OpenSuse (kernel 2.6.27), a Debian with a lightened kernel (2.6.30) - bare = the OS is up-to-date but no other software installed - all OS were running as vmware guest (in vm workstation 6.5) with two cpus (dual-core T7200) and 512Mo of RAM - Mono version was 2.4.2.3 on Suse, and 2.5 (compiled from SVN r140059) on Debian. .NET version is 3.5 - On Suse and Debian, I both run the tests with the exe compiled with csc (.net) and with the exe compiled with gmcs - the results were the same. You're benching different Mono versions. Mono 2.4 was branched in January, whereas r140059 is pretty up-to-date. Besides, while measuring DictionaryT, K performance has some relation to actual program performance, I doubt whether anyone will call IPAdress.ToString() often enough to even notice the speed difference... ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
On Tue, Oct 6, 2009 at 3:27 AM, Leszek Ciesielski skol...@gmail.com wrote: Besides, while measuring DictionaryT, K performance has some relation to actual program performance, I doubt whether anyone will call IPAdress.ToString() often enough to even notice the speed difference... True, but a 6 - 7 X slow down is so large that it's interesting to ask why. Dictionary's 2 X slow down is significant. I still boot up Windows in a VM on some occasions just to develop and run CLR code in half the time. Run-time performance can affect the development cycle as you loop through: 1. Edit code 2. Build 3. Run 4. Observe results, contemplate changes 5. Go To 1 (Oh no, a GOTO statement!) It would be nice if the performance gap was smaller. Same with the C# compilers--Novell vs. MS. Re: Euan's concerns about background processes, I have found Mono on Mac to be *consistently* slower than .NET on Windows. In fact, I run Windows in a VM which introduces some I/O overhead, and it still runs substantially faster, every time. -Chuck ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
Calling 'ToString' is just a GC benchmark. Yes, the current mono GC is slower than the .NET GC in this scenario. Your other tests also do a fair bit of allocating, so it's quite possible the entire performance loss you're seeing is just from the GC kicking in. If you want to test how fast you can add elements to a Dictionary, pre-allocate all the objects first and then time just the addition. Alan. On Tue, Oct 6, 2009 at 10:15 AM, Lionel Cuir lionel_em...@aulofee.comwrote: Hello, Can anybody help me to understand why, on simple performance tests, I see so different results between .NET and Mono on various Linux platforms? Are my tests incorrect? Did I have no luck and simply fall on some methods/functionality better implemented by MS? And why is there such a difference between Suse and Debian for the IPAddress dictionary? Any help appreciated (please, no trolling, no political debate - just ideas about how to improve things or ideas to point out methodology errors I would have made in my tests) Regards, Lionel A few hints on how I did my tests: - the OS are: a bare XP SP3, a bare OpenSuse (kernel 2.6.27), a Debian with a lightened kernel (2.6.30) - bare = the OS is up-to-date but no other software installed - all OS were running as vmware guest (in vm workstation 6.5) with two cpus (dual-core T7200) and 512Mo of RAM - Mono version was 2.4.2.3 on Suse, and 2.5 (compiled from SVN r140059) on Debian. .NET version is 3.5 - On Suse and Debian, I both run the tests with the exe compiled with csc (.net) and with the exe compiled with gmcs - the results were the same. You'll find attached the sample code I used - I test basically adictionary of int, a dictionary of IPAddress and the IPAddress.ToString method. The duration of the test (in millisec are: int32 Dictionary test 4812 DotNet 9493 Mono on light Debian 9815 Mono on Suse IPAddress Dictionary test 13500 DotNet 29533 Mono on light Debian 41732 Mono on Suse IPAddress.ToString test 2453 DotNet 16665 Mono on light Debian 17747 Mono on Suse ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
Hi, Can anybody help me to understand why, on simple performance tests, I see so different results between .NET and Mono on various Linux platforms? Well, your tests are probably not very real-world, but the answer is quite simple. Mono has a horrible slow GC, and its code-generator/jit is ok but not exceptional good. Its simply a matter of money - MS has a lot of resources dedicated to its CLR, whereas Mono is a smaller project. The situation reminds me a lot to the java situation, before SUN open-sourced their runtime. They worked, and some quite well, but SUN's JVM (especially the server runtime) was most of the time a lot faster - especially for large systems. Gladly, Sun open-sourced java, so you now get the source of their well-tuned VM under GPL :) - Clemens ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
Yes, the testing conditions were the same for all virtual machines (xp, suse, debian): nothing else was running (and only the base service/daemon - no web server, no database and so on: just the bare OS). Besides, I ran the tests several times and the figures were stable. _ De : mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] De la part de Euan MacInnes Envoyé : mardi 6 octobre 2009 12:32 À : mono-devel-list@lists.ximian.com Objet : Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ? What is the background noise on the CPU load in the VMWare environments? Are all the CPUs running at the same spare capcity, and the same/similar CPU utilisation during the tests, or are tasks running in the background, system updates etc.. on the Linuxes that's eating resources? Cheers, Euan. _ Date: Tue, 6 Oct 2009 12:27:12 +0200 From: skol...@gmail.com To: mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ? On Tue, Oct 6, 2009 at 12:15 PM, Robert Jordan robe...@gmx.net wrote: Lionel Cuir wrote: A few hints on how I did my tests: - the OS are: a bare XP SP3, a bare OpenSuse (kernel 2.6.27), a Debian with a lightened kernel (2.6.30) - bare = the OS is up-to-date but no other software installed - all OS were running as vmware guest (in vm workstation 6.5) with two cpus (dual-core T7200) and 512Mo of RAM - Mono version was 2.4.2.3 on Suse, and 2.5 (compiled from SVN r140059) on Debian. .NET version is 3.5 - On Suse and Debian, I both run the tests with the exe compiled with csc (.net) and with the exe compiled with gmcs - the results were the same. You're benching different Mono versions. Mono 2.4 was branched in January, whereas r140059 is pretty up-to-date. Besides, while measuring DictionaryT, K performance has some relation to actual program performance, I doubt whether anyone will call IPAdress.ToString() often enough to even notice the speed difference... ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
Thanks for the answers
Interesting remarks. I voluntarily added the allocation of objects within
the mesurable time, to have an overall picture (and I was not aware of the
mono GC's relative slowness).
I modified my tests as you advised but the results still show a big
difference between .net and mono. Could it be that the measuring window
still include GC work? Could you have a look at my code (see attached file)
to see if we agree on what should be preallocated in order to put apart the
GC? Apart for the int32 preallocation (I don't know if preallocating value
type can be meaningful), the code is supposed to preallocate both
dictionaries + the IPAddresses.
Here are the results with preallocation (again, what's matter is not the
exact value but the magnitude):
Preallocated int32 Dictionary test
4937 DotNet
9856 Mono on light Debian
9554 Mono on Suse
Preallocated IPAddress Dictionary test
8375 DotNet
16414 Mono on light Debian
17084 Mono on Suse
Lionel
_
De : Alan McGovern [mailto:alan.mcgov...@gmail.com]
Envoyé : mardi 6 octobre 2009 12:46
À : Lionel Cuir
Cc : mono-devel-list@lists.ximian.com
Objet : Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian)
?
Calling 'ToString' is just a GC benchmark. Yes, the current mono GC is
slower than the .NET GC in this scenario. Your other tests also do a fair
bit of allocating, so it's quite possible the entire performance loss you're
seeing is just from the GC kicking in. If you want to test how fast you can
add elements to a Dictionary, pre-allocate all the objects first and then
time just the addition.
Alan.
On Tue, Oct 6, 2009 at 10:15 AM, Lionel Cuir lionel_em...@aulofee.com
wrote:
Hello,
Can anybody help me to understand why, on simple performance tests, I see so
different results between .NET and Mono on various Linux platforms? Are my
tests incorrect? Did I have no luck and simply fall on some
methods/functionality better implemented by MS? And why is there such a
difference between Suse and Debian for the IPAddress dictionary?
Any help appreciated (please, no trolling, no political debate - just ideas
about how to improve things or ideas to point out methodology errors I would
have made in my tests)
Regards,
Lionel
A few hints on how I did my tests:
- the OS are: a bare XP SP3, a bare OpenSuse (kernel 2.6.27), a Debian with
a lightened kernel (2.6.30) - bare = the OS is up-to-date but no other
software installed
- all OS were running as vmware guest (in vm workstation 6.5) with two
cpus (dual-core T7200) and 512Mo of RAM
- Mono version was 2.4.2.3 on Suse, and 2.5 (compiled from SVN r140059) on
Debian. .NET version is 3.5
- On Suse and Debian, I both run the tests with the exe compiled with csc
(.net) and with the exe compiled with gmcs - the results were the same.
You'll find attached the sample code I used - I test basically a dictionary
of int, a dictionary of IPAddress and the IPAddress.ToString method. The
duration of the test (in millisec are:
int32 Dictionary test
4812 DotNet
9493 Mono on light Debian
9815 Mono on Suse
IPAddress Dictionary test
13500 DotNet
29533 Mono on light Debian
41732 Mono on Suse
IPAddress.ToString test
2453 DotNet
16665 Mono on light Debian
17747 Mono on Suse
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
using System;
using System.Collections.Generic;
using System.Net;
namespace Test
{
public class Class1
{
static void Main(string[] args)
{
TestPerfOnPreAllocatedDico();
}
static void TestPerfOnPreAllocatedDico()
{
int count = (int)Math.Pow(10, 8);
Console.Write(Iterations (def={0:0,0})? , count);
string sChoice = Console.ReadLine();
if (!string.IsNullOrEmpty(sChoice))
count = int.Parse(sChoice);
Console.WriteLine(Preallocated int32 Dictionary test);
// Let's preallocate all our objects first. Is
preallocation possible/meaningful for value types (the int32)?
int intDicoSizeMax = 1;
Dictionaryint, int dicoInt = new Dictionaryint,
int(intDicoSizeMax);
// Now we run the test.
DateTime start = DateTime.UtcNow;
for (int i = 0; i count; i++)
{
int key = (int)long)i) * 5 + 11) %
intDicoSizeMax);
dicoInt[key] = i;
}
TimeSpan duration = DateTime.UtcNow.Subtract(start);
Console.WriteLine(Duration for Preallocated int dico
(in millisec
Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
A web search showed me that GC can be turned off with the environment variable GC_DONT_GC. Maybe that should be called MONO_GC_ON. In any case, any value will turn off the GC, including blank, so to turn it back on, unset it. I tested with a command line Mono program that is not a microbenchmark. With GC, it peaks at around 180MB and without, it peaks around 840MB and runs about 18% faster. I gave the same program less work to do and its performance boost was only 10%. So now you can try your programs without GC to see how much it contributes to performance drag. Just watch for swap. :-) Does anyone know what's involved with plugging in your own GC into Mono? -Chuck ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Big performance gaps between .NET, Mono (Suse/Debian) ?
Hi, Chuck Esterbrook wrote: A web search showed me that GC can be turned off with the environment variable GC_DONT_GC. Maybe that should be called MONO_GC_ON. In any case, any value will turn off the GC, including blank, so to turn it back on, unset it. I tested with a command line Mono program that is not a microbenchmark. With GC, it peaks at around 180MB and without, it peaks around 840MB and runs about 18% faster. I gave the same program less work to do and its performance boost was only 10%. So now you can try your programs without GC to see how much it contributes to performance drag. Just watch for swap. :-) Does anyone know what's involved with plugging in your own GC into Mono? Nightmare level?? -Chuck ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
