Re: [Flightgear-devel] Cygwin slowness
Matthias Fröhlich wrote: > I recently profiled flightgear with gprof and with callgrind. > One of the functions most cpu intensive function under Linux (fedora core 3) > is the isspace() function in split. > That might be even worse with the windows implementation of isspace. Well, at this point it's pretty clear that there is a cygwin performance issue -- after the flame war died out, they started actually working on it and traced it down to an interaction between the signal handler emulation code, pthreads, and C++ exceptions. Or so I understand it, anyway. My intuition about a synchronization bug seems to have been correct, but it's not in the allocator; there's a lot more library interaction going on behind the scenes in cygwin than I realized. But yes, we ended up zeroing in on the loading of apt.dat.gz because that step *is* a big chunk of initialization. Even on my machine, which starts in 10 seconds, this takes 40% of the initialization time. The reason you're seeing std::vector and isspace() show up in the profile isn't really that these functions are slow, it's that FlightGear is using them on every character/field of a 160k line file. Andy ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
On Freitag 27 Mai 2005 22:14, Andy Ross wrote: > Even weirder: comment out the split() call, which is 100% CPU-bound > (let me say that again: split() does no I/O and makes no direct calls > to the OS kernel), and the runtime goes down to 1.3 seconds. Huh? I recently profiled flightgear with gprof and with callgrind. One of the functions most cpu intensive function under Linux (fedora core 3) is the isspace() function in split. That might be even worse with the windows implementation of isspace. I am at the moment not shure, but I can imagine that this function needs to be locale aware like for example the isupper() function definitly is. May be this one is a starting point. Also I can well imagine that allocating that string each time we read some few bytes might be time consuming. By just calling reserve(some number) for the vector and/or for the strings appearing here might reduce the allocation/copying overhead a bit. At least some internal routine from std::vector used to insert a new element shows up in the -pg profile of split. While this will most propably not affect the windows problems, it is still a heavyweight function at startup time. Greetings Mathias -- Mathias Fröhlich, email: [EMAIL PROTECTED] ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
Just out of WAGing curiousity, what happens when you run cygwin "--enable-dmalloc" ? Does it have any effect on the split time? I'm still new at this... so forgive me for any dumbass comments/ideas please - Original Message - From: Andy Ross To: FlightGear developers discussions Sent: Friday, May 27, 2005 4:14 PM Subject: [Flightgear-devel] Cygwin slowness Well, there's some progress. I have a cygwin build, and it's everybit at slow as Vivian says it is. And I've kinda/sorta isolated theproblem. Here's a program: // g++ -o aptdat aptdat.cc -I$FG/SimGear -L$FG/lib -lsgmisc -lz #include #include const int bufsz = 2048; char buf[bufsz]; int main(int argc, char** argv) { sg_gzifstream f(argv[1]); vector toks; while (!f.eof()) { f.getline(buf, bufsz); toks = simgear::strutils::split(buf); } }This basically duplicates the read loop in the apt.dat.gz loader. Itruns in 2 seconds on my linux box, but takes 40 (!) seconds on thewindows/cygwin machine. Bizarre.Even weirder: comment out the split() call, which is 100% CPU-bound(let me say that again: split() does no I/O and makes no direct callsto the OS kernel), and the runtime goes down to 1.3 seconds. Huh?Here's one more piece of evidence: you can watch this program instrace, which has a nice microseconds-since-the-last-call field. Itbasically opens up the file and starts reading 1024 bytes at a time.Most of these calls take about 10-100us per 1k chunk. But every 10 orso chunks (and at very regular intervals -- keep this in mind, it'simportant), one of the readv() calls takes something like 8-10*thousand* microsecondsNote to OS junkies: Quick, before reading further, do you seenanything special about time values in that range?So here's the hypothesis: the split routine does no I/O, but it *does*need to allocate memory. The heap needs to be threadsafe, obviously,so some heap operations need to be synchronized. What if after everyN allocations (which are, in this context, very regular with respectto the number of bytes read from the file), the heap needs to dosomething that requires a lock. And what if the synchronizationmechanism requires a spinlock or timeout? (Which is sadly necessaryin windows, as the win32 API has no condition variables). Well, thenthe process will need to wait for the next CPU timeslice to run. Andthe CPU timeslice on WinXP just happens to be (I think) 100 Hz. Hm...The bottom line is that this is a cygwin bug and we can't fix it,sorry. Hopefully someone with more love for this platform than I(I've done my time, heh) can forward this to them and get it fixed.It would need to eliminate the (red herring) gzip ifstream interfacein favor of plain old fgets(), obviously, and should include the splitcode inline. But the performance problem is clear as day: it's 20xslower than exactly the same program run under linux.I have to wonder how many other "cygwin is slow!" reports (not justFlightGear's) are related to this issue...Andy___Flightgear-devel mailing listFlightgear-devel@flightgear.orghttp://mail.flightgear.org/mailman/listinfo/flightgear-devel2f585eeea02e2c79d7b1d8c4963bae2d ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
'This basically duplicates the read loop in the apt.dat.gz loader. Itruns in 2 seconds on my linux box, but takes 40 (!) seconds on thewindows/cygwin machine. Bizarre.' The Zen Master of Algorithm Analysis would say it probably isn't so bizarre :-) - Original Message - From: Andy Ross To: FlightGear developers discussions Sent: Friday, May 27, 2005 4:14 PM Subject: [Flightgear-devel] Cygwin slowness Well, there's some progress. I have a cygwin build, and it's everybit at slow as Vivian says it is. And I've kinda/sorta isolated theproblem. Here's a program: // g++ -o aptdat aptdat.cc -I$FG/SimGear -L$FG/lib -lsgmisc -lz #include #include const int bufsz = 2048; char buf[bufsz]; int main(int argc, char** argv) { sg_gzifstream f(argv[1]); vector toks; while (!f.eof()) { f.getline(buf, bufsz); toks = simgear::strutils::split(buf); } }This basically duplicates the read loop in the apt.dat.gz loader. Itruns in 2 seconds on my linux box, but takes 40 (!) seconds on thewindows/cygwin machine. Bizarre.Even weirder: comment out the split() call, which is 100% CPU-bound(let me say that again: split() does no I/O and makes no direct callsto the OS kernel), and the runtime goes down to 1.3 seconds. Huh?Here's one more piece of evidence: you can watch this program instrace, which has a nice microseconds-since-the-last-call field. Itbasically opens up the file and starts reading 1024 bytes at a time.Most of these calls take about 10-100us per 1k chunk. But every 10 orso chunks (and at very regular intervals -- keep this in mind, it'simportant), one of the readv() calls takes something like 8-10*thousand* microsecondsNote to OS junkies: Quick, before reading further, do you seenanything special about time values in that range?So here's the hypothesis: the split routine does no I/O, but it *does*need to allocate memory. The heap needs to be threadsafe, obviously,so some heap operations need to be synchronized. What if after everyN allocations (which are, in this context, very regular with respectto the number of bytes read from the file), the heap needs to dosomething that requires a lock. And what if the synchronizationmechanism requires a spinlock or timeout? (Which is sadly necessaryin windows, as the win32 API has no condition variables). Well, thenthe process will need to wait for the next CPU timeslice to run. Andthe CPU timeslice on WinXP just happens to be (I think) 100 Hz. Hm...The bottom line is that this is a cygwin bug and we can't fix it,sorry. Hopefully someone with more love for this platform than I(I've done my time, heh) can forward this to them and get it fixed.It would need to eliminate the (red herring) gzip ifstream interfacein favor of plain old fgets(), obviously, and should include the splitcode inline. But the performance problem is clear as day: it's 20xslower than exactly the same program run under linux.I have to wonder how many other "cygwin is slow!" reports (not justFlightGear's) are related to this issue...Andy___Flightgear-devel mailing listFlightgear-devel@flightgear.orghttp://mail.flightgear.org/mailman/listinfo/flightgear-devel2f585eeea02e2c79d7b1d8c4963bae2d ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
RE: [Flightgear-devel] Cygwin slowness
Vivian Meazza writes: > > Andy Ross wrote: > > > > > Erik Hofman wrote: > > > Norman Vine wrote: > > > > FWIW I think these apply here > > > > http://www.catb.org/~esr/faqs/smart-questions.html#id3001405 > > > > http://www.catb.org/~esr/faqs/smart-questions.html#keepcool > > > > > > These two contradict (You can't offend them but they can offend you). > > > It's a nice document on how to approach a three year old though. > > > > It's actually a pretty good guide. The problem is I don't know how I > > could have followed it any better. I mean, I *am* the hacker type ESR > > is talking about. > > > > The was, IMHO, a fantasy bug report: > > > > + A symptom so painfully obvious that no technical knowlege is > > required to see it (1.6 seconds vs. 26 seconds run time) > > + Simple example code > > + Easily reproduced within a few commands > > + Requires no external dependencies (just gcc and the mingw libraries) > > + And a real world use case (us!) for why it's important that it be > > fixed. > > > > To turn it around: could you imaging me responding to this bug report: > > > > "Here's a simple Nasal script that reduces my FPS to 2-3." > > > > with: > > > > "If it is really important to you, you should try to fix it rather > >than posting here and trying to get lucky." (pretty much exactly > >Chris's words). > > > > I mean, sometimes I'm lazy or forget stuff, but I'm generally pretty > > good about admitting when a bug is a bug. That's my *job* as a > > developer. Who does that job for cygwin? > > > > Well, that all generated more heat than light! I almost wish I hadn't set > the hare running. > > I have never seen anyone react quite like this Faylor chap, from which I > suppose: > > A. He already knew he has a problem with Cygwin's speed. > > B. He doesn't know how to fix it. > > C. We're unlikely to see it sorted any time soon. > > I'm more than willing to be proven wrong. > > Meanwhile, thanks to Andy for daring to enter the lion's den :-) Cygwin is a bit of a lions den in that it has a *huge* user base. Again I point folks to http://cygwin.com/problems.html And FWIW I think y'all are selling Chris short :-) http://cygwin.com/ml/cygwin/2005-05/msg01329.html I know it is easy to snipe off list but really :-) Norman ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
RE: [Flightgear-devel] Cygwin slowness
Andy Ross wrote: > > Erik Hofman wrote: > > Norman Vine wrote: > > > FWIW I think these apply here > > > http://www.catb.org/~esr/faqs/smart-questions.html#id3001405 > > > http://www.catb.org/~esr/faqs/smart-questions.html#keepcool > > > > These two contradict (You can't offend them but they can offend you). > > It's a nice document on how to approach a three year old though. > > It's actually a pretty good guide. The problem is I don't know how I > could have followed it any better. I mean, I *am* the hacker type ESR > is talking about. > > The was, IMHO, a fantasy bug report: > > + A symptom so painfully obvious that no technical knowlege is > required to see it (1.6 seconds vs. 26 seconds run time) > + Simple example code > + Easily reproduced within a few commands > + Requires no external dependencies (just gcc and the mingw libraries) > + And a real world use case (us!) for why it's important that it be > fixed. > > To turn it around: could you imaging me responding to this bug report: > > "Here's a simple Nasal script that reduces my FPS to 2-3." > > with: > > "If it is really important to you, you should try to fix it rather >than posting here and trying to get lucky." (pretty much exactly >Chris's words). > > I mean, sometimes I'm lazy or forget stuff, but I'm generally pretty > good about admitting when a bug is a bug. That's my *job* as a > developer. Who does that job for cygwin? > Well, that all generated more heat than light! I almost wish I hadn't set the hare running. I have never seen anyone react quite like this Faylor chap, from which I suppose: A. He already knew he has a problem with Cygwin's speed. B. He doesn't know how to fix it. C. We're unlikely to see it sorted any time soon. I'm more than willing to be proven wrong. Meanwhile, thanks to Andy for daring to enter the lion's den :-) Regards, Vivian ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
Erik Hofman wrote: > Norman Vine wrote: > > FWIW I think these apply here > > http://www.catb.org/~esr/faqs/smart-questions.html#id3001405 > > http://www.catb.org/~esr/faqs/smart-questions.html#keepcool > > These two contradict (You can't offend them but they can offend you). > It's a nice document on how to approach a three year old though. It's actually a pretty good guide. The problem is I don't know how I could have followed it any better. I mean, I *am* the hacker type ESR is talking about. The was, IMHO, a fantasy bug report: + A symptom so painfully obvious that no technical knowlege is required to see it (1.6 seconds vs. 26 seconds run time) + Simple example code + Easily reproduced within a few commands + Requires no external dependencies (just gcc and the mingw libraries) + And a real world use case (us!) for why it's important that it be fixed. To turn it around: could you imaging me responding to this bug report: "Here's a simple Nasal script that reduces my FPS to 2-3." with: "If it is really important to you, you should try to fix it rather than posting here and trying to get lucky." (pretty much exactly Chris's words). I mean, sometimes I'm lazy or forget stuff, but I'm generally pretty good about admitting when a bug is a bug. That's my *job* as a developer. Who does that job for cygwin? Andy ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
Norman Vine wrote: FWIW I think these apply here http://www.catb.org/~esr/faqs/smart-questions.html#id3001405 http://www.catb.org/~esr/faqs/smart-questions.html#keepcool These two contradict (You can't offend them but they can offend you). It's a nice document on how to approach a three year old though. Erik ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
RE: [Flightgear-devel] Cygwin slowness
Erik Hofman writes: > > bass pumped wrote: > >>Well, I submitted it. Alas, it didn't go well. You can follow the > >>flame war here: > >> > >> http://cygwin.com/ml/cygwin/2005-05/threads.html#01305 > > > > I just read that... that guy certainly has a problem!!! > > Tell me about it, I've worked with such an individual for more than five > years. If they don't think there's a problem then everybody else has. > > I can only say; don't use cygwin as long as this person is involved. With due respect Cygwin is a *high* volume list with well documented protocols with prominent links on its home page for submitting problems. http://cygwin.com/problems.html FWIW I think these apply here http://www.catb.org/~esr/faqs/smart-questions.html#id3001405 http://www.catb.org/~esr/faqs/smart-questions.html#keepcool I suggest you see the follow ups to this http://cygwin.com/ml/cygwin/2005-05/msg01317.html rather then continue an uniformed off list flame Then again we can have another GlutFest or Or PlibBash which is bound to be productive :-) Norman ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
bass pumped wrote: Well, I submitted it. Alas, it didn't go well. You can follow the flame war here: http://cygwin.com/ml/cygwin/2005-05/threads.html#01305 I just read that... that guy certainly has a problem!!! Tell me about it, I've worked with such an individual for more than five years. If they don't think there's a problem then everybody else has. I can only say; don't use cygwin as long as this person is involved. Erik ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
Andy Ross a écrit : // // VERY bad performance characteristics when linked against // cygwin.dll. Watch under strace, and note that every N reads, there // is an inexplicable delay of several milliseconds. Spinning in the // malloc implementation? The Debug build of MSVC has the exact same behavior : In that case, it is memory heap check that clobber performance, and not system related, but located in the C runtime library. Some numbers : it takes 5 seconds to load airport data in Release mode, and 25s ( 5x more ) in Debug mode. -Fred ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
> Well, I submitted it. Alas, it didn't go well. You can follow the > flame war here: > > http://cygwin.com/ml/cygwin/2005-05/threads.html#01305 I just read that... that guy certainly has a problem!!! ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
I wrote: > No, that's dumb. I may not like windows, but cygwin is one of the few > things that makes it bearable. :) > [...] > I'll try to dig up a cygwin mailing list to which to submit this. Well, I submitted it. Alas, it didn't go well. You can follow the flame war here: http://cygwin.com/ml/cygwin/2005-05/threads.html#01305 If this was a project that I used on a daily basis, I might push harder or work more on a solution. But sorry, I'm just not up for debugging the Cygwin DLL on (gack) windows. I just don't swing that way. Anyway, I've done my good deed. If anyone more, er, politic can pummel some sense into Chris Faylor, feel free. The test code is in the public record, at least. Andy ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
Re: [Flightgear-devel] Cygwin slowness
I wrote: > The bottom line is that this is a cygwin bug and we can't fix it, > sorry. Hopefully someone with more love for this platform than I > (I've done my time, heh) can forward this to them and get it fixed. No, that's dumb. I may not like windows, but cygwin is one of the few things that makes it bearable. :) Attached is a SimGear-less version of the test file with the same performance characteristics. Happily, the problem can be exhibited against a mingw32 build with the same compiler, so you don't even have to reboot to see the awful breakage. :) I'll try to dig up a cygwin mailing list to which to submit this. Andy #include #include #include // Reads and splits the input file on whitespace. Sample input file: // // perl -e 'for($i=0; $i<16; $i++) { print "A " x 12, "\n" }' \ //> cygspd.dat // // $ g++ -O2 -o cygspd cygspd.cc // $ time ./cygspd cygspd.dat // // real0m26.545s // user0m26.468s // sys 0m0.031s // // g++ -mno-cygwin -O2 -o cygspd cygspd.cc // time ./cygspd cygspd.dat // // real0m1.689s // user0m0.015s // sys 0m0.015s // // VERY bad performance characteristics when linked against // cygwin.dll. Watch under strace, and note that every N reads, there // is an inexplicable delay of several milliseconds. Spinning in the // malloc implementation? using namespace std; vector split(const string& str) { vector result; string::size_type i=0, j, len=str.length(); while(i < len) { while(i < len && isspace(str[i])) ++i; j = i; while (i < len && !isspace(str[i])) ++i; if(j < i) { result.push_back(str.substr(j, i-j)); while(i < len && isspace(str[i])) ++i; } } return result; } #define BUFSZ 2048 char buf[BUFSZ]; int main(int argc, char** argv) { FILE* input = argc > 1 ? fopen(argv[1], "r") : stdin; vector toks; while(fgets(buf, BUFSZ, input)) toks = split(buf); } ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d
RE: [Flightgear-devel] Cygwin slowness
Andy Ross wrote: > Well, there's some progress. I have a cygwin build, and it's every > bit at slow as Vivian says it is. And I've kinda/sorta isolated the > problem. Here's a program: > >// g++ -o aptdat aptdat.cc -I$FG/SimGear -L$FG/lib -lsgmisc -lz >#include >#include >const int bufsz = 2048; >char buf[bufsz]; >int main(int argc, char** argv) >{ >sg_gzifstream f(argv[1]); >vector toks; >while (!f.eof()) { > f.getline(buf, bufsz); > toks = simgear::strutils::split(buf); >} >} > > This basically duplicates the read loop in the apt.dat.gz loader. It > runs in 2 seconds on my linux box, but takes 40 (!) seconds on the > windows/cygwin machine. Bizarre. > > Even weirder: comment out the split() call, which is 100% CPU-bound > (let me say that again: split() does no I/O and makes no direct calls > to the OS kernel), and the runtime goes down to 1.3 seconds. Huh? > > Here's one more piece of evidence: you can watch this program in > strace, which has a nice microseconds-since-the-last-call field. It > basically opens up the file and starts reading 1024 bytes at a time. > Most of these calls take about 10-100us per 1k chunk. But every 10 or > so chunks (and at very regular intervals -- keep this in mind, it's > important), one of the readv() calls takes something like 8-10 > *thousand* microseconds > > Note to OS junkies: Quick, before reading further, do you seen > anything special about time values in that range? > > So here's the hypothesis: the split routine does no I/O, but it *does* > need to allocate memory. The heap needs to be threadsafe, obviously, > so some heap operations need to be synchronized. What if after every > N allocations (which are, in this context, very regular with respect > to the number of bytes read from the file), the heap needs to do > something that requires a lock. And what if the synchronization > mechanism requires a spinlock or timeout? (Which is sadly necessary > in windows, as the win32 API has no condition variables). Well, then > the process will need to wait for the next CPU timeslice to run. And > the CPU timeslice on WinXP just happens to be (I think) 100 Hz. Hm... > > The bottom line is that this is a cygwin bug and we can't fix it, > sorry. Hopefully someone with more love for this platform than I > (I've done my time, heh) can forward this to them and get it fixed. > It would need to eliminate the (red herring) gzip ifstream interface > in favor of plain old fgets(), obviously, and should include the split > code inline. But the performance problem is clear as day: it's 20x > slower than exactly the same program run under linux. > > I have to wonder how many other "cygwin is slow!" reports (not just > FlightGear's) are related to this issue... > Most informative - now we know! I'd already seen the program laboriously loading 1 Gb of data into paged memory small chunks at a time. Interestingly, only one of the CPUs seems to be involved in this process: the other seems to idle. I suppose the solution is simple - cross-compile into a Windows binary. That said - a Windows binary while obviously faster, still appears significantly slower then Linux. I'll do a bit more digging around. Thanks, Vivian ___ Flightgear-devel mailing list Flightgear-devel@flightgear.org http://mail.flightgear.org/mailman/listinfo/flightgear-devel 2f585eeea02e2c79d7b1d8c4963bae2d