Re: [HACKERS] small parallel restore optimization

2009-03-09 Thread Magnus Hagander
Alvaro Herrera wrote: Tom Lane wrote: Worst case, we could say that parallel restore isn't supported on mingw. I'm not entirely sure why we bother with that platform at all... I think you're confusing it with cygwin ... Yeah. Much as I hate working around the quirks of mingw, I

Re: [HACKERS] small parallel restore optimization

2009-03-09 Thread Magnus Hagander
Andrew Dunstan wrote: Alvaro Herrera wrote: Andrew Dunstan wrote: I have found the source of the problem I saw. dumputils.c:fmtId() uses a static PQExpBuffer which it initialises the first time it's called. This gets clobbered by simultaneous calls by Windows threads. I could just

Re: [HACKERS] small parallel restore optimization

2009-03-09 Thread Andrew Dunstan
Magnus Hagander wrote: Andrew Dunstan wrote: Alvaro Herrera wrote: Andrew Dunstan wrote: I have found the source of the problem I saw. dumputils.c:fmtId() uses a static PQExpBuffer which it initialises the first time it's called. This gets clobbered by simultaneous calls

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Andrew Dunstan
Tom Lane wrote: I've seen a recent error that suggests we are clobbering memory somewhere in the parallel code, as well as Olivier Prennant's reported error that suggests the same thing, although I'm blessed if I can see where it might be. Maybe some more eyeballs on the code would help.

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: I have found the source of the problem I saw. dumputils.c:fmtId() uses a static PQExpBuffer which it initialises the first time it's called. This gets clobbered by simultaneous calls by Windows threads. Ugh. But that doesn't explain the original

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Alvaro Herrera
Andrew Dunstan wrote: I have found the source of the problem I saw. dumputils.c:fmtId() uses a static PQExpBuffer which it initialises the first time it's called. This gets clobbered by simultaneous calls by Windows threads. I could just make it auto and set it up on each call, but that

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Andrew Dunstan
Alvaro Herrera wrote: Andrew Dunstan wrote: I have found the source of the problem I saw. dumputils.c:fmtId() uses a static PQExpBuffer which it initialises the first time it's called. This gets clobbered by simultaneous calls by Windows threads. I could just make it auto and set it

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Alvaro Herrera wrote: Could you use a different static PQExpBuffer on each thread? pthread_getspecific sort of thing, only to be used on Windows. For MSVC we could declare it with _declspec(thread) and it would be allocated in thread-local storage,

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Alvaro Herrera
Tom Lane wrote: Worst case, we could say that parallel restore isn't supported on mingw. I'm not entirely sure why we bother with that platform at all... I think you're confusing it with cygwin ... -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL

Re: [HACKERS] small parallel restore optimization

2009-03-07 Thread Tom Lane
o...@pyrenet.fr writes: the only thing I could come with is a calloc(1,12) that seems to alloc mem for filename, in that case sdewitte.dmp; so the alloc is not counting the null char at the end. Where do you see that? The memtool dump you sent doesn't show it AFAICS.

Re: [HACKERS] small parallel restore optimization

2009-03-07 Thread Andrew Dunstan
Tom Lane wrote: o...@pyrenet.fr writes: the only thing I could come with is a calloc(1,12) that seems to alloc mem for filename, in that case sdewitte.dmp; so the alloc is not counting the null char at the end. Where do you see that? The memtool dump you sent doesn't show it

[HACKERS] small parallel restore optimization

2009-03-06 Thread Andrew Dunstan
Here's a little optimization for the parallel restore code, that inhibits reopening the archive file unless the worker will actually need to read from the file (i.e. a data member). It seems to work OK on both Linux and Windows, and I propose to apply it in a day or two. I've seen a recent

Re: [HACKERS] small parallel restore optimization

2009-03-06 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Here's a little optimization for the parallel restore code, that inhibits reopening the archive file unless the worker will actually need to read from the file (i.e. a data member). It seems to work OK on both Linux and Windows, and I propose to

Re: [HACKERS] small parallel restore optimization

2009-03-06 Thread Guillaume Smet
On Fri, Mar 6, 2009 at 6:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Can you put together even a weakly reproducible test case?  Something that only fails every tenth or hundredth time would still help. It seems that Olivier can reproduce the problem at will on Unixware. I don't know if it's easy

Re: [HACKERS] small parallel restore optimization

2009-03-06 Thread ohp
] small parallel restore optimization On Fri, Mar 6, 2009 at 6:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Can you put together even a weakly reproducible test case?  Something that only fails every tenth or hundredth time would still help. not sure, none of my tests did fail at the same place