Re: [HACKERS] parallel restore vs. windows

2008-12-24 Thread Jaime Casanova
On Sun, Dec 14, 2008 at 12:13 PM, Andrew Dunstan and...@dunslane.net wrote: This version completes properly on Windows with the regression database. actually, this one doesn't apply cleanly on head -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de

Re: [HACKERS] parallel restore vs. windows

2008-12-24 Thread Andrew Dunstan
Jaime Casanova wrote: On Sun, Dec 14, 2008 at 12:13 PM, Andrew Dunstan and...@dunslane.net wrote: This version completes properly on Windows with the regression database. actually, this one doesn't apply cleanly on head I will have a new patch a day or two after Christmas,

Re: [HACKERS] parallel restore vs. windows

2008-12-17 Thread ITAGAKI Takahiro
Andrew Dunstan and...@dunslane.net wrote: I did this, but it turned out that the problem was a logic error that I found once I had managed to get a working debugger. However, the Windows thread code should now be more robust, so thanks to Andrew and Magnus for the suggestions. Hello, I

Re: [HACKERS] parallel restore vs. windows

2008-12-17 Thread Andrew Dunstan
ITAGAKI Takahiro wrote: Andrew Dunstan and...@dunslane.net wrote: I did this, but it turned out that the problem was a logic error that I found once I had managed to get a working debugger. However, the Windows thread code should now be more robust, so thanks to Andrew and Magnus for

Re: [HACKERS] parallel restore vs. windows

2008-12-14 Thread Andrew Dunstan
Andrew Chernow wrote: If it previously worked without threads, than in theory a deep copy of the thread_arg should fix the core dump; especially if the non-windows fork() method works with this patch. Maybe you can get away with only copying some of the members (trial-n-error), I don't

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Andrew Dunstan
Andrew Chernow wrote: Looks like the ArchiveHandle variable 'AH' and the TocEntry 'next_work_item' are not being deep copied at line 315 of your patch, where you prepare the RestoreArgs struct for the thread. Every thread is accessing and possibly updating the members of these structs

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Andrew Chernow
Parts of AH need deep cloning, notably the formatData member, which is done in _ReopenArchive(). Is it okay to clone this from within the thread? The reopen() appears to mess with AH-FH, which mutltiple threads are calling fclose on. The second thread is going to fail and the first

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Andrew Dunstan
Andrew Chernow wrote: Parts of AH need deep cloning, notably the formatData member, which is done in _ReopenArchive(). Is it okay to clone this from within the thread? I don't see why not. The reopen() appears to mess with AH-FH, which mutltiple threads are calling fclose on. The

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Andrew Chernow
Andrew Dunstan wrote: No, as this fragment shows, fclose() is NOT called on Windows. Oooppps. I'm the village idiot today. The program dies with a nasty dialog box when restoring a dump of the regression database after the second COPY thread disconnects. I'll poke around but

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Andrew Chernow
Parts of AH need deep cloning, notably the formatData member, which is done in _ReopenArchive(). Is it okay to clone this from within the thread? I don't see why not. Because another thread may be modifying the memory you are trying to clone. If no one modifies the formatData struct,

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Andrew Dunstan
Andrew Chernow wrote: Parts of AH need deep cloning, notably the formatData member, which is done in _ReopenArchive(). Is it okay to clone this from within the thread? I don't see why not. Because another thread may be modifying the memory you are trying to clone. If no one

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Magnus Hagander
Andrew Dunstan wrote: Andrew Chernow wrote: Parts of AH need deep cloning, notably the formatData member, which is done in _ReopenArchive(). Is it okay to clone this from within the thread? I don't see why not. Because another thread may be modifying the memory you are trying to

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Tom Lane
Magnus Hagander [EMAIL PROTECTED] writes: Andrew Dunstan wrote: I'll try. It's unfortunately not as simple as it sounds, because of the way the abstractions are arranged. I can't count the number of times I have had to stop and try to clear my head while working on this code. That's what

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Magnus Hagander
Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Andrew Dunstan wrote: I'll try. It's unfortunately not as simple as it sounds, because of the way the abstractions are arranged. I can't count the number of times I have had to stop and try to clear my head while working on this code.

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Andrew Chernow
Magnus Hagander wrote: Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Andrew Dunstan wrote: I'll try. It's unfortunately not as simple as it sounds, because of the way the abstractions are arranged. I can't count the number of times I have had to stop and try to clear my head while

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Philip Warner
Tom Lane wrote: I think pretty much everybody except Philip Warner has found the stuff around the TOC data structure and the archiver API to be confusing. I'm not immediately sure about a better design though, at least not if you don't want to duplicate a lot of code between the plain pg_dump

Re: [HACKERS] parallel restore vs. windows

2008-12-09 Thread Andrew Dunstan
Philip Warner wrote: Tom Lane wrote: I think pretty much everybody except Philip Warner has found the stuff around the TOC data structure and the archiver API to be confusing. I'm not immediately sure about a better design though, at least not if you don't want to duplicate a lot of code

Re: [HACKERS] parallel restore vs. windows

2008-12-08 Thread Andrew Chernow
Andrew Dunstan wrote: OK, after quite some trying I have hit a brick wall. I have been unable to get parallel restore to work with Windows threading. No doubt I am missing something, but I really don't know what. Unless someone can tell me what I am doing wrong, I have these possibilities:

Re: [HACKERS] parallel restore vs. windows

2008-12-08 Thread Andrew Dunstan
Andrew Chernow wrote: Andrew Dunstan wrote: OK, after quite some trying I have hit a brick wall. I have been unable to get parallel restore to work with Windows threading. No doubt I am missing something, but I really don't know what. Unless someone can tell me what I am doing wrong, I

Re: [HACKERS] parallel restore vs. windows

2008-12-08 Thread Andrew Chernow
HANDLE h = (HANDLE)_beginthreadex(NULL, 0, thread_start, arg, 0, NULL); This didn't give me any more joy, unfortunately. But you're right, I should be using it. Are these threads sharing memory, intentionally or by mistake? if(h) CloseHandle(h); Umm, even if I wait on the handle

Re: [HACKERS] parallel restore vs. windows

2008-12-08 Thread Andrew Dunstan
Andrew Chernow wrote: HANDLE h = (HANDLE)_beginthreadex(NULL, 0, thread_start, arg, 0, NULL); This didn't give me any more joy, unfortunately. But you're right, I should be using it. Are these threads sharing memory, intentionally or by mistake? Things they write, and things they

Re: [HACKERS] parallel restore vs. windows

2008-12-08 Thread Andrew Chernow
Andrew Dunstan wrote: Andrew Chernow wrote: HANDLE h = (HANDLE)_beginthreadex(NULL, 0, thread_start, arg, 0, NULL); This didn't give me any more joy, unfortunately. But you're right, I should be using it. Are these threads sharing memory, intentionally or by mistake? Things they

Re: [HACKERS] parallel restore vs. windows

2008-12-08 Thread Andrew Chernow
Are these threads sharing memory, intentionally or by mistake? Things they write, and things they read but might not be stable, are not supposed to be shared. If they are it's a mistake. Looks like the ArchiveHandle variable 'AH' and the TocEntry 'next_work_item' are not being deep