Re: [PoC PATCH] Parallel dump to /dev/null

On Thu, Mar 22, 2018 at 08:42:05AM +0100, Michael Banck wrote: > Am Donnerstag, den 22.03.2018, 10:33 +0900 schrieb Michael Paquier: >> Hence, is there any objection to mark this patch as returned with >> feedback? > > I've done so now. Thanks Michael. -- Michael signature.asc Description:

Re: [PoC PATCH] Parallel dump to /dev/null

Hi, Am Donnerstag, den 22.03.2018, 10:33 +0900 schrieb Michael Paquier: > Hence, is there any objection to mark this patch as returned with > feedback? I've done so now. Cheers, Michael -- Michael Banck Projektleiter / Senior Berater Tel.: +49 2166 9901-171 Fax: +49 2166 9901-100 Email:

Re: [PoC PATCH] Parallel dump to /dev/null

On Wed, Mar 21, 2018 at 09:07:41AM +0100, Michael Banck wrote: > Am Dienstag, den 20.03.2018, 19:19 -0400 schrieb Stephen Frost: >> Instead of trying to use pg_dump for this, we should provide a way to >> actually check for corruption across everything (instead of just the >> heap..), and have all

Re: [PoC PATCH] Parallel dump to /dev/null

On Tue, Mar 20, 2018 at 6:57 AM, Michael Paquier wrote: > Now, why are people using pg_dump > /dev/null? Mainly the lack of > better tools, which would be actually able to detect pages in corrupted > pages in one run, and not only heap pages. I honestly think that > amcheck

Re: [PoC PATCH] Parallel dump to /dev/null

Hi, Am Dienstag, den 20.03.2018, 19:19 -0400 schrieb Stephen Frost: > * Christoph Berg (christoph.b...@credativ.de) wrote: > > Re: Tom Lane 2018-03-20 <12960.1521557...@sss.pgh.pa.us> > > > It might help if the patch were less enthusiastic about trying to > > > "optimize" by avoiding extra file

Re: [PoC PATCH] Parallel dump to /dev/null

Greetings, * Christoph Berg (christoph.b...@credativ.de) wrote: > Re: Tom Lane 2018-03-20 <12960.1521557...@sss.pgh.pa.us> > > It might help if the patch were less enthusiastic about trying to > > "optimize" by avoiding extra file opens/closes in the case of output > > to /dev/null. That seems

Re: [PoC PATCH] Parallel dump to /dev/null

Re: Tom Lane 2018-03-20 <12960.1521557...@sss.pgh.pa.us> > It might help if the patch were less enthusiastic about trying to > "optimize" by avoiding extra file opens/closes in the case of output > to /dev/null. That seems to account for a lot of the additional > complication, and I can't see a

Re: [PoC PATCH] Parallel dump to /dev/null

Christoph Berg writes: > Re: Michael Paquier 2018-03-20 <20180320135720.ge13...@paquier.xyz> >> Now, why are people using pg_dump > /dev/null? Mainly the lack of >> better tools, which would be actually able to detect pages in corrupted >> pages in one run, and not

Re: [PoC PATCH] Parallel dump to /dev/null

Re: Michael Paquier 2018-03-20 <20180320135720.ge13...@paquier.xyz> > Now, why are people using pg_dump > /dev/null? Mainly the lack of > better tools, which would be actually able to detect pages in corrupted > pages in one run, and not only heap pages. I honestly think that > amcheck is

Re: [PoC PATCH] Parallel dump to /dev/null

On Tue, Mar 20, 2018 at 09:54:07AM +0100, Christoph Berg wrote: > Otherwise, +1 from me. I have been thinking about this patch lately, and on the contrary I am voting -1 for the concepts behind this patch. pg_dump is by nature a tool aimed at fetching data from the database, at putting it in a

Re: [PoC PATCH] Parallel dump to /dev/null

Re: Alvaro Herrera 2018-03-05 <20180305165609.kq5y7uzy64o45vsg@alvherre.pgsql> > The reason I noticed is I wondered about the amount of open() calls > (plus zlib function calls) we would save by keeping an FD open to > /dev/null, rather than opening it over and over for each object -- ie., > maybe

Re: [PoC PATCH] Parallel dump to /dev/null

I made a few amendments (here's v5) and was ready to push, when I noticed that _StartBlobs() does not seem to be doing the right thing. Did you test this with a few large objects? The reason I noticed is I wondered about the amount of open() calls (plus zlib function calls) we would save by

Re: [PoC PATCH] Parallel dump to /dev/null

Hi Alvaro, Am Montag, den 05.03.2018, 12:48 -0300 schrieb Alvaro Herrera: > Please make ctx->discard a bool, not an int. Ok, done so now. I forgot to mention that in the earlier resubmission, but I had a look at the other pg_backup_*.c files and isSpecialScript and hasSeek were ints as well, so

Re: [PoC PATCH] Parallel dump to /dev/null

Please make ctx->discard a bool, not an int. You can initialize it like this: ctx->discard = strcmp(ctx->directory, DEVNULL) == 0; Why do you change the mode to create directory to 0711 from 0700? You have a cuddled "else" in the last hunk. Please uncuddle. -- Álvaro Herrera

Re: [PoC PATCH] Parallel dump to /dev/null

Hi Andreas, On Mon, Mar 05, 2018 at 03:15:19PM +0100, Andreas 'ads' Scherbaum wrote: > The null device is already defined in port.h, as DEVNULL. No need to > redefine it as NULL_DEVICE. Thanks for pointing that out, a new patch removing NULL_DEVICE is attached. Michael -- Michael Banck

Re: [PoC PATCH] Parallel dump to /dev/null

On Mon, Mar 5, 2018 at 1:49 PM, Michael Banck wrote: > > In any case, new patch attached which does this. > The null device is already defined in port.h, as DEVNULL. No need to redefine it as NULL_DEVICE. Alternatively paths.h defines _PATH_DEVNULL. Regards, --

Re: [PoC PATCH] Parallel dump to /dev/null

Hi Alvaro, On Thu, Mar 01, 2018 at 04:00:52PM -0300, Alvaro Herrera wrote: > strdup -> pg_strdup() Fixed. > I wonder if, instead of doing strcmp() all over the place, we should > give this behavior a separate boolean flag in lclContext. I mused a bit about what to name that flag and came up

Re: [PoC PATCH] Parallel dump to /dev/null

strdup -> pg_strdup() I wonder if, instead of doing strcmp() all over the place, we should give this behavior a separate boolean flag in lclContext. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PoC PATCH] Parallel dump to /dev/null

On 2018-03-01 14:21:24 +0100, Michael Banck wrote: > Hi, > > Am Donnerstag, den 01.03.2018, 01:28 -0800 schrieb Andres Freund: > > Hi, > > > > On 2018-02-28 14:28:41 +0100, Michael Banck wrote: > > > I have added this patch to the next commitfest: > > > > > >

Re: [PoC PATCH] Parallel dump to /dev/null

Hi, Am Donnerstag, den 01.03.2018, 01:28 -0800 schrieb Andres Freund: > Hi, > > On 2018-02-28 14:28:41 +0100, Michael Banck wrote: > > I have added this patch to the next commitfest: > > > > https://commitfest.postgresql.org/17/1576/ > > Given this is a new patch, submitted for the last

Re: [PoC PATCH] Parallel dump to /dev/null

Hi, On 2018-02-28 14:28:41 +0100, Michael Banck wrote: > I have added this patch to the next commitfest: > > https://commitfest.postgresql.org/17/1576/ Given this is a new patch, submitted for the last commitfest, and not completely trivial, I'd argue this is too late for v11. Does anybody

Re: [PoC PATCH] Parallel dump to /dev/null

Hi, On Thu, Feb 01, 2018 at 02:24:46PM +0100, Michael Banck wrote: > dumping a database to /dev/null via pg_dump is (AFAIK) one recommended > way to check for corruption. However, dumping to /dev/null is currently > not supported in directory mode which makes it not possible to dump to >

Re: [PoC PATCH] Parallel dump to /dev/null

Hi. > 2 февр. 2018 г., в 13:25, Andrey Borodin написал(а): > >> Do folks here think that speedups would be worth it? > I've seen backup verification via COPY TO to /dev/null in production. These > copies were parallelized on the side of verification script. > Not sure

Re: [PoC PATCH] Parallel dump to /dev/null

Hi, Am Freitag, den 02.02.2018, 13:22 +0900 schrieb Michael Paquier: > On Thu, Feb 01, 2018 at 02:24:46PM +0100, Michael Banck wrote: > > dumping a database to /dev/null via pg_dump is (AFAIK) one recommended > > way to check for corruption. However, dumping to /dev/null is currently > > not

Re: [PoC PATCH] Parallel dump to /dev/null

On Thu, Feb 01, 2018 at 02:24:46PM +0100, Michael Banck wrote: > dumping a database to /dev/null via pg_dump is (AFAIK) one recommended > way to check for corruption. However, dumping to /dev/null is currently > not supported in directory mode which makes it not possible to dump to > /dev/null in