Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-07-26 Thread Chapman Flack
On 07/17/17 11:29, Michael Paquier wrote: > FWIW, I would rather see any optimization done in > AdvanceXLInsertBuffer() instead of seeing a second memset re-zeroing > the WAL page header after its data has been initialized by > AdvanceXLInsertBuffer() once. Is that an aesthetic 'rather', or is

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-07-17 Thread Michael Paquier
On Thu, Jul 6, 2017 at 3:48 PM, Heikki Linnakangas wrote: > On 07/03/2017 06:30 PM, Chapman Flack wrote: >> Although it's moot in the straightforward approach of re-zeroing in >> the loop, it would still help my understanding of the system to know >> if there is some subtlety

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-07-17 Thread Michael Paquier
On Fri, Jun 23, 2017 at 6:08 AM, Chapman Flack wrote: > Well, gzip was doing pretty well; it could get a 16 MB segment file down > to under 27 kB, or less than 14 bytes for each of 2000 pages, when a page > header is what, 20 bytes, it looks like? I'm not sure how much

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-07-06 Thread Heikki Linnakangas
On 07/03/2017 06:30 PM, Chapman Flack wrote: On 07/03/2017 09:39 AM, Heikki Linnakangas wrote: Hmm. That's not the problem, though. Imagine that instead of the loop above, you do just: WALInsertLockUpdateInsertingAt(CurrPos); AdvanceXLInsertBuffer(EndPos, false); AdvanceXLInsertBuffer() will

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-07-03 Thread Chapman Flack
On 07/03/2017 09:39 AM, Heikki Linnakangas wrote: > Hmm. That's not the problem, though. Imagine that instead of the loop > above, you do just: > > WALInsertLockUpdateInsertingAt(CurrPos); > AdvanceXLInsertBuffer(EndPos, false); > > AdvanceXLInsertBuffer() will call XLogWrite(), to flush out

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-07-03 Thread Chapman Flack
On 07/03/2017 09:39 AM, Heikki Linnakangas wrote: > The most straightforward solution would be to just clear each page with > memset() in the loop. It's a bit wasteful to clear the page again, just > after AdvanceXLInsertBuffer() has initialized it, but this isn't > performance-critical. An in

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-07-03 Thread Heikki Linnakangas
On 06/26/2017 04:20 AM, Chapman Flack wrote: I notice CopyXLogRecordToWAL contains this loop (in the case where the record being copied is a switch): while (CurrPos < EndPos) { /* initialize the next page (if not initialized already) */ WALInsertLockUpdateInsertingAt(CurrPos);

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-06-29 Thread Chapman Flack
On 06/25/17 21:20, Chapman Flack wrote: > I want to make sure I understand what the deadlock potential is > in this case. AdvanceXLInsertBuffer will call WaitXLogInsertionsToFinish > ... > Does not the fact we hold all the insertion slots exclude the possibility > that any dirty buffer (preceding

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-06-25 Thread Chapman Flack
I notice CopyXLogRecordToWAL contains this loop (in the case where the record being copied is a switch): while (CurrPos < EndPos) { /* initialize the next page (if not initialized already) */ WALInsertLockUpdateInsertingAt(CurrPos); AdvanceXLInsertBuffer(CurrPos, false); CurrPos

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-06-22 Thread Chapman Flack
On 06/21/17 04:51, Heikki Linnakangas wrote: > (I'm cleaning up my inbox, hence the delayed reply) I had almost completely forgotten ever bringing it up. :) > When I wrote that code, I don't remember if I realized that we're > initializing the page headers, or if I thought that it's good enough

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2017-06-21 Thread Heikki Linnakangas
(I'm cleaning up my inbox, hence the delayed reply) On 08/02/2016 10:51 PM, Robert Haas wrote: On Tue, Aug 2, 2016 at 2:33 PM, Bruce Momjian wrote: On Tue, Jul 26, 2016 at 05:42:43PM -0400, Chapman Flack wrote: Even so, I'd be curious whether it would break anything to have

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-08-02 Thread Robert Haas
On Tue, Aug 2, 2016 at 2:33 PM, Bruce Momjian wrote: > On Tue, Jul 26, 2016 at 05:42:43PM -0400, Chapman Flack wrote: >> Even so, I'd be curious whether it would break anything to have >> xlp_pageaddr simply set to InvalidXLogRecPtr in the dummy zero >> pages written to fill out

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-08-02 Thread Chapman Flack
On 08/02/2016 02:33 PM, Bruce Momjian wrote: > My guess is that the bytes are there to detect problems where > a 512-byte disk sector is zeroed by a disk failure. Does that seem plausible? (a) there is only one such header for every 16 512-byte disk sectors, so it only affords a 6% chance of

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-08-02 Thread Bruce Momjian
On Tue, Jul 26, 2016 at 05:42:43PM -0400, Chapman Flack wrote: > Even so, I'd be curious whether it would break anything to have > xlp_pageaddr simply set to InvalidXLogRecPtr in the dummy zero > pages written to fill out a segment. At least until it's felt > that archive_timeout has been so

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-26 Thread Chapman Flack
On 07/26/16 20:01, Michael Paquier wrote: > On Tue, Jul 26, 2016 at 9:48 PM, Amit Kapila wrote: >> Does any body else see the use case >> reported by Chapman important enough that we try to have some solution >> for it in-core? > > The lack of updates in the pg_lesslog

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-26 Thread Michael Paquier
On Tue, Jul 26, 2016 at 9:48 PM, Amit Kapila wrote: > Does any body else see the use case > reported by Chapman important enough that we try to have some solution > for it in-core? The lack of updates in the pg_lesslog project is a sign that it is not that much used. I

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-26 Thread Chapman Flack
On 07/26/2016 04:21 PM, Robert Haas wrote: > I'm kind of curious WHY you are using archiving and forcing regular > segment switches rather than just using streaming replication. > ... AFAIK, streaming replication > essentially obsoleted that use case. You can just dribble the > individual bytes

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-26 Thread Robert Haas
On Fri, Jul 22, 2016 at 6:02 PM, Chapman Flack wrote: > At $work, we have a usually-low-activity PG database, so that almost > always the used fraction of each 16 MB WAL segment is far smaller > than 16 MB, and so it's a big win for archived-WAL storage space > if an

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-26 Thread Chapman Flack
On 07/26/2016 08:48 AM, Amit Kapila wrote: > general, if you have a very low WAL activity, then the final size of > compressed WAL shouldn't be much even if you use gzip. It seems your 9.5 pg_xlog, low activity test cluster (segment switches forced only by checkpoint timeouts), compressed with

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-26 Thread Amit Kapila
On Tue, Jul 26, 2016 at 9:02 AM, Chapman Flack wrote: > On 07/25/16 22:09, Michael Paquier wrote: > >> This is over-complicating things for little gain. The new behavior of >> filling in with zeros the tail of a segment makes things far better >> when using gzip in

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-25 Thread Chapman Flack
On 07/25/16 22:09, Michael Paquier wrote: > This is over-complicating things for little gain. The new behavior of > filling in with zeros the tail of a segment makes things far better > when using gzip in archive_command. Then how about filling with actual zeros, instead of with mostly-zeros as

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-25 Thread Michael Paquier
On Mon, Jul 25, 2016 at 11:21 PM, Chapman Flack wrote: > The impression that leaves is of tools that relied too heavily > on internal format knowledge to be viable outside of core, which > have had at least periods of incompatibility with newer PG versions, > and whose

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-25 Thread Chapman Flack
On 07/23/2016 08:25 AM, Amit Kapila wrote: > On Sat, Jul 23, 2016 at 3:32 AM, Chapman Flack wrote: >> >> Would it then be possible to go back to the old behavior (or make >> it selectable) of not overwriting the full 16 MB every time? >> > > I don't see going back to old

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-23 Thread Amit Kapila
On Sat, Jul 23, 2016 at 3:32 AM, Chapman Flack wrote: > > Would it then be possible to go back to the old behavior (or make > it selectable) of not overwriting the full 16 MB every time? > I don't see going back to old behaviour is an improvement, because as as you pointed

[HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2016-07-22 Thread Chapman Flack
Teaser: change made in 9.4 to simplify WAL segment compression made it easier to compress a low-activity-period WAL segment from 16 MB to about 27 kB ... but much harder to do better than that, as I was previously doing (about two orders of magnitude better). At $work, we have a