On Wed, Mar 11, 2015 at 3:57 PM, Fujii Masao wrote:
> On Wed, Mar 11, 2015 at 7:08 AM, Rahila Syed wrote:
>> Hello,
>>
>>>I have some minor comments
>>
>> The comments have been implemented in the attached patch.
>
> Thanks for updating the patch! I just changed a bit and finally pushed it.
> Tha
On Wed, Mar 11, 2015 at 7:08 AM, Rahila Syed wrote:
> Hello,
>
>>I have some minor comments
>
> The comments have been implemented in the attached patch.
Thanks for updating the patch! I just changed a bit and finally pushed it.
Thanks everyone involved in this patch!
Regards,
--
Fujii Masao
On Mon, Mar 9, 2015 at 9:08 PM, Michael Paquier
wrote:
> On Mon, Mar 9, 2015 at 4:29 PM, Fujii Masao wrote:
>> On Thu, Mar 5, 2015 at 10:08 PM, Michael Paquier
>> wrote:
>>> On Thu, Mar 5, 2015 at 9:14 PM, Syed, Rahila
>>> wrote:
Please find attached a patch. As discussed, flag to denote
Hello,
>I have some minor comments
The comments have been implemented in the attached patch.
>I think that extra parenthesis should be used for the first expression
>with BKPIMAGE_HAS_HOLE.
Parenthesis have been added to improve code readability.
Thank you,
Rahila Syed
On Mon, Mar 9, 2015 at
On Mon, Mar 9, 2015 at 9:08 PM, Michael Paquier wrote:
> On Mon, Mar 9, 2015 at 4:29 PM, Fujii Masao wrote:
>> Thanks for updating the patch! Attached is the refactored version of the
>> patch.
Fujii-san and I had a short chat about tuning a bit the PGLZ strategy
which is now PGLZ_strategy_defaul
On Mon, Mar 9, 2015 at 4:29 PM, Fujii Masao wrote:
> On Thu, Mar 5, 2015 at 10:08 PM, Michael Paquier
> wrote:
>> On Thu, Mar 5, 2015 at 9:14 PM, Syed, Rahila wrote:
>>> Please find attached a patch. As discussed, flag to denote compression and
>>> presence of hole in block image has been adde
On Thu, Mar 5, 2015 at 10:08 PM, Michael Paquier
wrote:
> On Thu, Mar 5, 2015 at 9:14 PM, Syed, Rahila wrote:
>> Please find attached a patch. As discussed, flag to denote compression and
>> presence of hole in block image has been added in XLogRecordImageHeader
>> rather than block header.
T
On Mon, Feb 16, 2015 at 9:08 PM, Andres Freund wrote:
> On 2015-02-16 20:55:20 +0900, Michael Paquier wrote:
>> On Mon, Feb 16, 2015 at 8:30 PM, Syed, Rahila
>> wrote:
>>
>> >
>> > Regarding the sanity checks that have been added recently. I think that
>> > they are useful but I am suspecting as
On Thu, Mar 5, 2015 at 10:28 PM, Andres Freund wrote:
> On 2015-03-05 12:14:04 +, Syed, Rahila wrote:
>> Please find attached a patch. As discussed, flag to denote
>> compression and presence of hole in block image has been added in
>> XLogRecordImageHeader rather than block header.
>
> FWIW,
On 2015-03-05 12:14:04 +, Syed, Rahila wrote:
> Please find attached a patch. As discussed, flag to denote
> compression and presence of hole in block image has been added in
> XLogRecordImageHeader rather than block header.
FWIW, I personally won't commit it with things done that way. I thin
On Thu, Mar 5, 2015 at 9:14 PM, Syed, Rahila wrote:
> Please find attached a patch. As discussed, flag to denote compression and
> presence of hole in block image has been added in XLogRecordImageHeader
> rather than block header.
>
> Following are WAL numbers based on attached test script po
Hello,
Please find attached a patch. As discussed, flag to denote compression and
presence of hole in block image has been added in XLogRecordImageHeader rather
than block header.
Following are WAL numbers based on attached test script posted by Michael
earlier in the thread.
Hello,
>Are there any other flag bits that we should or are planning to add into WAL
>header newly, except the above two? If yes and they are required by even a
>block which doesn't have an image, I will change my mind and agree to add
>something like chunk ID to a block header.
>But I guess t
On Wed, Mar 4, 2015 at 12:41 AM, Syed, Rahila wrote:
> Please find attached updated patch with WAL replay error fixed. The patch
> follows chunk ID approach of xlog format.
(Review done independently of the chunk_id stuff being good or not,
already gave my opinion on the matter).
* readRecord
On Tue, Mar 3, 2015 at 9:34 AM, Michael Paquier
wrote:
> On Tue, Mar 3, 2015 at 9:24 AM, Andres Freund wrote:
>> On 2015-03-03 08:59:30 +0900, Michael Paquier wrote:
>>> Already mentioned upthread, but I agree with Fujii-san here: adding
>>> information related to the state of a block image in
>>
Hello,
>It would be good to get those problems fixed first. Could you send an updated
>patch?
Please find attached updated patch with WAL replay error fixed. The patch
follows chunk ID approach of xlog format.
Following are brief measurement numbers.
On Tue, Mar 3, 2015 at 9:24 AM, Andres Freund wrote:
> On 2015-03-03 08:59:30 +0900, Michael Paquier wrote:
>> Already mentioned upthread, but I agree with Fujii-san here: adding
>> information related to the state of a block image in
>> XLogRecordBlockHeader makes little sense because we are not
On 2015-03-03 08:59:30 +0900, Michael Paquier wrote:
> Already mentioned upthread, but I agree with Fujii-san here: adding
> information related to the state of a block image in
> XLogRecordBlockHeader makes little sense because we are not sure to
> have a block image, perhaps there is only data as
On Tue, Mar 3, 2015 at 5:17 AM, Rahila Syed wrote:
> Hello,
>
>>When I test the WAL or replication related features, I usually run
>>"make installcheck" and pgbench against the master at the same time
>>after setting up the replication environment.
> I will conduct these tests before sending updat
Hello,
>When I test the WAL or replication related features, I usually run
>"make installcheck" and pgbench against the master at the same time
>after setting up the replication environment.
I will conduct these tests before sending updated version.
>Seems this increases the header size of WAL re
On Fri, Feb 27, 2015 at 12:44 PM, Michael Paquier
wrote:
> On Fri, Feb 27, 2015 at 8:01 AM, Michael Paquier
> wrote:
>> On Fri, Feb 27, 2015 at 6:54 AM, Rahila Syed wrote:
Even this patch doesn't work fine. The standby emit the following
error messages.
>>>
>>> Yes this bug remains unsol
On Fri, Feb 27, 2015 at 8:01 AM, Michael Paquier
wrote:
> On Fri, Feb 27, 2015 at 6:54 AM, Rahila Syed wrote:
>>>Even this patch doesn't work fine. The standby emit the following
>>>error messages.
>>
>> Yes this bug remains unsolved. I am still working on resolving this.
>>
>> Following chunk ID
On Fri, Feb 27, 2015 at 6:54 AM, Rahila Syed wrote:
> Hello,
>
>>Even this patch doesn't work fine. The standby emit the following
>>error messages.
>
> Yes this bug remains unsolved. I am still working on resolving this.
>
> Following chunk IDs have been added in the attached patch as suggested
>
Hello,
>Even this patch doesn't work fine. The standby emit the following
>error messages.
Yes this bug remains unsolved. I am still working on resolving this.
Following chunk IDs have been added in the attached patch as suggested
upthread.
+#define XLR_CHUNK_BLOCK_REFERENCE 0x10
+#define XLR_C
On Tue, Feb 24, 2015 at 6:46 PM, Syed, Rahila wrote:
> Hello ,
>
>>I've not read this logic yet, but ISTM there is a bug in that new WAL format
>>because I got the following error and the startup process could not replay
>>any WAL records when I set up replication and enabled wal_compression.
>
On 2015-02-24 16:03:41 +0900, Michael Paquier wrote:
> Looking at this code, I think that it is really confusing to move the data
> related to the status of the backup block out of XLogRecordBlockImageHeader
> to the chunk ID itself that may *not* include a backup block at all as it
> is conditione
tgresql.org
[mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Fujii Masao
Sent: Monday, February 23, 2015 5:52 PM
To: Rahila Syed
Cc: PostgreSQL-development; Andres Freund; Michael Paquier
Subject: Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes
On Mon, Feb 23, 2015 at 5:28 PM, R
On Mon, Feb 23, 2015 at 9:22 PM, Fujii Masao wrote:
> On Mon, Feb 23, 2015 at 5:28 PM, Rahila Syed
> wrote:
> > Hello,
> >
> > Attached is a patch which has following changes,
> >
> > As suggested above block ID in xlog structs has been replaced by chunk
> ID.
> > Chunk ID is used to distinguish
On Mon, Feb 23, 2015 at 5:28 PM, Rahila Syed wrote:
> Hello,
>
> Attached is a patch which has following changes,
>
> As suggested above block ID in xlog structs has been replaced by chunk ID.
> Chunk ID is used to distinguish between different types of xlog record
> fragments.
> Like,
> XLR_CHUNK
> From: Andres Freund [mailto:and...@2ndquadrant.com]
> Sent: Monday, February 16, 2015 5:26 PM
> To: Syed, Rahila
> Cc: Michael Paquier; Fujii Masao; PostgreSQL mailing lists
> Subject: Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes
>
> On 2015-02-16 11:30:
On Mon, Feb 16, 2015 at 8:55 PM, Andres Freund wrote:
> On 2015-02-16 11:30:20 +, Syed, Rahila wrote:
>> - * As a trivial form of data compression, the XLOG code is aware that
>> - * PG data pages usually contain an unused "hole" in the middle, which
>> - * contains only zero bytes. If hole_l
PostgreSQL mailing lists
Subject: Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes
On 2015-02-16 11:30:20 +, Syed, Rahila wrote:
> - * As a trivial form of data compression, the XLOG code is aware that
> - * PG data pages usually contain an unused "hole" in the middl
On 2015-02-16 20:55:20 +0900, Michael Paquier wrote:
> On Mon, Feb 16, 2015 at 8:30 PM, Syed, Rahila
> wrote:
>
> >
> > Regarding the sanity checks that have been added recently. I think that
> > they are useful but I am suspecting as well that only a check on the record
> > CRC is done because t
On 2015-02-16 11:30:20 +, Syed, Rahila wrote:
> - * As a trivial form of data compression, the XLOG code is aware that
> - * PG data pages usually contain an unused "hole" in the middle, which
> - * contains only zero bytes. If hole_length > 0 then we have removed
> - * such a "hole" from the
On Mon, Feb 16, 2015 at 8:30 PM, Syed, Rahila
wrote:
>
> Regarding the sanity checks that have been added recently. I think that
> they are useful but I am suspecting as well that only a check on the record
> CRC is done because that's reliable enough and not doing those checks
> accelerates a bi
Hello,
Thank you for reviewing and testing the patch.
>+ /* leave if data cannot be compressed */
>+ if (compressed_len == 0)
>+ return false;
>This should be < 0, pglz_compress returns -1 when compression fails.
>
>+ if (pglz_decompress(block_image, bkpb->
On Thu, Feb 12, 2015 at 8:08 PM, Syed, Rahila
wrote:
>
>
>
> Thank you for comments. Please find attached the updated patch.
>
>
>
> >This patch fails to compile:
> >xlogreader.c:1049:46: error: extraneous ')' after condition, expected a
statement
> >blk->wi
Thank you for comments. Please find attached the updated patch.
>This patch fails to compile:
>xlogreader.c:1049:46: error: extraneous ')' after condition, expected a
>statement
>blk->with_hole && blk->hole_offset <=
> 0))
This has been rectified.
>Note
On Wed, Feb 11, 2015 at 11:03 PM, Syed, Rahila
wrote:
> >IMO, we should add details about how this new field is used in the
> comments on top of XLogRecordBlockImageHeader, meaning that when a page
> hole is present we use the compression info structure and when there is no
> hole, we are sure th
>IMO, we should add details about how this new field is used in the comments on
>top of XLogRecordBlockImageHeader, meaning that when a page hole is present we
>use the compression info structure and when there is no hole, we are sure that
>the FPW raw length is BLCKSZ meaning that the two byte
greSQL mailing lists
Subject: Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes
Hello,
>> Do we always need extra two bytes for compressed backup block?
>> ISTM that extra bytes are not necessary when the hole length is zero.
>> In this case the length of the origina
On Mon, Feb 9, 2015 at 10:27 PM, Syed, Rahila wrote:
> (snip)
Thanks for showing up here! I have not tested the test the patch,
those comments are based on what I read from v17.
>>> Do we always need extra two bytes for compressed backup block?
>>> ISTM that extra bytes are not necessary when the
all the
>above code to it for more simplicity
This is also implemented in the patch attached.
Thank you,
Rahila Syed
-Original Message-
From: Michael Paquier [mailto:michael.paqu...@gmail.com]
Sent: Friday, February 06, 2015 6:00 PM
To: Fujii Masao
Cc: Syed, Rahila; PostgreSQL mailing lis
On Fri, Feb 6, 2015 at 3:42 AM, Michael Paquier
wrote:
> Fujii Masao wrote:
>> I wrote
>>> This is an inspiration from lz4 APIs. Wouldn't it be buggy for a
>>> compression algorithm to return a size of 0 bytes as compressed or
>>> decompressed length btw? We could as well make it return a negative
On Fri, Feb 6, 2015 at 6:35 PM, Syed, Rahila wrote:
> The compression patch can use the latest interface
> MemoryContextAllocExtended to proceed without compression when sufficient
> memory is not available for
> scratch buffer.
> The attached patch introduces OutOfMem flag which is set on when
, 2015 12:46 AM
To: Syed, Rahila
Cc: PostgreSQL mailing lists
Subject: Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes
On Thu, Feb 5, 2015 at 11:06 PM, Syed, Rahila wrote:
>>/*
>>+* We recheck the actual size even if pglz_compress() report success,
>>+* because
On Fri, Feb 6, 2015 at 4:30 PM, Michael Paquier wrote:
> On Fri, Feb 6, 2015 at 3:03 PM, Fujii Masao wrote:
>> Do we always need extra two bytes for compressed backup block?
>> ISTM that extra bytes are not necessary when the hole length is zero.
>> In this case the length of the original backup bl
On Fri, Feb 6, 2015 at 3:03 PM, Fujii Masao wrote:
> Do we always need extra two bytes for compressed backup block?
> ISTM that extra bytes are not necessary when the hole length is zero.
> In this case the length of the original backup block (i.e., uncompressed)
> must be BLCKSZ, so we don't need
On Fri, Feb 6, 2015 at 4:15 AM, Michael Paquier
wrote:
> On Thu, Feb 5, 2015 at 11:06 PM, Syed, Rahila wrote:
>>>/*
>>>+* We recheck the actual size even if pglz_compress() report success,
>>>+* because it might be satisfied with having saved as little as one byte
>>>+* in the compres
On Thu, Feb 5, 2015 at 11:06 PM, Syed, Rahila wrote:
>>/*
>>+* We recheck the actual size even if pglz_compress() report success,
>>+* because it might be satisfied with having saved as little as one byte
>>+* in the compressed data.
>>+*/
>>+ *len = (uint16) compressed_len;
>>+
Fujii Masao wrote:
> I wrote
>> This is an inspiration from lz4 APIs. Wouldn't it be buggy for a
>> compression algorithm to return a size of 0 bytes as compressed or
>> decompressed length btw? We could as well make it return a negative
>> value when a failure occurs if you feel more comfortable w
iling lists
Subject: Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes
On Wed, Jan 7, 2015 at 12:51 AM, Rahila Syed wrote:
> Following are some comments,
Thanks for the feedback.
>>uint16 hole_offset:15, /* number of bytes in "hole" */
> Typo in description of h
On Tue, Jan 6, 2015 at 11:09 AM, Michael Paquier
wrote:
> On Mon, Jan 5, 2015 at 10:29 PM, Fujii Masao wrote:
>> On Sun, Dec 28, 2014 at 10:57 PM, Michael Paquier wrote:
>> The patch 1 cannot be applied to the master successfully because of
>> recent change.
> Yes, that's caused by ccb161b. Attac
On Wed, Jan 7, 2015 at 12:51 AM, Rahila Syed wrote:
> Following are some comments,
Thanks for the feedback.
>>uint16 hole_offset:15, /* number of bytes in "hole" */
> Typo in description of hole_offset
Fixed. That's "before hole".
>>for (block_id = 0; block_id <= record->max_block_id; b
Hello,
>Yes, that's caused by ccb161b. Attached are rebased versions.
Following are some comments,
>uint16 hole_offset:15, /* number of bytes in "hole" */
Typo in description of hole_offset
>for (block_id = 0; block_id <= record->max_block_id; block_id++)
>- {
>-
On Mon, Jan 5, 2015 at 10:29 PM, Fujii Masao wrote:
> On Sun, Dec 28, 2014 at 10:57 PM, Michael Paquier wrote:
> The patch 1 cannot be applied to the master successfully because of
> recent change.
Yes, that's caused by ccb161b. Attached are rebased versions.
>> - The real stuff comes with patch
On Sun, Dec 28, 2014 at 10:57 PM, Michael Paquier
wrote:
>
>
> On Fri, Dec 26, 2014 at 4:16 PM, Michael Paquier
> wrote:
>> On Fri, Dec 26, 2014 at 3:24 PM, Fujii Masao
>> wrote:
>>> pglz_compress() and pglz_decompress() still use PGLZ_Header, so the
>>> frontend
>>> which uses those functions n
On Fri, Dec 26, 2014 at 4:16 PM, Michael Paquier
wrote:
> On Fri, Dec 26, 2014 at 3:24 PM, Fujii Masao
wrote:
>> pglz_compress() and pglz_decompress() still use PGLZ_Header, so the
frontend
>> which uses those functions needs to handle PGLZ_Header. But it basically
should
>> be handled via the va
On Fri, Dec 26, 2014 at 3:24 PM, Fujii Masao wrote:
> pglz_compress() and pglz_decompress() still use PGLZ_Header, so the frontend
> which uses those functions needs to handle PGLZ_Header. But it basically
> should
> be handled via the varlena macros. That is, the frontend still seems to need
>
On Fri, Dec 26, 2014 at 12:31 PM, Michael Paquier
wrote:
> On Thu, Dec 25, 2014 at 10:10 PM, Michael Paquier
> wrote:
>> On Wed, Dec 24, 2014 at 9:03 PM, Michael Paquier
>> wrote:
>>> Returning only a boolean is fine for me (that's what my first patch
>>> did), especially if we add at some point
On Thu, Dec 25, 2014 at 10:10 PM, Michael Paquier
wrote:
> On Wed, Dec 24, 2014 at 9:03 PM, Michael Paquier
> wrote:
>> Returning only a boolean is fine for me (that's what my first patch
>> did), especially if we add at some point hooks for compression and
>> decompression calls.
> Here is a pat
On Wed, Dec 24, 2014 at 9:03 PM, Michael Paquier
wrote:
> Returning only a boolean is fine for me (that's what my first patch
> did), especially if we add at some point hooks for compression and
> decompression calls.
Here is a patch rebased on current HEAD (60838df) for the core feature
with the
On Wed, Dec 24, 2014 at 8:44 PM, Fujii Masao wrote:
> On Fri, Dec 19, 2014 at 12:19 AM, Michael Paquier
> wrote:
> Firstly I'm thinking to commit the
> 0001-Move-pg_lzcompress.c-to-src-common.patch.
>
> pg_lzcompress.h still exists in include/utils, but it should be moved to
> include/common?
You
On Fri, Dec 19, 2014 at 12:19 AM, Michael Paquier
wrote:
> On Thu, Dec 18, 2014 at 5:27 PM, Fujii Masao wrote:
>> Thanks!
> Thanks for your input.
>
>> +else
>> +memcpy(compression_scratch, page, page_len);
>>
>> I don't think the block image needs to be copied
Hello,
>Updated patches addressing all those things are attached.
Below are some performance numbers using latest patch
20141219_fpw_compression_v9 . Compression looks promising with reduced
impact on CPU usage, tps and runtime.
pgbench command : pgbench -c 16 -j 16 -r -t 25 -M prepared
To
On Thu, Dec 18, 2014 at 5:27 PM, Fujii Masao wrote:
> Thanks!
Thanks for your input.
> +else
> +memcpy(compression_scratch, page, page_len);
>
> I don't think the block image needs to be copied to scratch buffer here.
> We can try to compress the "page" directl
On Thu, Dec 18, 2014 at 7:31 PM, Rahila Syed wrote:
>>Isn't it better to allocate the memory for compression_scratch in
>>InitXLogInsert()
>>like hdr_scratch?
>
> I think making compression_scratch a statically allocated global variable
> is the result of following discussion earlier,
> http://ww
On Thu, Dec 18, 2014 at 7:31 PM, Rahila Syed wrote:
>>Isn't it better to allocate the memory for compression_scratch in
>>InitXLogInsert()
>>like hdr_scratch?
>
> I think making compression_scratch a statically allocated global variable
> is the result of following discussion earlier,
>
> http://
>Isn't it better to allocate the memory for compression_scratch in
>InitXLogInsert()
>like hdr_scratch?
I think making compression_scratch a statically allocated global variable
is the result of following discussion earlier,
http://www.postgresql.org/message-id/ca+tgmoaznbuwnls4bpwyqgqteeznoavy
On Thu, Dec 18, 2014 at 2:21 PM, Michael Paquier
wrote:
>
>
> On Wed, Dec 17, 2014 at 11:33 PM, Rahila Syed
> wrote:
>>
>> I had a look at code. I have few minor points,
>
> Thanks!
>
>> + bkpb.fork_flags |= BKPBLOCK_HAS_IMAGE;
>> +
>> + if (is_compressed)
>> {
>>
On Wed, Dec 17, 2014 at 11:33 PM, Rahila Syed
wrote:
> I had a look at code. I have few minor points,
>
Thanks!
+ bkpb.fork_flags |= BKPBLOCK_HAS_IMAGE;
> +
> + if (is_compressed)
> {
> - rdt_datas_last->data = page;
> - rdt_datas_last-
On Thu, Dec 18, 2014 at 1:05 PM, Fujii Masao wrote:
> On Wed, Dec 17, 2014 at 1:34 AM, Michael Paquier
> wrote:
> I think that neither pg_control nor xl_parameter_change need to have the info
> about WAL compression because each backup block has that entry.
>
> Will review the remaining part late
On Wed, Dec 17, 2014 at 1:34 AM, Michael Paquier
wrote:
>
>
> On Wed, Dec 17, 2014 at 12:00 AM, Michael Paquier
> wrote:
>>
>> Actually, the original length of the compressed block in saved in
>> PGLZ_Header, so if we are fine to not check the size of the block
>> decompressed when decoding WAL w
Hello,
>Patches as well as results are attached.
I had a look at code. I have few minor points,
+ bkpb.fork_flags |= BKPBLOCK_HAS_IMAGE;
+
+ if (is_compressed)
{
- rdt_datas_last->data = page;
- rdt_datas_last->len = BLCKSZ;
+
On Wed, Dec 17, 2014 at 12:12 AM, Merlin Moncure wrote:
> Compressing *one* file with lz4 and a quick/n/dirty plgz i hacked out
> of the source (borrowing heavily from
> https://github.com/maropu/pglz_bench/blob/master/pglz_bench.cpp), I
> tested the results:
>
> lz4 real time: 0m0.032s
> pglz
On Wed, Dec 17, 2014 at 12:00 AM, Michael Paquier wrote:
>
> Actually, the original length of the compressed block in saved in
> PGLZ_Header, so if we are fine to not check the size of the block
> decompressed when decoding WAL we can do without the hole filled with
> zeros, and use only 1 bit to
On Mon, Dec 15, 2014 at 5:37 PM, Michael Paquier
wrote:
> On Tue, Dec 16, 2014 at 5:14 AM, Merlin Moncure wrote:
>> OTOH, Our built in compressor as we all know is a complete dog in
>> terms of cpu when stacked up against some more modern implementations.
>> All that said, as long as there is a c
On Tue, Dec 16, 2014 at 11:30 PM, Michael Paquier wrote:
>
>
>
> On Tue, Dec 16, 2014 at 11:24 PM, Alvaro Herrera > wrote:
>>
>> Michael Paquier wrote:
>>
>> > And here are attached fresh patches reducing the WAL record size to
>> what it
>> > is in head when the compression switch is off. Lookin
On Tue, Dec 16, 2014 at 11:24 PM, Alvaro Herrera
wrote:
>
> Michael Paquier wrote:
>
> > And here are attached fresh patches reducing the WAL record size to what
> it
> > is in head when the compression switch is off. Looking at the logic in
> > xlogrecord.h, the block header stores the hole lengt
Michael Paquier wrote:
> And here are attached fresh patches reducing the WAL record size to what it
> is in head when the compression switch is off. Looking at the logic in
> xlogrecord.h, the block header stores the hole length and hole offset. I
> changed that a bit to store the length of raw b
On Tue, Dec 16, 2014 at 8:35 AM, Michael Paquier
wrote:
> On Tue, Dec 16, 2014 at 3:46 AM, Robert Haas
wrote:
>> On Sat, Dec 13, 2014 at 9:36 AM, Michael Paquier
>> wrote:
>>> Something to be aware of btw is that this patch introduces an
>>> additional 8 bytes per block image in WAL as it contai
On Tue, Dec 16, 2014 at 5:14 AM, Merlin Moncure wrote:
> OTOH, Our built in compressor as we all know is a complete dog in
> terms of cpu when stacked up against some more modern implementations.
> All that said, as long as there is a clean path to migrating to
> another compression alg should one
On Tue, Dec 16, 2014 at 3:46 AM, Robert Haas wrote:
> On Sat, Dec 13, 2014 at 9:36 AM, Michael Paquier
> wrote:
>> Something to be aware of btw is that this patch introduces an
>> additional 8 bytes per block image in WAL as it contains additional
>> information to control the compression. In thi
On Fri, Dec 12, 2014 at 8:27 AM, Andres Freund wrote:
> On 2014-12-12 09:24:27 -0500, Bruce Momjian wrote:
>> On Fri, Dec 12, 2014 at 03:22:24PM +0100, Andres Freund wrote:
>> > > Well, the larger question is why wouldn't we just have the user compress
>> > > the entire WAL file before archiving -
On Sat, Dec 13, 2014 at 9:36 AM, Michael Paquier
wrote:
> Something to be aware of btw is that this patch introduces an
> additional 8 bytes per block image in WAL as it contains additional
> information to control the compression. In this case this is the
> uint16 compress_len present in XLogReco
Note: this patch has been moved to CF 2014-12 and I marked myself as
an author if that's fine... I've finished by being really involved in
that.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailp
On Sun, Dec 14, 2014 at 1:16 PM, Andres Freund wrote:
> On 2014-12-14 09:56:59 +0900, Michael Paquier wrote:
>> On Sun, Dec 14, 2014 at 5:45 AM, Simon Riggs wrote:
>> > On 13 December 2014 at 14:36, Michael Paquier
>> > wrote:
>> >
>> >> Something to be aware of btw is that this patch introduce
On 2014-12-14 09:56:59 +0900, Michael Paquier wrote:
> On Sun, Dec 14, 2014 at 5:45 AM, Simon Riggs wrote:
> > On 13 December 2014 at 14:36, Michael Paquier
> > wrote:
> >
> >> Something to be aware of btw is that this patch introduces an
> >> additional 8 bytes per block image in WAL as it cont
On Sun, Dec 14, 2014 at 5:45 AM, Simon Riggs wrote:
> On 13 December 2014 at 14:36, Michael Paquier
> wrote:
>
>> Something to be aware of btw is that this patch introduces an
>> additional 8 bytes per block image in WAL as it contains additional
>> information to control the compression. In thi
On 13 December 2014 at 14:36, Michael Paquier wrote:
> Something to be aware of btw is that this patch introduces an
> additional 8 bytes per block image in WAL as it contains additional
> information to control the compression. In this case this is the
> uint16 compress_len present in XLogRecord
On Fri, Dec 12, 2014 at 11:50 PM, Michael Paquier
wrote:
>
>
> On Wed, Dec 10, 2014 at 11:25 PM, Bruce Momjian wrote:
>>
>> On Wed, Dec 10, 2014 at 07:40:46PM +0530, Rahila Syed wrote:
>> > The tests ran for around 30 mins.Manual checkpoint was run before each
>> > test.
>> >
>> > Compression W
On 12 December 2014 at 21:40, Robert Haas wrote:
> On Fri, Dec 12, 2014 at 1:51 PM, Simon Riggs wrote:
>> What I don't understand is why we aren't working on double buffering,
>> since that cost would be paid in a background process and would be
>> evenly spread out across a checkpoint. Plus we'd
On Fri, Dec 12, 2014 at 7:25 PM, Michael Paquier
wrote:
> On Sat, Dec 13, 2014 at 1:08 AM, Robert Haas wrote:
>> On Fri, Dec 12, 2014 at 10:04 AM, Andres Freund wrote:
Note that autovacuum and fsync are off.
=# select phase, user_diff, system_diff,
pg_size_pretty(pre_update - pre_
On Sat, Dec 13, 2014 at 1:08 AM, Robert Haas wrote:
> On Fri, Dec 12, 2014 at 10:04 AM, Andres Freund wrote:
>>> Note that autovacuum and fsync are off.
>>> =# select phase, user_diff, system_diff,
>>> pg_size_pretty(pre_update - pre_insert),
>>> pg_size_pretty(post_update - pre_update) from resu
On Fri, Dec 12, 2014 at 1:51 PM, Simon Riggs wrote:
> What I don't understand is why we aren't working on double buffering,
> since that cost would be paid in a background process and would be
> evenly spread out across a checkpoint. Plus we'd be able to remove
> FPWs altogether, which is like 100
On 12 December 2014 at 18:04, Bruce Momjian wrote:
> Well, it seems we need to see some actual cases where compression does
> help before moving forward. I thought Amit had some amazing numbers for
> WAL compression --- has that changed?
For background processes, like VACUUM, then WAL compressi
On Fri, Dec 12, 2014 at 05:19:42PM +0100, Andres Freund wrote:
> On 2014-12-12 11:15:46 -0500, Robert Haas wrote:
> > On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund wrote:
> > > On 2014-12-12 11:08:52 -0500, Robert Haas wrote:
> > >> Unless I'm missing something, this test is showing that FPW
> >
On 2014-12-12 11:15:46 -0500, Robert Haas wrote:
> On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund wrote:
> > On 2014-12-12 11:08:52 -0500, Robert Haas wrote:
> >> Unless I'm missing something, this test is showing that FPW
> >> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
> >>
On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund wrote:
> On 2014-12-12 11:08:52 -0500, Robert Haas wrote:
>> Unless I'm missing something, this test is showing that FPW
>> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
>> against master. And compressing the whole record saves a f
On 2014-12-12 11:08:52 -0500, Robert Haas wrote:
> Unless I'm missing something, this test is showing that FPW
> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
> against master. And compressing the whole record saves a further 1MB
> of WAL for a further 13.39 seconds of CPU time.
1 - 100 of 257 matches
Mail list logo
