Re: [HACKERS] Custom compression methods

On Mon, 29 Jan 2018 17:29:29 +0300 Ildar Musin wrote: > > Patch applies cleanly, builds without any warnings, documentation > builds ok, all tests pass. > > A remark for the committers. The patch is quite big, so I really wish > more reviewers looked into it for more

Re: [HACKERS] Custom compression methods

Hello Ildus, On 29.01.2018 14:44, Ildus Kurbangaliev wrote: Thanks! Attached new version of the patch. Patch applies cleanly, builds without any warnings, documentation builds ok, all tests pass. A remark for the committers. The patch is quite big, so I really wish more reviewers looked

Re: [HACKERS] Custom compression methods

Hello Ildus, I continue reviewing your patch. Here are some thoughts. 1. When I set column storage to EXTERNAL then I cannot set compression. Seems reasonable: create table test(id serial, msg text); alter table test alter column msg set storage external; alter table test alter column msg set

Re: [HACKERS] Custom compression methods

Hello Ildus, On 23.01.2018 16:04, Ildus Kurbangaliev wrote: On Mon, 22 Jan 2018 23:26:31 +0300 Ildar Musin wrote: Thanks for review! Attached new version of the patch. Fixed few bugs, added more documentation and rebased to current master. You need to rebase to the

Re: [HACKERS] Custom compression methods

Hello Ildus, 15/01/2018 00:49, Ildus Kurbangaliev пишет: > Attached a new version of the patch. Main changes: > > * compression as an access method > * pglz as default compression access method. > * PRESERVE syntax for tables rewrite control. > * pg_upgrade fixes > * support partitioned tables.

Re: [HACKERS] Custom compression methods

On Mon, Dec 18, 2017 at 10:43 AM, Tomas Vondra wrote: > I personally am not quite convinced about that, for the reason I tried > to explain in my previous messages. I see it as a poor alternative to > compression built into the data type. I do like the idea of

Re: [HACKERS] Custom compression methods

On 12/17/2017 04:32 AM, Robert Haas wrote: > On Thu, Dec 14, 2017 at 12:23 PM, Tomas Vondra > wrote: >> Can you give an example of such algorithm? Because I haven't seen such >> example, and I find arguments based on hypothetical compression methods >> somewhat

Re: [HACKERS] Custom compression methods

On Thu, 14 Dec 2017 10:29:10 -0500 Robert Haas wrote: > On Wed, Dec 13, 2017 at 7:18 AM, Ildus Kurbangaliev > wrote: > > Since we agreed on ALTER syntax, i want to clear things about > > CREATE. Should it be CREATE ACCESS METHOD .. TYPE

Re: [HACKERS] Custom compression methods

On Thu, Dec 14, 2017 at 12:23 PM, Tomas Vondra wrote: > Can you give an example of such algorithm? Because I haven't seen such > example, and I find arguments based on hypothetical compression methods > somewhat suspicious. > > FWIW I'm not against considering such

Re: [HACKERS] Custom compression methods

On 12/14/2017 04:21 PM, Robert Haas wrote: > On Wed, Dec 13, 2017 at 5:10 AM, Tomas Vondra > wrote: >>> 2. If several data types can benefit from a similar approach, it has >>> to be separately implemented for each one. >> >> I don't think the current solution

Re: [HACKERS] Custom compression methods

On Wed, Dec 13, 2017 at 7:18 AM, Ildus Kurbangaliev wrote: > Since we agreed on ALTER syntax, i want to clear things about CREATE. > Should it be CREATE ACCESS METHOD .. TYPE СOMPRESSION or CREATE > COMPRESSION METHOD? I like the access method approach, and it >

Re: [HACKERS] Custom compression methods

On Wed, Dec 13, 2017 at 5:10 AM, Tomas Vondra wrote: >> 2. If several data types can benefit from a similar approach, it has >> to be separately implemented for each one. > > I don't think the current solution improves that, though. If you want to > exploit internal

Re: [HACKERS] Custom compression methods

On 12/13/2017 05:55 PM, Alvaro Herrera wrote: > Tomas Vondra wrote: > >> On 12/13/2017 01:54 AM, Robert Haas wrote: > >>> 3. Compression is only applied to large-ish values. If you are just >>> making the data type representation more compact, you probably want to >>> apply the new

Re: [HACKERS] Custom compression methods

On Tue, 12 Dec 2017 15:52:01 -0500 Robert Haas wrote: > > Yes. I wonder if \d or \d+ can show it somehow. > Yes, in current version of the patch, \d+ shows current compression. It can be extended to show a list of current compression methods. Since we agreed on ALTER

Re: [HACKERS] Custom compression methods

On Tue, Dec 12, 2017 at 5:07 PM, Tomas Vondra wrote: >> I definitely think there's a place for compression built right into >> the data type. I'm still happy about commit >> 145343534c153d1e6c3cff1fa1855787684d9a38 -- although really, more >> needs to be done there.

Re: [HACKERS] Custom compression methods

On 12/12/2017 04:33 PM, Robert Haas wrote: > you want to upgrade your existing tables to use lz4 rather than pglz, > you can change the compression option for those columns to COMPRESS > lz4 PRESERVE pglz if you want to do it incrementally or just COMPRESS This is a thread I've only been

Re: [HACKERS] Custom compression methods

On 12/12/2017 10:33 PM, Robert Haas wrote: > On Mon, Dec 11, 2017 at 2:53 PM, Tomas Vondra > wrote: >> But let me play the devil's advocate for a while and question the >> usefulness of this approach to compression. Some of the questions were >> mentioned in the

Re: [HACKERS] Custom compression methods

On Mon, Dec 11, 2017 at 2:53 PM, Tomas Vondra wrote: > But let me play the devil's advocate for a while and question the > usefulness of this approach to compression. Some of the questions were > mentioned in the thread before, but I don't think they got the

Re: [HACKERS] Custom compression methods

On Mon, Dec 11, 2017 at 1:06 PM, Alexander Korotkov wrote: > OK, but NOTICE that presumably unexpected table rewrite takes place could be > still useful. I'm not going to complain too much about that, but I think that's mostly a failure of expectation rather than a

Re: [HACKERS] Custom compression methods

On Tue, Dec 12, 2017 at 6:07 PM, Alexander Korotkov wrote: > Hi! > > Let me add my two cents too. > > On Tue, Dec 12, 2017 at 2:41 PM, Ildus Kurbangaliev > wrote: >> >> On Mon, 11 Dec 2017 20:53:29 +0100 Tomas Vondra >>

Re: [HACKERS] Custom compression methods

Hi! Let me add my two cents too. On Tue, Dec 12, 2017 at 2:41 PM, Ildus Kurbangaliev < i.kurbangal...@postgrespro.ru> wrote: > On Mon, 11 Dec 2017 20:53:29 +0100 Tomas Vondra < > tomas.von...@2ndquadrant.com> wrote: > > Replacing the algorithm used to compress all varlena values (in a way > >

Re: [HACKERS] Custom compression methods

On Mon, 11 Dec 2017 20:53:29 +0100 Tomas Vondra wrote: > But let me play the devil's advocate for a while and question the > usefulness of this approach to compression. Some of the questions were > mentioned in the thread before, but I don't think they got the >

Re: [HACKERS] Custom compression methods

Hi, I see there's an ongoing discussion about the syntax and ALTER TABLE behavior when changing a compression method for a column. So the patch seems to be on the way to be ready in the January CF, I guess. But let me play the devil's advocate for a while and question the usefulness of this

Re: [HACKERS] Custom compression methods

On Mon, Dec 11, 2017 at 8:46 PM, Robert Haas wrote: > On Mon, Dec 11, 2017 at 12:41 PM, Alexander Korotkov > wrote: > > Thus, in your example if user would like to further change awesome > > compression for evenbetter compression, she should

Re: [HACKERS] Custom compression methods

On Mon, Dec 11, 2017 at 12:41 PM, Alexander Korotkov wrote: > Thus, in your example if user would like to further change awesome > compression for evenbetter compression, she should write. > > SET COMPRESSION evenbetter PRESERVE pglz, awesome; -- full list of previous >

Re: [HACKERS] Custom compression methods

On Mon, Dec 11, 2017 at 8:25 PM, Robert Haas wrote: > On Mon, Dec 11, 2017 at 7:55 AM, Ildus Kurbangaliev > wrote: > > On Fri, 8 Dec 2017 15:12:42 -0500 > > Robert Haas wrote: > >> Maybe a better idea is ALTER COLUMN

Re: [HACKERS] Custom compression methods

On Mon, Dec 11, 2017 at 7:55 AM, Ildus Kurbangaliev wrote: > On Fri, 8 Dec 2017 15:12:42 -0500 > Robert Haas wrote: >> Maybe a better idea is ALTER COLUMN SET COMPRESSION x1, x2, x3 ... >> meaning that x1 is the default for new tuples but x2,

Re: [HACKERS] Custom compression methods

On Fri, 8 Dec 2017 15:12:42 -0500 Robert Haas wrote: > > Maybe a better idea is ALTER COLUMN SET COMPRESSION x1, x2, x3 ... > meaning that x1 is the default for new tuples but x2, x3, etc. are > still allowed if present. If you issue a command that only adds > things to

Re: [HACKERS] Custom compression methods

On Wed, Dec 6, 2017 at 10:07 AM, Ildus Kurbangaliev wrote: > On Fri, 1 Dec 2017 21:47:43 +0100 > Tomas Vondra wrote: >> +1 to do the rewrite, just like for other similar ALTER TABLE commands > > Ok. What about the following syntax: > >

Re: [HACKERS] Custom compression methods

On Fri, 1 Dec 2017 21:47:43 +0100 Tomas Vondra wrote: > > +1 to do the rewrite, just like for other similar ALTER TABLE commands Ok. What about the following syntax: ALTER COLUMN DROP COMPRESSION - removes compression from the column with the rewrite and removes

Re: [HACKERS] Custom compression methods

Tomas Vondra wrote: > On 12/01/2017 08:48 PM, Alvaro Herrera wrote: > > Maybe our dependency code needs to be extended in order to support this. > > I think the current logic would drop the column if you were to do "DROP > > COMPRESSION .. CASCADE", but I'm not sure we'd see that as a feature. >

Re: [HACKERS] Custom compression methods

On 12/02/2017 09:38 PM, Andres Freund wrote: > Hi, > > On 2017-12-02 16:04:52 +0100, Tomas Vondra wrote: >> Firstly, it's going to be quite hard (or perhaps impossible) to find an >> algorithm that is "universally better" than pglz. Some algorithms do >> work better for text documents, some for

Re: [HACKERS] Custom compression methods

On 12/02/2017 09:24 PM, konstantin knizhnik wrote: > > On Dec 2, 2017, at 6:04 PM, Tomas Vondra wrote: > >> On 12/01/2017 10:52 PM, Andres Freund wrote: >> ... >> >> Other algorithms (e.g. zstd) got significantly better compression (25%) >> compared to pglz, but in exchange for longer

Re: [HACKERS] Custom compression methods

Hi, On 2017-12-02 16:04:52 +0100, Tomas Vondra wrote: > Firstly, it's going to be quite hard (or perhaps impossible) to find an > algorithm that is "universally better" than pglz. Some algorithms do > work better for text documents, some for binary blobs, etc. I don't > think there's a win-win

Re: [HACKERS] Custom compression methods

On 12/01/2017 10:52 PM, Andres Freund wrote: > On 2017-12-01 16:14:58 -0500, Robert Haas wrote: >> Honestly, if we can give everybody a 4% space reduction by >> switching to lz4, I think that's totally worth doing -- but let's >> not make people choose it, let's make it the default going forward,

Re: [HACKERS] Custom compression methods

On Fri, 1 Dec 2017 16:38:42 -0300 Alvaro Herrera wrote: > > To me it makes sense to say "let's create this method which is for > data compression" (CREATE ACCESS METHOD hyperz TYPE COMPRESSION) > followed by either "let's use this new compression method for the > type

Re: [HACKERS] Custom compression methods

On 2017-12-01 16:14:58 -0500, Robert Haas wrote: > Honestly, if we can give everybody a 4% space reduction by switching > to lz4, I think that's totally worth doing -- but let's not make > people choose it, let's make it the default going forward, and keep > pglz support around so we don't break

Re: [HACKERS] Custom compression methods

On 12/01/2017 08:38 PM, Alvaro Herrera wrote: > Tomas Vondra wrote: > >> On 11/30/2017 09:51 PM, Alvaro Herrera wrote: > >>> Just passing by, but wouldn't this fit in the ACCESS METHOD group of >>> commands? So this could be simplified down to >>> CREATE ACCESS METHOD ts1 TYPE COMPRESSION >>>

Re: [HACKERS] Custom compression methods

On Fri, Dec 1, 2017 at 4:06 PM, Tomas Vondra wrote: >>> I agree with these thoughts in general, but I'm not quite sure >>> what is your conclusion regarding the patch. >> >> I have not reached one. Sometimes I like to discuss problems before >> deciding what I think.

Re: [HACKERS] Custom compression methods

On Fri, Dec 1, 2017 at 2:38 PM, Alvaro Herrera wrote: > In mine, they define how things are accessed (i.e. more general than > what you're thinking). We *currently* use them to store rows [in > indexes], but there is no reason why we couldn't expand that. > > So we group

Re: [HACKERS] Custom compression methods

On 12/01/2017 08:20 PM, Robert Haas wrote: > On Fri, Dec 1, 2017 at 10:18 AM, Tomas Vondra > wrote: >> It has very little impact on this patch, as it has nothing to do with >> columnar storage. That is, each value is compressed independently. > > I understand that

Re: [HACKERS] Custom compression methods

On 12/01/2017 08:48 PM, Alvaro Herrera wrote: > Ildus Kurbangaliev wrote: > >> If the table is big, decompression could take an eternity. That's why i >> decided to only to disable it and the data could be decompressed using >> compression options. >> >> My idea was to keep compression options

Re: [HACKERS] Custom compression methods

On Fri, Dec 1, 2017 at 10:18 AM, Tomas Vondra wrote: > It has very little impact on this patch, as it has nothing to do with > columnar storage. That is, each value is compressed independently. I understand that this patch is not about columnar storage, but I think

Re: [HACKERS] Custom compression methods

On 12/01/2017 03:23 PM, Robert Haas wrote: > On Thu, Nov 30, 2017 at 2:47 PM, Tomas Vondra > wrote: >> OK. I think it's a nice use case (and nice gains on the compression >> ratio), demonstrating the datatype-aware compression. The question is >> why shouldn't this

Re: [HACKERS] Custom compression methods

On Thu, Nov 30, 2017 at 2:47 PM, Tomas Vondra wrote: > OK. I think it's a nice use case (and nice gains on the compression > ratio), demonstrating the datatype-aware compression. The question is > why shouldn't this be built into the datatypes directly? Tomas,

Re: [HACKERS] Custom compression methods

On 11/30/2017 09:51 PM, Alvaro Herrera wrote: > Tomas Vondra wrote: > >> On 11/30/2017 04:20 PM, Ildus Kurbangaliev wrote: > >>> CREATE COMPRESSION METHOD ts1 FOR tsvector HANDLER >>> tsvector_compression_handler; >> >> Understood. Good to know you've considered it, and I agree it doesn't >>

Re: [HACKERS] Custom compression methods

Tomas Vondra wrote: > On 11/30/2017 04:20 PM, Ildus Kurbangaliev wrote: > > CREATE COMPRESSION METHOD ts1 FOR tsvector HANDLER > > tsvector_compression_handler; > > Understood. Good to know you've considered it, and I agree it doesn't > need to be there from the start (which makes the patch

Re: [HACKERS] Custom compression methods

On 11/30/2017 04:20 PM, Ildus Kurbangaliev wrote: > On Thu, 30 Nov 2017 00:30:37 +0100 > Tomas Vondra wrote: > > ... > >> I can imagine other interesting use cases - for example values in >> JSONB columns often use the same "schema" (keys, nesting, ...), so >> can

Re: [HACKERS] Custom compression methods

On Thu, Nov 30, 2017 at 8:30 AM, Tomas Vondra wrote: > On 11/28/2017 02:29 PM, Ildus Kurbangaliev wrote: >> On Mon, 27 Nov 2017 18:20:12 +0100 >> Tomas Vondra wrote: >> >>> I guess the trick might be -DRANDOMIZE_ALLOCATED_MEMORY (I

Re: [HACKERS] Custom compression methods

Hi, On 11/27/2017 04:52 PM, Ildus Kurbangaliev wrote: > ... > > Hi. This looks like a serious bug, but I couldn't reproduce it yet. > Did you upgrade some old database or this bug happened after > insertion of all data to new database? I tried using your 'archie' > tool to download mailing

Re: [HACKERS] Custom compression methods

Hi, I ran into another issue - after inserting some data into a table with a tsvector column (without any compression defined), I can no longer read the data. This is what I get in the console: db=# select max(md5(body_tsvector::text)) from messages; ERROR: cache lookup failed for compression

Re: [HACKERS] Custom compression methods

On Thu, 23 Nov 2017 21:54:32 +0100 Tomas Vondra wrote: > > Hmm, this seems to have fixed it, but only in one direction. Consider > this: > > create table t_pglz (v text); > create table t_lz4 (v text compressed lz4); > > insert into t_pglz select

Re: [HACKERS] Custom compression methods

Hi, On 11/23/2017 10:38 AM, Ildus Kurbangaliev wrote: > On Tue, 21 Nov 2017 18:47:49 +0100 > Tomas Vondra wrote: > >>> >> >> Hmmm, it still doesn't work for me. See this: >> >> test=# create extension pg_lz4 ; >> CREATE EXTENSION >> test=# create

Re: [HACKERS] Custom compression methods

On 11/21/2017 09:28 PM, Ildus K wrote: >> Hmmm, it still doesn't work for me. See this: >> >> test=# create extension pg_lz4 ; >> CREATE EXTENSION >> test=# create table t_lz4 (v text compressed lz4); >> CREATE TABLE >> test=# create table t_pglz (v text); >> CREATE TABLE

Re: [HACKERS] Custom compression methods

On Tue, 21 Nov 2017 18:47:49 +0100 Tomas Vondra wrote: > > I propose to use either > >CompressionMethodOptions (and CompressionMethodRoutine) > > or > >CompressionOptions (and CompressionRoutine) Sounds good, thanks. > > OK. But then I don't

Re: [HACKERS] Custom compression methods

On Mon, 20 Nov 2017 16:29:11 +0100 Tomas Vondra wrote: > On 11/20/2017 04:21 PM, Евгений Шишкин wrote: > > > > > >> On Nov 20, 2017, at 18:18, Tomas Vondra > >> >> > wrote: > >> > >> > >> I

Re: [HACKERS] Custom compression methods

On 11/15/2017 02:13 PM, Robert Haas wrote: > On Wed, Nov 15, 2017 at 4:09 AM, Ildus Kurbangaliev > wrote: >> So in the next version of the patch I can just unlink the options from >> compression methods and dropping compression method will not affect >> already

Re: [HACKERS] Custom compression methods

Hi, On 11/14/2017 02:23 PM, Ildus Kurbangaliev wrote: > > ... > > Attached version 4 of the patch. Fixed pg_upgrade and few other bugs. > I did a review of this today, and I think there are some things that need improvement / fixing. Firstly, some basic comments from just eye-balling the diff,

Re: [HACKERS] Custom compression methods

On Wed, Nov 15, 2017 at 4:09 AM, Ildus Kurbangaliev wrote: > So in the next version of the patch I can just unlink the options from > compression methods and dropping compression method will not affect > already compressed tuples. They still could be decompressed.

<    1   2   3   4