Re: libpq compression

On 2/9/19 3:02 PM, Konstantin Knizhnik wrote: > > > On 09.02.2019 1:38, Tomas Vondra wrote: >> On 2/8/19 11:10 PM, Konstantin Knizhnik wrote: >>> >>> On 08.02.2019 21:57, Andres Freund wrote: On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote: > Frankly speaking, I do not think

Re: libpq compression

On 2/9/19 3:14 PM, Andres Freund wrote: > Hi, > > On 2019-02-08 23:38:12 +0100, Tomas Vondra wrote: >> On 2/8/19 11:10 PM, Konstantin Knizhnik wrote: >>> Does it mean that it is necessary to support multiple compression >>> algorithms and make it possible to perform switch between them at >>>

Re: libpq compression

Hi, On 2019-02-08 23:38:12 +0100, Tomas Vondra wrote: > On 2/8/19 11:10 PM, Konstantin Knizhnik wrote: > > Does it mean that it is necessary to support multiple compression > > algorithms and make it possible to perform switch between them at > > runtime? > > IMHO the negotiation should happen

Re: libpq compression

On 09.02.2019 1:38, Tomas Vondra wrote: On 2/8/19 11:10 PM, Konstantin Knizhnik wrote: On 08.02.2019 21:57, Andres Freund wrote: On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote: Frankly speaking, I do not think that such flexibility in choosing compression algorithms is really

Re: libpq compression

On 2/8/19 11:10 PM, Konstantin Knizhnik wrote: > > > On 08.02.2019 21:57, Andres Freund wrote: >> On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote: >>> Frankly speaking, I do not think that such flexibility in choosing >>> compression algorithms is really needed. >>> I do not expect that

Re: libpq compression

On 08.02.2019 21:57, Andres Freund wrote: On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote: Frankly speaking, I do not think that such flexibility in choosing compression algorithms is really needed. I do not expect that there will be many situations where old client has to

Re: libpq compression

On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote: > Frankly speaking, I do not think that such flexibility in choosing > compression algorithms is really needed. > I do not expect that there will be many situations where old client has to > communicate with new server or visa versa. > In

Re: libpq compression

On 08.02.2019 19:26, Robbie Harwood wrote: Konstantin Knizhnik writes: On 08.02.2019 10:01, Iwata, Aya wrote: I fixed all issues you have reported except using list of supported compression algorithms. Sure. I confirmed that. It will require extra round of communication between client

Re: libpq compression

On 08.02.2019 12:33, Daniel Gustafsson wrote: On 8 Feb 2019, at 10:15, Konstantin Knizhnik wrote: Frankly speaking, I do not think that such flexibility in choosing compression algorithms is really needed. I do not expect that there will be many situations where old client has to

Re: libpq compression

Konstantin Knizhnik writes: > On 08.02.2019 10:01, Iwata, Aya wrote: > >>> I fixed all issues you have reported except using list of supported >>> compression algorithms. >> >> Sure. I confirmed that. >> >>> It will require extra round of communication between client and >>> server to make a

Re: libpq compression

> On 8 Feb 2019, at 10:15, Konstantin Knizhnik > wrote: > Frankly speaking, I do not think that such flexibility in choosing > compression algorithms is really needed. > I do not expect that there will be many situations where old client has to > communicate with new server or visa versa. >

Re: libpq compression

On 08.02.2019 10:01, Iwata, Aya wrote: Hi, I am sorry for my late reply. I fixed all issues you have reported except using list of supported compression algorithms. Sure. I confirmed that. It will require extra round of communication between client and server to make a decision about

Re: libpq compression

On 08.02.2019 10:14, Andres Freund wrote: Hi, On 2018-03-30 15:53:39 +0300, Konstantin Knizhnik wrote: Taken in account that vulnerability was found in SSL compression and so SSLComppression is considered to be deprecated and insecure

Re: libpq compression

Hi, On 2018-03-30 15:53:39 +0300, Konstantin Knizhnik wrote: > Taken in account that vulnerability was found in SSL compression and so > SSLComppression is considered to be deprecated and insecure > (http://www.postgresql-archive.org/disable-SSL-compression-td6010072.html), > it will be nice to

Re: libpq compression

Hi, On 2019-02-08 07:01:01 +, Iwata, Aya wrote: > > I still not sure whether it is good idea to make it possible to user to > > explicitly specify compression algorithm. > > Right now used streaming compression algorithm is hardcoded and depends on > > --use-zstd ort --use-zlib configuration

RE: libpq compression

Hi, I am sorry for my late reply. > I fixed all issues you have reported except using list of supported > compression > algorithms. Sure. I confirmed that. > It will require extra round of communication between client and server to > make a decision about used compression algorithm. In

Re: libpq compression

On 09.01.2019 13:25, Iwata, Aya wrote: Hi, I agree with the critiques from Robbie Harwood and Michael Paquier that the way in that compression is being hooked into the existing architecture looks like a kludge. I'm not sure I know exactly how it should be done, but the current approach

Re: libpq compression

> On Wed, Jan 9, 2019 at 11:25 AM Iwata, Aya wrote: > > This thread seems to be stopped. > In last e-mail, Dmitry suggest to update the patch that implements the > function in another way, and as far as I saw, he has not updated patch yet. Yep, I'm still working on it, hopefully I can submit

Re: libpq compression

> On Mon, Aug 13, 2018 at 8:48 PM Robert Haas wrote: > > I agree with the critiques from Robbie Harwood and Michael Paquier > that the way in that compression is being hooked into the existing > architecture looks like a kludge. I'm not sure I know exactly how it > should be done, but the

Re: libpq compression

On 01.10.2018 09:49, Michael Paquier wrote: On Mon, Aug 20, 2018 at 06:00:39PM +0300, Konstantin Knizhnik wrote: New version of the patch is attached: I removed -Z options form pgbench and psql and add checking that server and client are implementing the same compression algorithm. The patch

Re: libpq compression

On Mon, Aug 20, 2018 at 06:00:39PM +0300, Konstantin Knizhnik wrote: > New version of the patch is attached: I removed -Z options form pgbench and > psql and add checking that server and client are implementing the same > compression algorithm. The patch had no reviews, and does not apply

Re: libpq compression

On 13.08.2018 23:06, Andrew Dunstan wrote: On 08/13/2018 02:47 PM, Robert Haas wrote: On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan wrote: This thread appears to have gone quiet. What concerns me is that there appears to be substantial disagreement between the author and the reviewers.

Re: libpq compression

Hi Robert, First of all thank you for review and your time spent on analyzing this patch. My comments are inside. On 13.08.2018 21:47, Robert Haas wrote: On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan wrote: This thread appears to have gone quiet. What concerns me is that there appears to

Re: libpq compression

On 08/13/2018 02:47 PM, Robert Haas wrote: On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan wrote: This thread appears to have gone quiet. What concerns me is that there appears to be substantial disagreement between the author and the reviewers. Since the last thing was this new patch it

Re: libpq compression

On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan wrote: > This thread appears to have gone quiet. What concerns me is that there > appears to be substantial disagreement between the author and the reviewers. > Since the last thing was this new patch it should really have been put back > into

Re: libpq compression

On 06/25/2018 05:32 AM, Konstantin Knizhnik wrote: On 18.06.2018 23:34, Robbie Harwood wrote: ### Documentation!  You're going to need it.  There needs to be enough around for other people to implement the protocol (or if you prefer, enough for us to debug the protocol as it exists). In

Re: libpq compression

On 18.06.2018 23:34, Robbie Harwood wrote: ### Documentation! You're going to need it. There needs to be enough around for other people to implement the protocol (or if you prefer, enough for us to debug the protocol as it exists). In conjunction with that, please add information on how

Re: libpq compression

On 22.06.2018 20:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 22.06.2018 18:59, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik

Re: libpq compression

Konstantin Knizhnik writes: > On 22.06.2018 18:59, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >>> On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: > On 21.06.2018 17:56, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >>> On 20.06.2018

Re: libpq compression

On 22.06.2018 19:05, Nico Williams wrote: On Fri, Jun 22, 2018 at 10:18:12AM +0300, Konstantin Knizhnik wrote: On 22.06.2018 00:34, Nico Williams wrote: So I think you just have to have lengths. Now, this being about compression, I understand that you might now want to have 4-byte lengths,

Re: libpq compression

On 22.06.2018 18:59, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik

Re: libpq compression

On Fri, Jun 22, 2018 at 10:18:12AM +0300, Konstantin Knizhnik wrote: > On 22.06.2018 00:34, Nico Williams wrote: > >So I think you just have to have lengths. > > > >Now, this being about compression, I understand that you might now want > >to have 4-byte lengths, especially given that most

Re: libpq compression

Konstantin Knizhnik writes: > On 21.06.2018 20:14, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >>> On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: > On 20.06.2018 23:34, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >> >> Well,

Re: libpq compression

On 22.06.2018 00:34, Nico Williams wrote: On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: Well, that's a design decision you've made. You could put lengths on chunks that are sent out - then you'd

Re: libpq compression

On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: Well, that's a design decision you've made. You could put lengths on

Re: libpq compression

On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote: > On 20.06.2018 23:34, Robbie Harwood wrote: > >Konstantin Knizhnik writes: > >Well, that's a design decision you've made. You could put lengths on > >chunks that are sent out - then you'd know exactly how much is needed. >

Re: libpq compression

Konstantin Knizhnik writes: > On 21.06.2018 17:56, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >>> On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: Well, that's a design decision you've made. You could put lengths on chunks that are sent out

Re: libpq compression

On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: My idea was the following: client want to use compression. But server may reject this attempt (for any reasons: it doesn't support it, has no

Re: libpq compression

Konstantin Knizhnik writes: > On 20.06.2018 23:34, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >> >> >> My idea was the following: client want to use compression. But server >> may reject this attempt (for any reasons: it doesn't support it, has >> no proper compression library, do not

Re: libpq compression

On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: My idea was the following: client want to use compression. But server may reject this attempt (for any reasons: it doesn't support it, has no proper compression library, do not want to spend CPU for decompression,...)

Re: libpq compression

Konstantin Knizhnik writes: > On 20.06.2018 00:04, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >>> On 18.06.2018 23:34, Robbie Harwood wrote: >>> I also don't like that you've injected into the *startup* path - before authentication takes place. Fundamentally, authentication

Re: libpq compression

On 20.06.2018 00:04, Robbie Harwood wrote: Konstantin Knizhnik writes: On 18.06.2018 23:34, Robbie Harwood wrote: I also don't like that you've injected into the *startup* path - before authentication takes place. Fundamentally, authentication (if it happens) consists of exchanging some

Re: libpq compression

Konstantin Knizhnik writes: > On 18.06.2018 23:34, Robbie Harwood wrote: > >> I also don't like that you've injected into the *startup* path - >> before authentication takes place. Fundamentally, authentication (if >> it happens) consists of exchanging some combination of short strings >>

Re: libpq compression

On 18.06.2018 23:34, Robbie Harwood wrote: t Konstantin Knizhnik writes: On 06.06.2018 02:03, Thomas Munro wrote: On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik wrote: Thank you for review. Updated version of the patch fixing all reported problems is attached. Small problem on

Re: libpq compression

tKonstantin Knizhnik writes: > On 06.06.2018 02:03, Thomas Munro wrote: >> On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik >> wrote: >>> Thank you for review. Updated version of the patch fixing all reported >>> problems is attached. >> Small problem on Windows[1]: >> >>

Re: libpq compression

On 7 June 2018 at 04:01, Peter Eisentraut wrote: > On 6/6/18 13:20, Konstantin Knizhnik wrote: > > Well, psql really allows to specify complete connection string with -d > > options (although it is not mentioned in help). > > But still I think that it is inconvenient to require user to write > >

Re: libpq compression

On 6/6/18 13:20, Konstantin Knizhnik wrote: > Well, psql really allows to specify complete connection string with -d > options (although it is not mentioned in help). > But still I think that it is inconvenient to require user to write > complete connection string to be able to specify

Re: libpq compression

On 06/06/2018 10:20 AM, Konstantin Knizhnik wrote: Well, psql really allows to specify complete connection string with -d options (although it is not mentioned in help). But still I think that it is inconvenient to require user to write complete connection string to be able to specify

Re: libpq compression

On 06.06.2018 19:33, Konstantin Knizhnik wrote: On 05.06.2018 20:06, Peter Eisentraut wrote: On 6/5/18 03:09, Michael Paquier wrote: I just had a quick look at this patch, lured by the smell of your latest messages...  And it seems to me that this patch needs a heavy amount of work as

Re: libpq compression

On 05.06.2018 20:06, Peter Eisentraut wrote: On 6/5/18 03:09, Michael Paquier wrote: I just had a quick look at this patch, lured by the smell of your latest messages... And it seems to me that this patch needs a heavy amount of work as presented. There are a couple of things which are not

Re: libpq compression

On 06.06.2018 02:03, Thomas Munro wrote: On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik wrote: Thank you for review. Updated version of the patch fixing all reported problems is attached. Small problem on Windows[1]: C:\projects\postgresql\src\include\common/zpq_stream.h(17): error

Re: libpq compression

On 06.06.2018 10:53, Michael Paquier wrote: On Tue, Jun 05, 2018 at 06:58:42PM +0300, Konstantin Knizhnik wrote: I have considered this patch mostly as prototype to estimate efficiency of libpq protocol compression and compare it with SSL compression. So I agree with you that there are a lot

Re: libpq compression

On Tue, Jun 05, 2018 at 06:58:42PM +0300, Konstantin Knizhnik wrote: > I have considered this patch mostly as prototype to estimate efficiency of > libpq protocol compression and compare it with SSL compression. > So I agree with you that there are a lot of things which should be > improved.

Re: libpq compression

On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik wrote: > Thank you for review. Updated version of the patch fixing all reported > problems is attached. Small problem on Windows[1]: C:\projects\postgresql\src\include\common/zpq_stream.h(17): error C2143: syntax error : missing ')' before

Re: libpq compression

On 5 June 2018 at 13:06, Peter Eisentraut wrote: > On 6/5/18 03:09, Michael Paquier wrote: > > I just had a quick look at this patch, lured by the smell of your latest > > messages... And it seems to me that this patch needs a heavy amount of > > work as presented. There are a couple of things

Re: libpq compression

On 6/5/18 03:09, Michael Paquier wrote: > I just had a quick look at this patch, lured by the smell of your latest > messages... And it seems to me that this patch needs a heavy amount of > work as presented. There are a couple of things which are not really > nice, like forcing the presentation

Re: libpq compression

On 05.06.2018 10:09, Michael Paquier wrote: On Tue, Jun 05, 2018 at 06:04:21PM +1200, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik Speaking of configuration, are you planning to support multiple compression libraries at the same time? It looks like the current

Re: libpq compression

On 05.06.2018 09:04, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik wrote: Concerning specification of compression level: I have made many experiments with different data sets and both zlib/zstd and in both cases using compression level higher than default doesn't

Re: libpq compression

On 05.06.2018 08:26, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik wrote: Thank you for this notice. Updated and rebased patch is attached. Hi Konstantin, Seems very useful. +1. + rc = inflate(>rx, Z_SYNC_FLUSH); + if (rc != Z_OK) + { + return

Re: libpq compression

On Tue, Jun 05, 2018 at 06:04:21PM +1200, Thomas Munro wrote: > On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik > Speaking of configuration, are you planning to support multiple > compression libraries at the same time? It looks like the current > patch implicitly requires client and server

Re: libpq compression

On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik wrote: > Concerning specification of compression level: I have made many experiments > with different data sets and both zlib/zstd and in both cases using > compression level higher than default doesn't cause some noticeable increase > of

Re: libpq compression

On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik wrote: > Thank you for this notice. > Updated and rebased patch is attached. Hi Konstantin, Seems very useful. +1. + rc = inflate(>rx, Z_SYNC_FLUSH); + if (rc != Z_OK) + { + return ZPQ_DECOMPRESS_ERROR; + } Does this actually guarantee

Re: libpq compression

On 16.05.2018 18:09, Grigory Smolkin wrote: Hello! I have noticed that psql --help lack -Z|--compression option. Also it would be nice to have option like --compression-level in psql and pgbench. Thank you for this notice. Updated and rebased patch is attached. Concerning specification of

Re: libpq compression

Hello! I have noticed that psql --help lack -Z|--compression option. Also it would be nice to have option like --compression-level in psql and pgbench. On 03/30/2018 03:53 PM, Konstantin Knizhnik wrote: Hi hackers, One of our customers was managed to improve speed about 10 times by using

Re: libpq compression

2018-05-15 9:53 GMT-03:00 Konstantin Knizhnik : > Looks like community is not so interested in this patch. Frankly speaking I > do not understand why. > AFAICS the lack of replies is due to feature freeze. I'm pretty sure people are interested in this topic (at least I

Re: libpq compression

On 15 May 2018 at 21:36, Andrew Dunstan wrote: > > >>> > To use zstd compression, Postgres should be configured with >>> --with-zstd. Otherwise compression will use zlib unless it is disabled by >>> --without-zlib option. >>> > I have added compression=on/off

Re: libpq compression

On 05/15/2018 08:53 AM, Konstantin Knizhnik wrote: On 15.05.2018 13:23, Dmitry Dolgov wrote: > On 30 March 2018 at 14:53, Konstantin Knizhnik > wrote: > Hi hackers, > One of our customers was managed to improve speed about 10

Re: libpq compression

On 15.05.2018 13:23, Dmitry Dolgov wrote: > On 30 March 2018 at 14:53, Konstantin Knizhnik > wrote: > Hi hackers, > One of our customers was managed to improve speed about 10 times by using SSL compression for the system where

Re: libpq compression

> On 30 March 2018 at 14:53, Konstantin Knizhnik wrote: > Hi hackers, > One of our customers was managed to improve speed about 10 times by using SSL compression for the system where client and servers are located in different geographical regions > and query results

<    1   2   3