Re: libpq compression

2018-06-22 Thread Konstantin Knizhnik
On 22.06.2018 19:05, Nico Williams wrote: On Fri, Jun 22, 2018 at 10:18:12AM +0300, Konstantin Knizhnik wrote: On 22.06.2018 00:34, Nico Williams wrote: So I think you just have to have lengths. Now, this being about compression, I understand that you might now want to have 4-byte lengths

Re: libpq compression

2018-06-22 Thread Konstantin Knizhnik
On 22.06.2018 18:59, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes

Re: Wrong cost estimation for foreign tables join with use_remote_estimate disabled

2018-06-22 Thread Konstantin Knizhnik
On 22.06.2018 13:30, Ashutosh Bapat wrote: On Fri, Jun 22, 2018 at 11:56 AM, Konstantin Knizhnik wrote: On 21.06.2018 20:08, Tom Lane wrote: Konstantin Knizhnik writes: The following very simple test reduce the problem with wrong cost estimation: create foreign table t1_fdw(x integer, y

Re: WAL prefetch

2018-06-22 Thread Konstantin Knizhnik
On 21.06.2018 19:57, Tomas Vondra wrote: On 06/21/2018 04:01 PM, Konstantin Knizhnik wrote: I continue my experiments with WAL prefetch. I have embedded prefetch in Postgres: now walprefetcher is started together with startup process and is able to help it to speedup recovery. The patch

Re: libpq compression

2018-06-22 Thread Konstantin Knizhnik
On 22.06.2018 00:34, Nico Williams wrote: On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: Well, that's a design decision you've made. You could put lengths on chunks that are sent out -

Re: libpq compression

2018-06-22 Thread Konstantin Knizhnik
On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: Well, that's a design decision you've made. You could put

Re: Wrong cost estimation for foreign tables join with use_remote_estimate disabled

2018-06-21 Thread Konstantin Knizhnik
On 21.06.2018 20:08, Tom Lane wrote: Konstantin Knizhnik writes: The following very simple test reduce the problem with wrong cost estimation: create foreign table t1_fdw(x integer, y integer) server pg_fdw options (table_name 't1', use_remote_estimate 'false'); create

Wrong cost estimation for foreign tables join with use_remote_estimate disabled

2018-06-21 Thread Konstantin Knizhnik
nfo_o->rel_total_cost $25 = 21925 2817                run_cost += nrows * join_cost.per_tuple; (gdb) p run_cost $26 = 36350 I wonder if it is possible to make estimation of foreign join cost more precise. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: libpq compression

2018-06-21 Thread Konstantin Knizhnik
On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: My idea was the following: client want to use compression. But server may reject this attempt (for any reasons: it doesn't support it, h

Re: WAL prefetch

2018-06-21 Thread Konstantin Knizhnik
t. Any comments are welcome. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 1b000a2..9730b42 100644 --- a/src/backend/access/transam/xlogreader.

Re: libpq compression

2018-06-21 Thread Konstantin Knizhnik
On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: My idea was the following: client want to use compression. But server may reject this attempt (for any reasons: it doesn't support it, has no proper compression library, do not want to spend CPU for decompre

Re: libpq compression

2018-06-19 Thread Konstantin Knizhnik
On 20.06.2018 00:04, Robbie Harwood wrote: Konstantin Knizhnik writes: On 18.06.2018 23:34, Robbie Harwood wrote: I also don't like that you've injected into the *startup* path - before authentication takes place. Fundamentally, authentication (if it happens) consists of excha

Re: WAL prefetch

2018-06-19 Thread Konstantin Knizhnik
On 19.06.2018 18:50, Andres Freund wrote: On 2018-06-19 12:08:27 +0300, Konstantin Knizhnik wrote: I do not think that prefetching in shared buffers requires much more efforts and make patch more envasive... It even somehow simplify it, because there is no to maintain own cache of prefetched

Re: WAL prefetch

2018-06-19 Thread Konstantin Knizhnik
n-cpu-vroc-scalability/3 -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: WAL prefetch

2018-06-19 Thread Konstantin Knizhnik
On 19.06.2018 14:03, Tomas Vondra wrote: On 06/19/2018 11:08 AM, Konstantin Knizhnik wrote: On 18.06.2018 23:47, Andres Freund wrote: On 2018-06-18 16:44:09 -0400, Robert Haas wrote: On Sat, Jun 16, 2018 at 3:41 PM, Andres Freund wrote: The posix_fadvise approach is not perfect, no

Re: Invisible Indexes

2018-06-19 Thread Konstantin Knizhnik
etely done at extension level. At least definitely, altering grammar is not possible at extension level. But it can be handled using index parameters. Both invisible and hypothetical indexes seems to be really useful things: steps forward to "zero administration" database. My point i

Re: WAL prefetch

2018-06-19 Thread Konstantin Knizhnik
me now) forcing full page write when distance between page LSN and current WAL insertion point exceeds some threshold should eliminate random reads also in this case. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: libpq compression

2018-06-19 Thread Konstantin Knizhnik
On 18.06.2018 23:34, Robbie Harwood wrote: t Konstantin Knizhnik writes: On 06.06.2018 02:03, Thomas Munro wrote: On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik wrote: Thank you for review. Updated version of the patch fixing all reported problems is attached. Small problem on

Re: WAL prefetch

2018-06-17 Thread Konstantin Knizhnik
On 17.06.2018 03:00, Andres Freund wrote: On 2018-06-16 23:25:34 +0300, Konstantin Knizhnik wrote: On 16.06.2018 22:02, Andres Freund wrote: On 2018-06-16 11:38:59 +0200, Tomas Vondra wrote: On 06/15/2018 08:01 PM, Andres Freund wrote: On 2018-06-14 10:13:44 +0300, Konstantin Knizhnik

Re: WAL prefetch

2018-06-16 Thread Konstantin Knizhnik
On 16.06.2018 22:23, Andres Freund wrote: Hi, On 2018-06-13 16:09:45 +0300, Konstantin Knizhnik wrote: Usage: 1. At master: create extension wal_prefetch 2. At replica: Call pg_wal_prefetch() function: it will not return until you interrupt it. FWIW, I think the proper design would rather

Re: WAL prefetch

2018-06-16 Thread Konstantin Knizhnik
On 16.06.2018 22:02, Andres Freund wrote: On 2018-06-16 11:38:59 +0200, Tomas Vondra wrote: On 06/15/2018 08:01 PM, Andres Freund wrote: On 2018-06-14 10:13:44 +0300, Konstantin Knizhnik wrote: On 14.06.2018 09:52, Thomas Munro wrote: On Thu, Jun 14, 2018 at 1:09 AM, Konstantin Knizhnik

Re: WAL prefetch

2018-06-15 Thread Konstantin Knizhnik
On 16.06.2018 06:33, Amit Kapila wrote: On Fri, Jun 15, 2018 at 11:31 PM, Andres Freund wrote: On 2018-06-14 10:13:44 +0300, Konstantin Knizhnik wrote: On 14.06.2018 09:52, Thomas Munro wrote: On Thu, Jun 14, 2018 at 1:09 AM, Konstantin Knizhnik wrote: pg_wal_prefetch function will

Re: WAL prefetch

2018-06-15 Thread Konstantin Knizhnik
On 16.06.2018 06:30, Amit Kapila wrote: On Fri, Jun 15, 2018 at 8:45 PM, Konstantin Knizhnik wrote: On 15.06.2018 18:03, Amit Kapila wrote: wal_prefetch is prefetching blocks referenced by WAL records. But in case of "full page writes" such prefetch is not needed and even

Re: WAL prefetch

2018-06-15 Thread Konstantin Knizhnik
On 15.06.2018 18:03, Amit Kapila wrote: On Fri, Jun 15, 2018 at 1:08 PM, Konstantin Knizhnik wrote: On 15.06.2018 07:36, Amit Kapila wrote: On Fri, Jun 15, 2018 at 12:16 AM, Stephen Frost wrote: I have tested wal_prefetch at two powerful servers with 24 cores, 3Tb NVME RAID 10 storage

Re: WAL prefetch

2018-06-15 Thread Konstantin Knizhnik
ted code. But it is easier to experiment with extension rather than with patch to Postgres core. And I have published this extension to make it possible to perform experiments and check whether it is useful on real workloads. -- Konstantin Knizhnik Postgres Professional: http://www.postgre

Re: WAL prefetch

2018-06-14 Thread Konstantin Knizhnik
On 14.06.2018 16:25, Robert Haas wrote: On Thu, Jun 14, 2018 at 9:23 AM, Konstantin Knizhnik wrote: Speed of random HDD access is limited by speed of disk head movement. By running several IO requests in parallel we just increase probability of head movement, so actually parallel access to

Re: WAL prefetch

2018-06-14 Thread Konstantin Knizhnik
ation is very small. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: WAL prefetch

2018-06-14 Thread Konstantin Knizhnik
On 14.06.2018 09:52, Thomas Munro wrote: On Thu, Jun 14, 2018 at 1:09 AM, Konstantin Knizhnik wrote: pg_wal_prefetch function will infinitely traverse WAL and prefetch block references in WAL records using posix_fadvise(WILLNEED) system call. Hi Konstantin, Why stop at the page cache

WAL prefetch

2018-06-13 Thread Konstantin Knizhnik
. It is possible to explicitly specify start LSN for pg_wal_prefetch() function. Otherwise, WAL redo position will be used as start LSN. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company #include #include #include #include #include #in

Re: libpq compression

2018-06-06 Thread Konstantin Knizhnik
On 06.06.2018 19:33, Konstantin Knizhnik wrote: On 05.06.2018 20:06, Peter Eisentraut wrote: On 6/5/18 03:09, Michael Paquier wrote: I just had a quick look at this patch, lured by the smell of your latest messages...  And it seems to me that this patch needs a heavy amount of work as

Re: libpq compression

2018-06-06 Thread Konstantin Knizhnik
on. The only problem is with pg_dump which establish connection with server to fetch data from the database and is able to compress output data. So here we may need two options: compress input and compress output.  But I do not think that because of it -Z option should be removed from psql and

Re: libpq compression

2018-06-06 Thread Konstantin Knizhnik
On 06.06.2018 02:03, Thomas Munro wrote: On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik wrote: Thank you for review. Updated version of the patch fixing all reported problems is attached. Small problem on Windows[1]: C:\projects\postgresql\src\include\common/zpq_stream.h(17): error

Re: libpq compression

2018-06-06 Thread Konstantin Knizhnik
On 06.06.2018 10:53, Michael Paquier wrote: On Tue, Jun 05, 2018 at 06:58:42PM +0300, Konstantin Knizhnik wrote: I have considered this patch mostly as prototype to estimate efficiency of libpq protocol compression and compare it with SSL compression. So I agree with you that there are a lot

Re: I'd like to discuss scaleout at PGCon

2018-06-06 Thread Konstantin Knizhnik
Parquet file format which provides efficient extraction and processing of data. With abstract storage API Postgres is also given a chance to implement efficient storage for OLAP data processing. But huge amount of work has to be done here. -- Konstantin Knizhnik Postgres Professional: http://www.p

Re: libpq compression

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 10:09, Michael Paquier wrote: On Tue, Jun 05, 2018 at 06:04:21PM +1200, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik Speaking of configuration, are you planning to support multiple compression libraries at the same time? It looks like the current

Re: libpq compression

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 09:04, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik wrote: Concerning specification of compression level: I have made many experiments with different data sets and both zlib/zstd and in both cases using compression level higher than default doesn&#

Re: libpq compression

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 08:26, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik wrote: Thank you for this notice. Updated and rebased patch is attached. Hi Konstantin, Seems very useful. +1. + rc = inflate(&zs->rx, Z_SYNC_FLUSH); + if (rc != Z_OK) + { +

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 13:29, Masahiko Sawada wrote: On Tue, Jun 5, 2018 at 6:47 PM, Konstantin Knizhnik wrote: On 05.06.2018 07:22, Masahiko Sawada wrote: On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik wrote: On 26.04.2018 09:10, Masahiko Sawada wrote: On Thu, Apr 26, 2018 at 3:30 AM

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Konstantin Knizhnik
On 04.06.2018 21:42, Andres Freund wrote: Hi, On 2018-06-04 16:47:29 +0300, Konstantin Knizhnik wrote: We in PostgresProc were faced with lock extension contention problem at two more customers and tried to use this patch (v13) to address this issue. Unfortunately replacing heavy lock with

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 07:22, Masahiko Sawada wrote: On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik wrote: On 26.04.2018 09:10, Masahiko Sawada wrote: On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas wrote: On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada wrote: Never mind. There was a lot

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-04 Thread Konstantin Knizhnik
his lock, then abort of the current transaction doesn't release this lock (unlike heavy weight lock) and the relation is kept locked. So database is actually stalled and server has to be restarted. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Possible bug in logical replication.

2018-05-17 Thread Konstantin Knizhnik
On 17.05.2018 10:45, Konstantin Knizhnik wrote: We got the following assertion failure at our buildfarm of master branch of Postgres in contrib/test_decoding regression tests: 2018-05-07 19:50:07.241 MSK [5af083bf.54ae:49] DETAIL: Streaming transactions committing after 0/2A0, reading

Re: Removing unneeded self joins

2018-05-17 Thread Konstantin Knizhnik
to have no noticeable impact on performance and relation size can not dramatically changed during this time. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Possible bug in logical replication.

2018-05-17 Thread Konstantin Knizhnik
cified start point ends exactly at page boundary. The probability of it seems to be quite small... I am not so familiar with wal iteration code and logical decoding, so I will be pleased to receive confirmation orconfutation of my analysis of the problem. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: libpq compression

2018-05-16 Thread Konstantin Knizhnik
On 16.05.2018 18:09, Grigory Smolkin wrote: Hello! I have noticed that psql --help lack -Z|--compression option. Also it would be nice to have option like --compression-level in psql and pgbench. Thank you for this notice. Updated and rebased patch is attached. Concerning specification of

Re: libpq compression

2018-05-15 Thread Konstantin Knizhnik
On 15.05.2018 13:23, Dmitry Dolgov wrote: > On 30 March 2018 at 14:53, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: > Hi hackers, > One of our customers was managed to improve speed about 10 times by using SSL compression for the system where client and serve

Re: [HACKERS] Surjective functional indexes

2018-05-11 Thread Konstantin Knizhnik
On 11.05.2018 07:48, David G. Johnston wrote: On Thursday, February 1, 2018, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Old + New for check = 2 plus calculate again in index = 3 Yes, we have to calculate the value of index expression for or

Re: Built-in connection pooling

2018-05-08 Thread Konstantin Knizhnik
y locks,... with built-in connection pooler you can use sessions but with some restrictions (lack of advisory locks, for example). It is better than nothing, isn't it? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-05-07 Thread Konstantin Knizhnik
ssible to change lifetime of temporary tables from session to transaction... -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Having query cache in core

2018-05-07 Thread Konstantin Knizhnik
mentation will be similar with other shared caches implementation. If we have infrastructure for building efficient shared caches (lockless algorithms, smart invalidation, ...) then it will be not so difficult to implement result cache on top of it. -- Konstantin Knizhnik Postgres Prof

Re: Having query cache in core

2018-05-07 Thread Konstantin Knizhnik
On 07.05.2018 11:24, Tsunakawa, Takayuki wrote: From: Konstantin Knizhnik [mailto:k.knizh...@postgrespro.ru] But I think it is better to start first with 1. Global prepared statements cache 2. Global catalog cache 3. Global relation cache May I ask why prepared statements need to precede

Re: Having query cache in core

2018-05-07 Thread Konstantin Knizhnik
me we will have to implement them in any case (as it was done in most of other DBMSes). Concerning result cache, I think it will be better to ask opinion of mysql users: how useful it is. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-05-04 Thread Konstantin Knizhnik
ed by differences in pg_prepared_statements caused by "mangled" prepared names. Failures of isolation tests are caused by unsupported advisory locks. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-04-28 Thread Konstantin Knizhnik
On 27.04.2018 23:43, Merlin Moncure wrote: On Fri, Apr 27, 2018 at 11:44 AM, Konstantin Knizhnik wrote: On 27.04.2018 18:33, Merlin Moncure wrote: On Fri, Apr 27, 2018 at 10:05 AM, Konstantin Knizhnik wrote: On 27.04.2018 16:49, Merlin Moncure wrote: I'm confused here...cou

Re: Built-in connection pooling

2018-04-27 Thread Konstantin Knizhnik
On 27.04.2018 18:33, Merlin Moncure wrote: On Fri, Apr 27, 2018 at 10:05 AM, Konstantin Knizhnik wrote: On 27.04.2018 16:49, Merlin Moncure wrote: *) How are you pinning client connections to an application managed transaction? (IMNSHO, this feature is useless without being able to do that

Re: Built-in connection pooling

2018-04-27 Thread Konstantin Knizhnik
On 27.04.2018 16:49, Merlin Moncure wrote: On Thu, Apr 26, 2018 at 6:04 AM, Konstantin Knizhnik wrote: On 25.04.2018 20:02, Merlin Moncure wrote: Yep. The main workaround today is to disable them. Having said that, it's not that difficult to imagine hooking prepared statement creati

Re: Built-in connection pooling

2018-04-26 Thread Konstantin Knizhnik
are not related with GUCs. Most of them are really associated with backend, not with session. So them should not be handled by reschedule. But there may be some variables which are intended to be session specific. And locating this variables is really non trivial task. -- Konstantin Knizhnik P

Re: Built-in connection pooling

2018-04-26 Thread Konstantin Knizhnik
ing engine. I'm struggling as to why we would take another approach than 'don't use the non-xact variants of them in a pooling environment'. merlin -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-04-25 Thread Konstantin Knizhnik
rt them for pooled sessions. But I will think about it. But IMHO neither notifications, neither advisory locks are so widely used, comparing with temporary tables and prepared statements... -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-04-25 Thread Konstantin Knizhnik
ust do not know some variables. -- -- Christophe Pettus x...@thebuild.com -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-04-24 Thread Konstantin Knizhnik
th static variables unless them cross transaction boundary. But I do not think that there are many such variables. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-04-24 Thread Konstantin Knizhnik
the most flexible and scalable solution. I have not implemented this apporach due to the problems with forking new backend you have mentioned. But certainly it can be addressed. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Custom compression methods

2018-04-23 Thread Konstantin Knizhnik
om compression API usage. From my point of view the most useful will be integration with zstd. But if it is possible to find some example of data-specific compression algorithms which show better results than universal compression, it will be even more impressive. -- Konstantin Knizhnik Postgr

Re: [HACKERS] Custom compression methods

2018-04-23 Thread Konstantin Knizhnik
On 22.04.2018 16:21, Alexander Korotkov wrote: On Fri, Apr 20, 2018 at 7:45 PM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: On 30.03.2018 19:50, Ildus Kurbangaliev wrote: On Mon, 26 Mar 2018 20:38:25 +0300 Ildus Kurbangaliev mailto:i.kur

Re: [HACKERS] Custom compression methods

2018-04-20 Thread Konstantin Knizhnik
couldnt be tested.  I seems to be useful (and not so difficult) to use custom compression methods also for WAL compression: replace direct calls of pglz_compress in xloginsert.c -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Postgres stucks in deadlock detection

2018-04-20 Thread Konstantin Knizhnik
On 20.04.2018 18:36, Robert Haas wrote: On Wed, Apr 18, 2018 at 10:08 AM, Konstantin Knizhnik wrote: And it is very hard not to notice 17-times difference. Certainly it is true in the assumption that most deadlock timeout expiration are caused by high workload and contention, and not by real

Re: Built-in connection pooling

2018-04-20 Thread Konstantin Knizhnik
. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-04-20 Thread Konstantin Knizhnik
ostgres: if you want to preserve session semantic, then you need to spawn as much backends as sessions. And number of clients is limited by number of backends/sessions. The primary idea and main benefit of built-in connection pooler is to support session semantic with limited number of backends. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-04-20 Thread Konstantin Knizhnik
r do it in postmaster), which will process startup package and only after it schedule session to one of the workers. But such policy is much more difficult to implement and it is unclear how to map database/user pairs to worker backends. -- Konstantin Knizhnik Postgres Professi

Re: Built-in connection pooling

2018-04-20 Thread Konstantin Knizhnik
there is very large number of connected clients which rarely drop connection but only few of them are active at each moment of time. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-04-20 Thread Konstantin Knizhnik
On 19.04.2018 17:27, Dave Cramer wrote: On Thu, Apr 19, 2018, 9:24 AM Konstantin Knizhnik, mailto:k.knizh...@postgrespro.ru>> wrote: On 19.04.2018 07:46, Tsunakawa, Takayuki wrote: > From: Konstantin Knizhnik [mailto:k.knizh...@postgrespro.ru <ma

Re: Built-in connection pooling

2018-04-19 Thread Konstantin Knizhnik
On 19.04.2018 07:46, Tsunakawa, Takayuki wrote: From: Konstantin Knizhnik [mailto:k.knizh...@postgrespro.ru] Oracle, for example, you can create dedicated and non-dedicated backends. I wonder why we do not want to have something similar in Postgres. Yes, I want it, too. In addition to

Re: Postgres stucks in deadlock detection

2018-04-18 Thread Konstantin Knizhnik
On 16.04.2018 14:11, Konstantin Knizhnik wrote: On 14.04.2018 10:09, Юрий Соколов wrote: пт, 13 апр. 2018 г., 21:10 Andres Freund <mailto:and...@anarazel.de>>: Hi, On 2018-04-13 19:13:07 +0300, Konstantin Knizhnik wrote: > On 13.04.2018 18:41, Andres Freund wrote

Re: Built-in connection pooling

2018-04-18 Thread Konstantin Knizhnik
On 18.04.2018 16:41, Heikki Linnakangas wrote: On 18/04/18 07:52, Konstantin Knizhnik wrote: On 18.04.2018 13:36, Heikki Linnakangas wrote: On 18/04/18 06:10, Konstantin Knizhnik wrote: But there are still use cases which can not be covered y external connection pooler. Can you name

Re: Built-in connection pooling

2018-04-18 Thread Konstantin Knizhnik
On 18.04.2018 16:24, David Fetter wrote: On Wed, Apr 18, 2018 at 02:52:39PM +0300, Konstantin Knizhnik wrote: Yandex team is following this approach with theirOdysseus (multithreaded version of pgbouncer with many of pgbouncer issues fixed). Have they opened the source to Odysseus? If not

Re: Built-in connection pooling

2018-04-18 Thread Konstantin Knizhnik
On 18.04.2018 16:09, Craig Ringer wrote: On 18 April 2018 at 19:52, Konstantin Knizhnik wrote: As far as I know most of DBMSes have some kind of internal connection pooling. Oracle, for example, you can create dedicated and non-dedicated backends. I wonder why we do not want to have

Re: Built-in connection pooling

2018-04-18 Thread Konstantin Knizhnik
On 18.04.2018 13:36, Heikki Linnakangas wrote: On 18/04/18 06:10, Konstantin Knizhnik wrote: But there are still use cases which can not be covered y external connection pooler. Can you name some? I understand that the existing external connection poolers all have their limitations. But

Re: Built-in connection pooling

2018-04-18 Thread Konstantin Knizhnik
018 г. в 0:44, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>>: On 13.04.2018 19:07, Nikolay Samokhvalov wrote: On Fri, Apr 13, 2018 at 2:59 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Development in built-in connection pooling wil

Re: Built-in connection pooling

2018-04-17 Thread Konstantin Knizhnik
On 13.04.2018 19:07, Nikolay Samokhvalov wrote: On Fri, Apr 13, 2018 at 2:59 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Development in built-in connection pooling will be continued in https://github.com/postgrespro/postgresql.builtin_pool.git

Re: Postgres stucks in deadlock detection

2018-04-16 Thread Konstantin Knizhnik
On 14.04.2018 10:09, Юрий Соколов wrote: пт, 13 апр. 2018 г., 21:10 Andres Freund <mailto:and...@anarazel.de>>: Hi, On 2018-04-13 19:13:07 +0300, Konstantin Knizhnik wrote: > On 13.04.2018 18:41, Andres Freund wrote: > > On 2018-04-13 16:43:09 +0300, Konstan

Re: Interesting paper: Contention-Aware Lock Scheduling

2018-04-13 Thread Konstantin Knizhnik
d move all concurrency control to some special workers, with which executors are communicated using message-passing. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Postgres stucks in deadlock detection

2018-04-13 Thread Konstantin Knizhnik
On 13.04.2018 18:41, Andres Freund wrote: Hi, On 2018-04-13 16:43:09 +0300, Konstantin Knizhnik wrote: Updated patch is attached. + /* +* Ensure that only one backend is checking for deadlock. +* Otherwise under high load cascade of deadlock timeout expirations can

Re: Postgres stucks in deadlock detection

2018-04-13 Thread Konstantin Knizhnik
Updated patch is attached. On 05.04.2018 11:03, Konstantin Knizhnik wrote: Hi, Thank for your feedback. On 04.04.2018 21:15, Andres Freund wrote: Hi, On 2018-04-04 11:54:14 +0300, Konstantin Knizhnik wrote: Several times we and our customers are suffered from the problem that Postgres got

Re: Built-in connection pooling

2018-04-13 Thread Konstantin Knizhnik
On 06.04.2018 20:03, Konstantin Knizhnik wrote: On 06.04.2018 20:00, Konstantin Knizhnik wrote: Attached please find new version of the patch with  several bug fixes + support of more than one session pools associated with different ports. Now it is possible to make postmaster listen

Re: psql leaks memory on query cancellation

2018-04-12 Thread Konstantin Knizhnik
e in this area. The only solution is to use some memory contexts like in Postgres backends, but it requires complete rewriting of libpq. I am not sure that somebody will want to do it. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Optimization of range queries

2018-04-10 Thread Konstantin Knizhnik
 Index Scan using foo_pkey on foo  (cost=0.42..8.48 rows=2 width=8)    Index Cond: ((x <= 11) AND (x >= 9))    Filter: (y IS NOT NULL) (3 rows) Konstantin Knizhnik wrote: Hi hackers, Postgres optimizer is not able to build efficient executio

Optimization of range queries

2018-04-09 Thread Konstantin Knizhnik
PLAN  Index Scan using "People_pkey" on people_raw  (cost=0.58..196745.57 rows=586160 width=336)    Index Cond: (("ID" >= 2068113880) AND ("ID" <= 2068629726)) (

Re: PATCH: pgbench - option to build using ppoll() for larger connection counts

2018-04-06 Thread konstantin knizhnik
On Apr 7, 2018, at 12:49 AM, Tom Lane wrote: > Andres Freund writes: >> I'm still not particularly happy with this. > > I'm a bit confused as to what the point is. It seems unlikely that one > pgbench process can effectively drive enough backends for select's > limitations to really be an issu

Re: Built-in connection pooling

2018-04-06 Thread Konstantin Knizhnik
On 06.04.2018 20:00, Konstantin Knizhnik wrote: Attached please find new version of the patch with  several bug fixes + support of more than one session pools associated with different ports. Now it is possible to make postmaster listen several ports for accepting pooled connections, while

Re: Built-in connection pooling

2018-04-06 Thread Konstantin Knizhnik
. Each session pool is intended to be used for particular database/user combination. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c index 93c4bbf..dfc072c 100644

Re: Postgres stucks in deadlock detection

2018-04-05 Thread Konstantin Knizhnik
Hi, Thank for your feedback. On 04.04.2018 21:15, Andres Freund wrote: Hi, On 2018-04-04 11:54:14 +0300, Konstantin Knizhnik wrote: Several times we and our customers are suffered from the problem that Postgres got stuck in deadlock detection. One scenario is YCSB workload with Zipf&#

Postgres stucks in deadlock detection

2018-04-04 Thread Konstantin Knizhnik
ion is involved. I want to know opinion of community concerning this approaches (or may we there are some other solutions). Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/storage/lmgr/proc.c b

Diagonal storage model

2018-04-01 Thread Konstantin Knizhnik
at most of TPC-H queries. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diagonal.patch.gz Description: GNU Zip compressed data

Re: JIT compiling with LLVM v12.2

2018-03-30 Thread Konstantin Knizhnik
On 30.03.2018 18:54, Andres Freund wrote: On 2018-03-30 15:12:05 +0300, Konstantin Knizhnik wrote: I have repeated performance tests at my computer and find out some regression comparing with previous JIT version. Previously JIT provides about 2 times improvement at TPC-H Q1. Now the

libpq compression

2018-03-30 Thread Konstantin Knizhnik
pgbench -t 10 -S 4.482 4.926 4.877 -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/configure b/configure index 6f659a5..f2ef62d 100755 --- a/configure +++ b/configure @@ -698,6 +698,7 @@ ELF_SYS EGREP GREP

Re: JIT compiling with LLVM v12.2

2018-03-30 Thread Konstantin Knizhnik
Inlining: true    Inlining Time: 15.364 ms    Optimization: true    Optimization Time: 298.833 ms    Emission Time: 155.257 ms  Execution Time: 6807.751 ms (31 rows) Time: 6808.216 ms (00:06.808) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Feature Request - DDL deployment with logical replication

2018-03-30 Thread Konstantin Knizhnik
for Postgres-to-Postgres replication. So it will be very interesting for us to know roadmap for DDL support in logical replication. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Question about WalSndWriteData

2018-03-29 Thread Konstantin Knizhnik
On 21.03.2018 10:08, Konstantin Knizhnik wrote: On 21.03.2018 04:50, Peter Eisentraut wrote: On 3/16/18 12:08, Konstantin Knizhnik wrote: pq_putmessage_noblock copies data from ctx->out buffer to libpq buffers. After it we write timestamp to ctx->out buffer. And comments says t

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2018-03-29 Thread Konstantin Knizhnik
-s it should considered. In this case WAL sender can skip large portions of WAL without decoding. We also want to know opinion of 2ndQuandarnt about this idea. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: disable SSL compression?

2018-03-28 Thread Konstantin Knizhnik
On 28.03.2018 20:26, Konstantin Knizhnik wrote: On 17.03.2018 17:18, Peter Eisentraut wrote: On 3/11/18 13:28, Tom Lane wrote: My proposal is the attached patch that sets the default in libpq to off and adjusts the documentation a bit so it doesn't sound like we have missed the

<    1   2   3   4   5   6   7   >