Re: block-level incremental backup

2019-04-10 Thread Konstantin Knizhnik
pg_basebackup or pg_probackup). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c index 61a8f11..f4b8506 100644 --- a/src/backend/storage/smgr/md.c +++ b/src/backend/stor

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Konstantin Knizhnik
l Postgres table. It seems to be very strange. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
On 09.04.2019 19:19, Heikki Linnakangas wrote: On 09/04/2019 18:00, Konstantin Knizhnik wrote: Looks like the original problem was caused by internal postgres compressor: I have not configured Postgres to use lz4. When I configured Postgres --with-lz4, data was correctly inserted in zedstore

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
On 09.04.2019 18:51, Alvaro Herrera wrote: On 2019-Apr-09, Konstantin Knizhnik wrote: On 09.04.2019 3:27, Ashwin Agrawal wrote: Heikki and I have been hacking recently for few weeks to implement in-core columnar storage for PostgreSQL. Here's the design and initial implementation

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
On 09.04.2019 18:08, Heikki Linnakangas wrote: On 09/04/2019 18:00, Konstantin Knizhnik wrote: On 09.04.2019 17:09, Konstantin Knizhnik wrote: standard Postgres heap and my VOPS extension. As test data I used TPC-H benchmark (actually only one lineitem table generated with tpch-dbgen

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
On 09.04.2019 17:09, Konstantin Knizhnik wrote: Hi, On 09.04.2019 3:27, Ashwin Agrawal wrote: Heikki and I have been hacking recently for few weeks to implement in-core columnar storage for PostgreSQL. Here's the design and initial implementation of Zedstore, compressed in-core columnar

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
on is why this table is not empty if INSERT statement was failed? Please let me know if I can somehow help you to reproduce and investigate the problem. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company vstore_bench.sql Description: application/sql

Re: [HACKERS] Cached plans and statement generalization

2019-04-09 Thread Konstantin Knizhnik
New version of the patching disabling autoprepare for rules and handling planner error. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/doc/src/sgml/autoprepare.sgml b/doc/src/sgml/autoprepare.sgml new file mode 100644 index

Inheritance, invalidations and prepared statements.

2019-04-04 Thread Konstantin Knizhnik
tation: s1b s1delc1 s2prep s2sel s1c s2sel   step s1b: BEGIN;   step s1delc1: ALTER TABLE c1 NO INHERIT p; ! step s2prep: PREPARE summa as SELECT SUM(a) FROM p; ! step s2sel: EXECUTE summa;   step s1c: COMMIT;   step s2sel: <... completed>   sum   11 ! step s2sel: EXECUTE summa;  

Re: [HACKERS] Cached plans and statement generalization

2019-04-03 Thread Konstantin Knizhnik
New version of the patch with fixed error of autopreparing CTE queries. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/doc/src/sgml/autoprepare.sgml b/doc/src/sgml/autoprepare.sgml new file mode 100644 index 000..b3309bd

Re: [HACKERS][Proposal] LZ4 Compressed Storage Manager

2019-04-01 Thread Konstantin Knizhnik
actually perform IO". In this it will be easier to implement compressed storage manager. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Multitenancy optimization

2019-03-29 Thread Konstantin Knizhnik
On 29.03.2019 11:06, Hadi Moshayedi wrote: On Thu, Mar 28, 2019 at 5:40 AM Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Certainly it is possible to create multicolumn statistics to notify Postgres about columns correlation. But unfortunately it is no

Multitenancy optimization

2019-03-28 Thread Konstantin Knizhnik
foo where x=100 and y=100;   QUERY PLAN ---  Seq Scan on foo  (cost=0.00..1943.00 rows=10 width=8)    Filter: ((x = 100) AND (y = 100)) (2 rows) Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postg

Re: libpq compression

2019-03-26 Thread Konstantin Knizhnik
Version of the patch correctly working when no compression algorithm are avaiable. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/configure b/configure index 8068108..8ebd961 100755 --- a/configure +++ b/configure @@ -701,6

Re: libpq compression

2019-03-26 Thread Konstantin Knizhnik
On 25.03.2019 13:38, David Steele wrote: On 3/25/19 1:04 PM, Konstantin Knizhnik wrote: On 25.03.2019 11:06, David Steele wrote: Konstantin, This patch appears to be failing tests so I have marked it Waiting on Author. I have also removed the reviewer since no review had been done

Re: libpq compression

2019-03-25 Thread Konstantin Knizhnik
On 25.03.2019 13:48, Dmitry Dolgov wrote: On Mon, Mar 25, 2019 at 11:39 AM David Steele wrote: On 3/25/19 1:04 PM, Konstantin Knizhnik wrote: On 25.03.2019 11:06, David Steele wrote: Konstantin, This patch appears to be failing tests so I have marked it Waiting on Author. I have also

Re: libpq compression

2019-03-25 Thread Konstantin Knizhnik
hout using compression. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooler

2019-03-20 Thread Konstantin Knizhnik
New version of the patch (rebased + bug fixes) is attached to this mail. On 20.03.2019 18:32, Konstantin Knizhnik wrote: Attached please find results of benchmarking of different connection poolers. Hardware configuration:    Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz    24 cores (12

Re: [HACKERS] Cached plans and statement generalization

2019-03-19 Thread Konstantin Knizhnik
Thank you very much for the review! On 19.03.2019 5:56, Yamaji, Ryo wrote: On Tue, Jan 29, 2019 at 10:46 AM, Konstantin Knizhnik wrote: Rebased version of the patch is attached. I'm sorry for the late review. I confirmed behavior of autoprepare-12.patch. It is summarized below. ・parameter

Re: Drop type "smgr"?

2019-03-01 Thread Konstantin Knizhnik
useful to keep it in mind in discussions concerning "generic storage manager". -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: readdir is incorrectly implemented at Windows

2019-02-28 Thread Konstantin Knizhnik
in investigation of this problem. (the irony is that the problem detected by Yuri was caused by another bug in pg_probackup, but we thought that it was related with permissions and come to this issue). -- Michael -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian

readdir is incorrectly implemented at Windows

2019-02-25 Thread Konstantin Knizhnik
File(d->dirname, );         if (d->handle == INVALID_HANDLE_VALUE)         {             errno = ENOENT;             return NULL;         }     } Attached please find small patch fixing the problem. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres C

Re: Prepared transaction releasing locks before deregistering its GID

2019-02-19 Thread Konstantin Knizhnik
cks); + +   PredicateLockTwoPhaseFinish(xid, isCommit); + +   /* Count the prepared xact as committed or aborted */ +   AtEOXact_PgStat(isCommit); +     RESUME_INTERRUPTS();     pfree(buf); -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: libpq compression

2019-02-15 Thread Konstantin Knizhnik
On 15.02.2019 18:26, Tomas Vondra wrote: On 2/15/19 3:03 PM, Konstantin Knizhnik wrote: On 15.02.2019 15:42, Peter Eisentraut wrote: On 2018-06-19 09:54, Konstantin Knizhnik wrote: The main drawback of streaming compression is that you can not decompress some particular message without

Re: libpq compression

2019-02-15 Thread Konstantin Knizhnik
On 15.02.2019 15:42, Peter Eisentraut wrote: On 2018-06-19 09:54, Konstantin Knizhnik wrote: The main drawback of streaming compression is that you can not decompress some particular message without decompression of all previous messages. It seems this would have an adverse effect

Re: libpq compression

2019-02-14 Thread Konstantin Knizhnik
On 14.02.2019 19:45, Dmitry Dolgov wrote: For the records, I'm really afraid of interfering with the conversation at this point, but I believe it's necessary for the sake of a good feature :) On Wed, Feb 13, 2019 at 4:03 PM Konstantin Knizhnik wrote: 1. When decompressor has not enough

Re: libpq compression

2019-02-13 Thread Konstantin Knizhnik
On 13.02.2019 17:54, Dmitry Dolgov wrote: On Wed, Feb 13, 2019 at 3:52 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: On Wed, Feb 13, 2019 at 3:46 PM Konstantin Knizhnik wrote: Moreover, please notice that your implementation is still passing functions tx/rx functions to

Re: libpq compression

2019-02-13 Thread Konstantin Knizhnik
ad, doesn't matter how much code is copied. Frankly speaking I think that duplication of IO code between backend and frontend is one of the most awful parts of Postgres. It is always better to avoid duplication when it is possible. 3. Sorry, it was really a bug in 11 version of the patch, fixed in

Re: libpq compression

2019-02-13 Thread Konstantin Knizhnik
of what risks we open our end users to. Andreas Attached please find updated version of the patch with more comments  and warning about possible vulnerabilities of using compression in documentation. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres

Re: libpq compression

2019-02-11 Thread Konstantin Knizhnik
I attach new version of the patch which support choice between different compression algorithms. Right now only zstd and zlib are supported. If postgres is configured with both of them, then zstd will be used. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian

Re: libpq compression

2019-02-11 Thread Konstantin Knizhnik
L level compression. The goal was to support compression without using SSL. It seems to me that there are many cases when security is not requires, but reducing network traffic is desired. The best example is replication between node in local network. -- Konstantin Knizhnik Postgres

Re: libpq compression

2019-02-09 Thread Konstantin Knizhnik
On 10.02.2019 3:25, Tomas Vondra wrote: On 2/9/19 3:02 PM, Konstantin Knizhnik wrote: On 09.02.2019 1:38, Tomas Vondra wrote: On 2/8/19 11:10 PM, Konstantin Knizhnik wrote: On 08.02.2019 21:57, Andres Freund wrote: On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote: Frankly

Re: libpq compression

2019-02-09 Thread Konstantin Knizhnik
On 09.02.2019 1:38, Tomas Vondra wrote: On 2/8/19 11:10 PM, Konstantin Knizhnik wrote: On 08.02.2019 21:57, Andres Freund wrote: On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote: Frankly speaking, I do not think that such flexibility in choosing compression algorithms is really

Re: libpq compression

2019-02-08 Thread Konstantin Knizhnik
On 08.02.2019 21:57, Andres Freund wrote: On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote: Frankly speaking, I do not think that such flexibility in choosing compression algorithms is really needed. I do not expect that there will be many situations where old client has

Re: libpq compression

2019-02-08 Thread Konstantin Knizhnik
On 08.02.2019 19:26, Robbie Harwood wrote: Konstantin Knizhnik writes: On 08.02.2019 10:01, Iwata, Aya wrote: I fixed all issues you have reported except using list of supported compression algorithms. Sure. I confirmed that. It will require extra round of communication between client

Re: libpq compression

2019-02-08 Thread Konstantin Knizhnik
On 08.02.2019 12:33, Daniel Gustafsson wrote: On 8 Feb 2019, at 10:15, Konstantin Knizhnik wrote: Frankly speaking, I do not think that such flexibility in choosing compression algorithms is really needed. I do not expect that there will be many situations where old client has

Re: libpq compression

2019-02-08 Thread Konstantin Knizhnik
compress" >&5 +$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; } +if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then : +  cat >>confdefs.h <<_ACEOF +#define HAVE_LIBZSTD 1 +_ACEOF + +  LIBS="-lzstd $LIBS" + +else +  as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5 +fi + +fi Regards, Aya Iwata -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: libpq compression

2019-02-07 Thread Konstantin Knizhnik
On 08.02.2019 10:14, Andres Freund wrote: Hi, On 2018-03-30 15:53:39 +0300, Konstantin Knizhnik wrote: Taken in account that vulnerability was found in SSL compression and so SSLComppression is considered to be deprecated and insecure (http://www.postgresql-archive.org/disable-SSL

Replication & recovery_min_apply_delay

2019-01-30 Thread Konstantin Knizhnik
better solution except iterating WAL until I reach the last valid record. I attach small patch which implements this approach. I wonder if it can be considered as acceptable solution of the problem or there can be some better approach? -- Konstantin Knizhnik Postgres Professional:http

Re: [HACKERS] Cached plans and statement generalization

2019-01-29 Thread Konstantin Knizhnik
execution. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/doc/src/sgml/autoprepare.sgml b/doc/src/sgml/autoprepare.sgml new file mode 100644 index 000..b3309bd --- /dev/null +++ b/doc/src/sgml/autoprepare.sgml @@ -0,0 +1,62

Re: Built-in connection pooler

2019-01-29 Thread Konstantin Knizhnik
which should be rewritten. I will add more comments but I will be pleased if you point me which places are obscure now and require better explanation. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooler

2019-01-29 Thread Konstantin Knizhnik
On 29.01.2019 0:10, Bruce Momjian wrote: On Thu, Jan 24, 2019 at 08:14:41PM +0300, Konstantin Knizhnik wrote: The main differences with pgbouncer are that: 1. It is embedded and requires no extra steps for installation and configurations. 2. It is not single threaded (no bottleneck) 3

Built-in connection pooler

2019-01-24 Thread Konstantin Knizhnik
ction pool is in conn_proxy branch in the following GIT repository: https://github.com/postgrespro/postgresql.builtin_pool.git Also I attach patch to the master to this mail. Will be please to receive your comments. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russia

Re: libpq compression

2019-01-09 Thread Konstantin Knizhnik
mory control: there is a call of zpq_free(PqStream) in socket_close() function which should deallocate all memory used by compressor: void zpq_free(ZpqStream *zs) {     if (zs != NULL)     {         ZSTD_freeCStream(zs->tx_stream);         ZSTD_freeDStream(zs->rx_stream);         free(zs);    

Re: VOPS-2.0

2018-11-28 Thread Konstantin Knizhnik
On 28.11.2018 16:45, Alvaro Herrera wrote: On 2018-Nov-28, Bruce Momjian wrote: On Wed, Nov 28, 2018 at 01:01:03PM +0300, Konstantin Knizhnik wrote: Hi, I want to introduce new version of VOPS extension for Postgres (vectorized operations) providing new, more convenient way of usage: auto

Re: VOPS-2.0

2018-11-28 Thread Konstantin Knizhnik
On 28.11.2018 16:18, Bruce Momjian wrote: On Wed, Nov 28, 2018 at 01:01:03PM +0300, Konstantin Knizhnik wrote: Hi, I want to introduce new version of VOPS extension for Postgres (vectorized operations) providing new, more convenient way of usage: auto-substitution of projections. [Announce

Index-only scan is slower than Index scan.

2018-11-23 Thread Konstantin Knizhnik
me time as index scan: index scan 1.995 indexonly scan (original) 2.686 indexonly scan (patch) 2.005 -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/com

Re: [HACKERS] Surjective functional indexes

2018-11-09 Thread Konstantin Knizhnik
On 08.11.2018 22:05, Alvaro Herrera wrote: On 2018-Nov-08, Konstantin Knizhnik wrote: Before doing any other refactoring of projection indexes I want to attach small bug fix patch which fixes the original problem (SIGSEGV) and also disables recheck_on_update by default. As Laurenz has

Re: [HACKERS] Surjective functional indexes

2018-11-09 Thread Konstantin Knizhnik
); Document description may include many attributes which are updated quite frequently, like "comments", "keywords",... But "title" is rarely changed, so this optimization will be very useful for such index. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Surjective functional indexes

2018-11-08 Thread Konstantin Knizhnik
ant to attach small bug fix patch which fixes the original problem (SIGSEGV) and also disables recheck_on_update by default. As Laurenz has suggested, I replaced boolean recheck_on_update option with "on","auto,"off" (default). -- Konstantin Knizhnik Postgres Profession

Re: [HACKERS] Surjective functional indexes

2018-11-08 Thread Konstantin Knizhnik
particularly helpful option name, though we may be stuck with the latter at this point. As I am not native english speaking, I will agree with any proposed terminology. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Parallel threads in query

2018-11-01 Thread Konstantin Knizhnik
f video cards). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Estimating number of distinct values.

2018-10-24 Thread Konstantin Knizhnik
Hello hackers, I will be pleased if somebody (first of all Robert) can comment me "strange" results of distinct values estimation. There is the following code in analyze.c:        /*--              * Estimate the number of distinct values using the estimator              *

Re: [HACKERS] Secondary index access optimizations

2018-10-12 Thread Konstantin Knizhnik
On 08.10.2018 00:16, David Rowley wrote: On 5 October 2018 at 04:45, Konstantin Knizhnik wrote: Will the following test be enough: -- check that columns for parent table are correctly mapped to child partition of their order doesn't match create table paren (a int, b text) partition

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-11 Thread Konstantin Knizhnik
that GetRunningTransactionData may return duplicated XIDs because of the dummy 2PC entries which overlap with the active ones, and also add a proper comment in ProcArrayApplyRecoveryInfo(). Konstantin, do you want to give it a try with a patch? Or should I? -- Michael Proposed patch is attached. -- Konstantin

Few remarks on JIT , parallel query execution and columnar store...

2018-10-11 Thread Konstantin Knizhnik
has noticeable impact on total performance. In some other my prototype DBMS with vertical data representation and multhreaded execution time of execution of this query is 195 msec. So there is still scope for improvements:) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.co

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-09 Thread Konstantin Knizhnik
is seems to be the most efficient alternative). But I also do not fill that moving sort to GetRunningTransactionData and elimination of duplicates here (requires one more scan and copying of the whole array) is principally better... -- Konstantin Knizhnik Postgres Professional: http

Race condition in create table

2018-10-09 Thread Konstantin Knizhnik
t_1" already exists, skipping NOTICE:  relation "t_1" already exists, skipping NOTICE:  relation "t_1" already exists, skipping client 2 aborted in command 0 (SQL) of script 0; ERROR:  relation "t_1" already exists NOTICE:  relation "t_1" already exists, skipping NOTICE:  relation "t_1" already exists, skipping NOTICE:  relation "t_1" already exists, skipping client 5 aborted in command 0 (SQL) of script 0; ERROR:  relation "t_1" already exists client 4 aborted in command 0 (SQL) of script 0; ERROR:  relation "t_1" already exists -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company create_table.sql Description: application/sql create_part.sql Description: application/sql

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-08 Thread Konstantin Knizhnik
On 08.10.2018 18:24, Andres Freund wrote: On October 8, 2018 2:04:28 AM PDT, Konstantin Knizhnik wrote: On 05.10.2018 11:04, Michael Paquier wrote: On Fri, Oct 05, 2018 at 10:06:45AM +0300, Konstantin Knizhnik wrote: As you can notice, XID 2004495308 is encountered twice which cause

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-08 Thread Konstantin Knizhnik
On 08.10.2018 12:14, Michael Paquier wrote: On Mon, Oct 08, 2018 at 12:04:28PM +0300, Konstantin Knizhnik wrote: The simplest way to fix the problem is to ignore duplicates before adding them to KnownAssignedXids. We in any case perform sort i this place... I may of course be missing

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-08 Thread Konstantin Knizhnik
On 05.10.2018 11:04, Michael Paquier wrote: On Fri, Oct 05, 2018 at 10:06:45AM +0300, Konstantin Knizhnik wrote: As you can notice, XID 2004495308 is encountered twice which cause error in KnownAssignedXidsAdd:     if (head > tail &&         TransactionIdFollowsOrEquals(KnownA

Re: now() vs transaction_timestamp()

2018-10-06 Thread Konstantin Knizhnik
On 07.10.2018 07:58, Amit Kapila wrote: On Sat, Oct 6, 2018 at 9:40 PM Tom Lane wrote: Konstantin Knizhnik writes: On 06.10.2018 00:25, Tom Lane wrote: So maybe the right answer is to change the parallel mode infrastructure so it transmits xactStartTimestamp, making transaction_timestamp

Re: now() vs transaction_timestamp()

2018-10-06 Thread Konstantin Knizhnik
On 06.10.2018 00:25, Tom Lane wrote: I wrote: Konstantin Knizhnik writes: Postgres documentation says that |"now()| is a traditional PostgreSQL equivalent to |transaction_timestamp()|". Also both use the same implementation. Right. But them have different parallel safet

now() vs transaction_timestamp()

2018-10-05 Thread Konstantin Knizhnik
ime (1 row) As a result using now() in query disable parallel execution while transaction_timestamp allows it. Was it done intentionally or it is just a bug? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-05 Thread Konstantin Knizhnik
On 05.10.2018 11:04, Michael Paquier wrote: On Fri, Oct 05, 2018 at 10:06:45AM +0300, Konstantin Knizhnik wrote: As you can notice, XID 2004495308 is encountered twice which cause error in KnownAssignedXidsAdd:     if (head > tail &&         TransactionIdFollowsOrEquals(KnownA

out-of-order XID insertion in KnownAssignedXids

2018-10-05 Thread Konstantin Knizhnik
Hi hackers, Looks like there is a bug with logging running transactions XIDs and prepared transactions. One of our customers get error "FATAL: out-of-order XID insertion in KnownAssignedXids" trying to apply backup. WAL contains the following record: rmgr: Standby len (rec/tot): 98/  

Re: [HACKERS] Secondary index access optimizations

2018-10-04 Thread Konstantin Knizhnik
On 04.10.2018 12:19, David Rowley wrote: On 4 October 2018 at 22:11, Konstantin Knizhnik wrote: On 04.10.2018 06:19, David Rowley wrote: Please, can you also add a test which tests this code which has a partition with columns in a different order than it's parent. Having an INT and a TEXT

Re: libpq compression

2018-10-04 Thread Konstantin Knizhnik
On 01.10.2018 09:49, Michael Paquier wrote: On Mon, Aug 20, 2018 at 06:00:39PM +0300, Konstantin Knizhnik wrote: New version of the patch is attached: I removed -Z options form pgbench and psql and add checking that server and client are implementing the same compression algorithm. The patch

Re: [HACKERS] Secondary index access optimizations

2018-10-04 Thread Konstantin Knizhnik
On 04.10.2018 06:19, David Rowley wrote: On 12 September 2018 at 08:32, Konstantin Knizhnik wrote: Also the patch proposed by you is much simple and does mostly the same. Yes, it is not covering CHECK constraints, I started to look at this and found a problem in regards to varno during

Re: [HACKERS] Secondary index access optimizations

2018-09-12 Thread Konstantin Knizhnik
On 12.09.2018 08:14, David Rowley wrote: On 12 September 2018 at 08:32, Konstantin Knizhnik wrote: Also the patch proposed by you is much simple and does mostly the same. Yes, it is not covering CHECK constraints, but as far as partitioning becomes now standard in Postgres, I do not think

Re: [HACKERS] Secondary index access optimizations

2018-09-11 Thread Konstantin Knizhnik
Hi David, On 11.09.2018 06:49, David Rowley wrote: On 9 July 2018 at 13:26, David Rowley wrote: I started looking over this patch and have a few comments: Hi Konstantin, Wondering, are you going to be submitting an updated patch for this commitfest? If not then I think we can mark this as

Re: Startup cost of sequential scan

2018-08-30 Thread Konstantin Knizhnik
ll think there's a costing issue. regards, tom lane -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Bug in slot.c and are replication slots ever used at Window?

2018-08-30 Thread Konstantin Knizhnik
error in this case which is silently ignored by RestoreSlotFromDisk function. I think that bug fix is trivial: we just need to use fsync_parent_path instead of fsync_fname in RestoreSlotFromDisk. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooling

2018-08-27 Thread Konstantin Knizhnik
. Also I provided some general mechanism for moving static variables to session context. File include/storage/sessionvars.h contains list of such variables which are stored to session context on reschedule. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian

Re: [HACKERS] Cached plans and statement generalization

2018-08-24 Thread Konstantin Knizhnik
On 22.08.2018 07:54, Yamaji, Ryo wrote: On Tuesday, August 7, 2018 at 0:36 AM, Konstantin Knizhnik wrote: I have registered the patch for next commitfest. For some reasons it doesn't find the latest autoprepare-10.patch and still refer to autoprepare-6.patch as the latest attachement. I am

Re: libpq compression

2018-08-20 Thread Konstantin Knizhnik
he status to waiting on author. New version of the patch is attached: I removed -Z options form pgbench and psql and add checking that server and client are implementing the same compression algorithm. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres

Re: libpq compression

2018-08-14 Thread Konstantin Knizhnik
on option in connection string. There will be no problem if we use old client to access new server or use new client to access old server without switching on compression. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Cached plans and statement generalization

2018-08-07 Thread Konstantin Knizhnik
mory contexts themselves (even with small block size) somehow minimize difference in memory footprint of different queries, because of chunked allocation. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Cached plans and statement generalization

2018-08-02 Thread Konstantin Knizhnik
On 02.08.2018 08:25, Yamaji, Ryo wrote: -Original Message- From: Konstantin Knizhnik [mailto:k.knizh...@postgrespro.ru] Sent: Wednesday, August 1, 2018 4:53 PM To: Yamaji, Ryo/山地 亮 Cc: PostgreSQL mailing lists Subject: Re: [HACKERS] Cached plans and statement generalization I failed

Re: [HACKERS] Cached plans and statement generalization

2018-08-01 Thread Konstantin Knizhnik
On 31.07.2018 12:12, Yamaji, Ryo wrote: 3. I confirmed the transition of the amount of the memory when it tried to prepare query of the number that exceeded the value specified for autoprepare_limit. [autoprepare_limit=1 and execute 10 different queries] plan cache context: 1032

Re: [HACKERS] Cached plans and statement generalization

2018-08-01 Thread Konstantin Knizhnik
On 01.08.2018 00:30, Konstantin Knizhnik wrote: Hi Yamaji, On 31.07.2018 12:12, Yamaji, Ryo wrote: -Original Message- From: Konstantin Knizhnik [mailto:k.knizh...@postgrespro.ru] Sent: Friday, January 12, 2018 9:53 PM To: Thomas Munro ; Stephen Frost Cc: Michael Paquier

Re: [HACKERS] Cached plans and statement generalization

2018-07-31 Thread Konstantin Knizhnik
Hi Yamaji, On 31.07.2018 12:12, Yamaji, Ryo wrote: -Original Message- From: Konstantin Knizhnik [mailto:k.knizh...@postgrespro.ru] Sent: Friday, January 12, 2018 9:53 PM To: Thomas Munro ; Stephen Frost Cc: Michael Paquier ; PostgreSQL mailing lists ; Tsunakawa, Takayuki/綱川 貴之

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-18 Thread Konstantin Knizhnik
On 18.07.2018 02:58, Tomas Vondra wrote: On 07/18/2018 12:41 AM, Konstantin Knizhnik wrote: ... Teodor Sigaev has proposed an alternative approach for calculating selectivity of multicolumn join or compound index search. Usually DBA creates compound indexes which can be used  by optimizer

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-17 Thread Konstantin Knizhnik
On 16.07.2018 23:55, Tomas Vondra wrote: On 07/16/2018 02:54 PM, Dean Rasheed wrote: On 16 July 2018 at 13:23, Tomas Vondra wrote: The top-level clauses allow us to make such deductions, with deeper clauses it's much more difficult (perhaps impossible). Because for example with (a=1 AND

Re: WAL prefetch

2018-07-09 Thread Konstantin Knizhnik
On 09.07.2018 21:28, Andres Freund wrote: Hi, On 2018-07-09 11:59:06 +0200, Tomas Vondra wrote: * During the design phase, I looked into using bgworkers but given the number of in-flight pread(2) calls required to fully utilize the IO subsystem, I opted for something threaded (I was

Re: WAL prefetch

2018-07-09 Thread Konstantin Knizhnik
On 08.07.2018 00:47, Tomas Vondra wrote: Hi, I've done a bit of testing on the current patch, mostly to see how much the prefetching can help (if at all). While the patch is still in early WIP stages (at least that's my assessment, YMMV), the improvement are already quite significant. I've

Re: Global shared meta cache

2018-07-05 Thread Konstantin Knizhnik
On 05.07.2018 17:00, Robert Haas wrote: On Mon, Jul 2, 2018 at 5:59 AM, Konstantin Knizhnik wrote: But I am not sure that just using RW lock will be enough replace local cache with global. I'm pretty sure it won't. In fact, no matter what kind of locking you use, it's bound to cost

Unusable index

2018-07-03 Thread Konstantin Knizhnik
so dramatically increases query execution time in another database also seems to be very disappointing. Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Global shared meta cache

2018-07-02 Thread Konstantin Knizhnik
surprises here: negative effect of shared cache is the largest for the case of non-prepared selects (because selects themselves are much faster than updates and during compilation we have to access relations multiple times). -- Konstantin Knizhnik Postgres Professi

Re: Monitoring time of fsyncing WALs

2018-06-29 Thread Konstantin Knizhnik
On 29.06.2018 15:48, Robert Haas wrote: On Wed, Jun 27, 2018 at 10:27 PM, Michael Paquier wrote: On Wed, Jun 27, 2018 at 07:32:18PM +0300, Konstantin Knizhnik wrote: I wonder why we are monitoring time of writing to WAL, but not time of fsyncing WAL segments? Is there are principle reason

Re: libpq compression

2018-06-25 Thread Konstantin Knizhnik
to set up compressed vs. uncompressed connections - similarly to how we've documentation on setting up TLS connection (though presumably compressed connection documentation will be shorter). Document protocol changes needed for libpq compression. -- Konstantin Knizhnik Postgres Professional: http

Re: libpq compression

2018-06-23 Thread Konstantin Knizhnik
On 22.06.2018 20:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 22.06.2018 18:59, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes

Re: libpq compression

2018-06-22 Thread Konstantin Knizhnik
On 22.06.2018 19:05, Nico Williams wrote: On Fri, Jun 22, 2018 at 10:18:12AM +0300, Konstantin Knizhnik wrote: On 22.06.2018 00:34, Nico Williams wrote: So I think you just have to have lengths. Now, this being about compression, I understand that you might now want to have 4-byte lengths

Re: libpq compression

2018-06-22 Thread Konstantin Knizhnik
On 22.06.2018 18:59, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes

Re: Wrong cost estimation for foreign tables join with use_remote_estimate disabled

2018-06-22 Thread Konstantin Knizhnik
On 22.06.2018 13:30, Ashutosh Bapat wrote: On Fri, Jun 22, 2018 at 11:56 AM, Konstantin Knizhnik wrote: On 21.06.2018 20:08, Tom Lane wrote: Konstantin Knizhnik writes: The following very simple test reduce the problem with wrong cost estimation: create foreign table t1_fdw(x integer, y

Re: WAL prefetch

2018-06-22 Thread Konstantin Knizhnik
On 21.06.2018 19:57, Tomas Vondra wrote: On 06/21/2018 04:01 PM, Konstantin Knizhnik wrote: I continue my experiments with WAL prefetch. I have embedded prefetch in Postgres: now walprefetcher is started together with startup process and is able to help it to speedup recovery. The patch

Re: libpq compression

2018-06-22 Thread Konstantin Knizhnik
On 22.06.2018 00:34, Nico Williams wrote: On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: Well, that's a design decision you've made. You could put lengths on chunks that are sent out - then you'd

Re: libpq compression

2018-06-22 Thread Konstantin Knizhnik
On 21.06.2018 20:14, Robbie Harwood wrote: Konstantin Knizhnik writes: On 21.06.2018 17:56, Robbie Harwood wrote: Konstantin Knizhnik writes: On 20.06.2018 23:34, Robbie Harwood wrote: Konstantin Knizhnik writes: Well, that's a design decision you've made. You could put lengths

Re: Wrong cost estimation for foreign tables join with use_remote_estimate disabled

2018-06-22 Thread Konstantin Knizhnik
On 21.06.2018 20:08, Tom Lane wrote: Konstantin Knizhnik writes: The following very simple test reduce the problem with wrong cost estimation: create foreign table t1_fdw(x integer, y integer) server pg_fdw options (table_name 't1', use_remote_estimate 'false'); create foreign table t2_fdw

Wrong cost estimation for foreign tables join with use_remote_estimate disabled

2018-06-21 Thread Konstantin Knizhnik
_cost += nrows * join_cost.per_tuple; (gdb) p run_cost $26 = 36350 I wonder if it is possible to make estimation of foreign join cost more precise. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

<    1   2   3   4   5   6   7   >