Re: [HACKERS] Partition-wise aggregation/grouping

2017-11-13 Thread Konstantin Knizhnik
On 11.11.2017 23:29, Konstantin Knizhnik wrote: On 10/27/2017 02:01 PM, Jeevan Chalke wrote: Hi, Attached new patch-set here. Changes include: 1. Added separate patch for costing Append node as discussed up-front in the patch-set. 2. Since we now cost Append node, we don't

Re: [HACKERS] Partition-wise aggregation/grouping

2017-11-11 Thread Konstantin Knizhnik
d approach is easier for implementation. But in case of sharded table, distributed query may need to traverse both remote and local shards and this approach doesn't allow to processed several local shards in parallel. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The R

Re: [HACKERS] Aggregates push-down to partitions

2017-11-10 Thread Konstantin Knizhnik
->  Seq Scan on derived2  (cost=0.00..14425.00 rows=100 width=0) ->  Foreign Scan on derived_fdw  (cost=100.00..212.39 rows=3413 width=0) (6 rows) Are there some principle problems? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgre

[HACKERS] Aggregates push-down to partitions

2017-11-09 Thread Konstantin Knizhnik
estions about the best approach to implement this feature? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SQL procedures

2017-11-08 Thread Konstantin Knizhnik
have not done it yet, because there seems to be no chances to push this patch to community. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Secondary index access optimizations

2017-11-06 Thread Konstantin Knizhnik
On 11/06/2017 04:27 AM, Thomas Munro wrote: On Fri, Sep 8, 2017 at 3:58 AM, Konstantin Knizhnik wrote: Updated version of the patch is attached to this mail. Also I added support of date type to operator_predicate_proof to be able to imply (logdate <= '2017-03-31') from (logdate

[HACKERS] Deadlock in ALTER SUBSCRIPTION REFRESH PUBLICATION

2017-10-24 Thread Konstantin Knizhnik
244d6e0) at main.c:228 The reason of this deadlock seems to be clear: ALTER SUBSCRIPTION starts transaction at one node and tries to create slot at other node, which waiting for completion of all active transaction while building scnapshpot. Is there any way to avoid this deadlock? The same de

Re: [HACKERS] Slow synchronous logical replication

2017-10-12 Thread Konstantin Knizhnik
On 12.10.2017 04:23, Craig Ringer wrote: On 12 October 2017 at 00:57, Konstantin Knizhnik wrote: The reason of such behavior is obvious: wal sender has to decode huge transaction generate by insert although it has no relation to this publication. It does. Though I wouldn't expect any

Re: [HACKERS] Slow synchronous logical replication

2017-10-11 Thread Konstantin Knizhnik
On 11.10.2017 10:07, Craig Ringer wrote: On 9 October 2017 at 15:37, Konstantin Knizhnik wrote: Thank you for explanations. On 08.10.2017 16:00, Craig Ringer wrote: I think it'd be helpful if you provided reproduction instructions, test programs, etc, making it very clear when thing

Re: [HACKERS] Columnar storage support

2017-10-10 Thread Konstantin Knizhnik
us is probably what you are looking for. jD -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] Slow synchronous logical replication

2017-10-09 Thread Konstantin Knizhnik
rarely than other or is not updated at all (for example because communication channels between this node is broken), then all backens will stuck. Also all backends are competing for the single SyncRepLock, which also can be a contention point. -- Konstantin Knizhnik Postgres Professional: http:/

Re: [HACKERS] Slow synchronous logical replication

2017-10-07 Thread Konstantin Knizhnik
On 10/07/2017 10:42 PM, Andres Freund wrote: Hi, On 2017-10-07 22:39:09 +0300, konstantin knizhnik wrote: In our sharded cluster project we are trying to use logical relication for providing HA (maintaining redundant shard copies). Using asynchronous logical replication has not so much sense

[HACKERS] Slow synchronous logical replication

2017-10-07 Thread konstantin knizhnik
In our sharded cluster project we are trying to use logical relication for providing HA (maintaining redundant shard copies). Using asynchronous logical replication has not so much sense in context of HA. This is why we try to use synchronous logical replication. Unfortunately it shows very bad p

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-07 Thread Konstantin Knizhnik
at logical decoding snapshot is being built and using that to skip catalog access which does not seem very pretty. It is not quite clear from the comment why it is not possible to overwrite MyPgXact->xmin or just use existed value. -- Konstantin Knizhnik Postgres Professional: http://www.postgr

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-06 Thread Konstantin Knizhnik
On 06.10.2017 15:29, Petr Jelinek wrote: On 06/10/17 12:16, Konstantin Knizhnik wrote: When creating logical replication slots we quite often get the following error: ERROR: cannot build an initial slot snapshot when MyPgXact->xmin already is valid which cause restart of WAL sender.

[HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-06 Thread Konstantin Knizhnik
7ef9e2 in BackendRun (port=0x24dee00) at postmaster.c:4357 #28 0x007ef10c in BackendStartup (port=0x24dee00) at postmaster.c:4029 #29 0x007eb6cc in ServerLoop () at postmaster.c:1753 #30 0x007eacb8 in PostmasterMain (argc=3, argv=0x24bd660) at postmaster.c:1361 #31 0x000

[HACKERS] Issues with logical replication

2017-10-02 Thread Konstantin Knizhnik
I have faced two issues with logical replication. Repro scenario: 1. start two Postgres instances (I start both at local machine). 2. Initialize pgbench tables at both instances: pgbench -i -s 10 postgres 3. Create publication of pgbench_accounts table at one node: create publication pub

[HACKERS] Issues with logical replication

2017-10-02 Thread Konstantin Knizhnik
sigusr1_handler (postgres_signal_arg=10) at postmaster.c:5073 #14 #15 0x7fbe83a054a3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:81 #16 0x007eb517 in ServerLoop () at postmaster.c:1717 #17 0x007eac48 in PostmasterMain (argc=3, argv=0x2d4e660) at postmaster

Re: [HACKERS] alter server for foreign table

2017-09-30 Thread konstantin knizhnik
On Sep 30, 2017, at 10:58 PM, Andrew Dunstan wrote: > > > On 09/30/2017 05:14 AM, Derry Hamilton wrote: >> Just to say, yes, this would be handy. I've been using a variant of >> that hack on reporting servers, while migrating systems from >> proprietary databases. It behaves quite gracefully w

Re: [HACKERS] pg_prepared_xact_status

2017-09-29 Thread konstantin knizhnik
On Sep 29, 2017, at 11:33 PM, Robert Haas wrote: > On Fri, Sep 29, 2017 at 4:22 AM, Craig Ringer wrote: >> This sounds kind-of like 1/4 of a distributed transaction resolver, without >> a way to make it reliable enough to build the other 3/4. >> >> To make this practical I think you'd need a wa

[HACKERS] alter server for foreign table

2017-09-29 Thread Konstantin Knizhnik
/may-i-alter-server-for-foreign-table I wonder how safe it is and if it is so simple, why it is not support in ALTER FOREIGN TABLE statement? Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] pg_prepared_xact_status

2017-09-29 Thread Konstantin Knizhnik
On 29.09.2017 11:27, Craig Ringer wrote: On 29 September 2017 at 15:57, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: So you are saying that Postgresql 2PC mechanism is not complete and user needs to maintain some extra information to make it work? No, it pr

Re: [HACKERS] Index expression syntax

2017-09-29 Thread Konstantin Knizhnik
On 29.09.2017 11:03, Marko Tiikkaja wrote: On Fri, Sep 29, 2017 at 9:31 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: I wonder why syntax error is produced in this case: postgres=# create index metaindex on foo using gin(to_tsvector('english',

Re: [HACKERS] pg_prepared_xact_status

2017-09-29 Thread Konstantin Knizhnik
On 29.09.2017 06:02, Michael Paquier wrote: On Fri, Sep 29, 2017 at 1:53 AM, Konstantin Knizhnik wrote: In Postgres 10 we have txid_status function which returns status of transaction by XID. I wonder if it will be also useful to have similar function for 2PC transactions which can operate

[HACKERS] Index expression syntax

2017-09-29 Thread Konstantin Knizhnik
entheses can be omitted if the expression has the form of a function call. -- So documentations states that sometimes it is possible to avoid parentheses, but it is unclear why I have to use double parentheses... I think that either grammar should be fixed, either documentatio

Re: [HACKERS] Surjective functional indexes

2017-09-28 Thread Konstantin Knizhnik
hema design it is possible to expect that most of updates are hot updates: do not actually affect any index. But certainly different attributes may have different probability of been updated. Unfortunately we do not know before check which attribute of JSON field (or any other fields used in indexed

[HACKERS] pg_prepared_xact_status

2017-09-28 Thread Konstantin Knizhnik
GetSystemIdentifier() to identify coordinator's node) and XID of coordinator's transaction. In this case we can use txid_status() to check status of transaction at coordinator. It eliminates need to scan WAL to determine status of prepared transaction. -- Konstantin Knizhnik Postgres Professi

Re: [HACKERS] JIT compiling expressions/deform + inlining prototype v2.0

2017-09-19 Thread Konstantin Knizhnik
tinue review of this code? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Surjective functional indexes

2017-09-15 Thread Konstantin Knizhnik
ction" index option explicitly set by user. This setting overrides 1) and 2) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/src/sgml/ref/create_index.sgml index 83ee7d3..52

Re: [HACKERS] Surjective functional indexes

2017-09-15 Thread Konstantin Knizhnik
r it will perform projection or not and whether comparing old/new expression value makes sense or is just waste of time. We can guess it from autotune, but such decision may be wrong (just because of application business logic). Postgres indexes already have a lot of options. And I think that "

Re: [HACKERS] Surjective functional indexes

2017-09-14 Thread Konstantin Knizhnik
On 14.09.2017 13:19, Simon Riggs wrote: On 14 September 2017 at 10:42, Konstantin Knizhnik wrote: On 13.09.2017 14:00, Simon Riggs wrote: On 13 September 2017 at 11:30, Konstantin Knizhnik wrote: The only reason of all this discussion about terms is that I need to choose name for

Re: [HACKERS] Surjective functional indexes

2017-09-14 Thread Konstantin Knizhnik
On 13.09.2017 14:00, Simon Riggs wrote: On 13 September 2017 at 11:30, Konstantin Knizhnik wrote: The only reason of all this discussion about terms is that I need to choose name for correspondent index option. Simon think that we do not need this option at all. In this case we should not

Re: [HACKERS] Surjective functional indexes

2017-09-13 Thread Konstantin Knizhnik
On 13.09.2017 14:00, Simon Riggs wrote: On 13 September 2017 at 11:30, Konstantin Knizhnik wrote: The only reason of all this discussion about terms is that I need to choose name for correspondent index option. Simon think that we do not need this option at all. In this case we should not

Re: [HACKERS] Surjective functional indexes

2017-09-13 Thread Konstantin Knizhnik
On 13.09.2017 13:14, Christoph Berg wrote: Re: Konstantin Knizhnik 2017-09-13 <2393c4b3-2ec4-dc68-4ea9-670597b56...@postgrespro.ru> On 13.09.2017 10:51, Christoph Berg wrote: Re: Konstantin Knizhnik 2017-09-01 + Functional index is based on on projection function: function

Re: [HACKERS] Surjective functional indexes

2017-09-13 Thread Konstantin Knizhnik
On 13.09.2017 10:51, Christoph Berg wrote: Re: Konstantin Knizhnik 2017-09-01 + Functional index is based on on projection function: function which extract subset of its argument. + In mathematic such functions are called non-injective. For injective function if any attribute

Re: [HACKERS] Surjective functional indexes

2017-09-12 Thread Konstantin Knizhnik
On 12.09.2017 19:28, Simon Riggs wrote: On 1 September 2017 at 09:47, Konstantin Knizhnik wrote: On 01.09.2017 09:25, Simon Riggs wrote: On 1 September 2017 at 05:40, Thomas Munro wrote: On Fri, Jun 9, 2017 at 8:08 PM, Konstantin Knizhnik wrote: Attached please find rebased version of

Re: [HACKERS] Cached plans and statement generalization

2017-09-12 Thread Konstantin Knizhnik
On 11.09.2017 12:24, Konstantin Knizhnik wrote: Attached please find rebased version of the patch. There are the updated performance results (pgbench -s 100 -c 1): protocol (-M) read-write read-only (-S) simple 3327 19325 extended 2256 16908

Re: [HACKERS] Cached plans and statement generalization

2017-09-11 Thread Konstantin Knizhnik
On 09.09.2017 06:35, Thomas Munro wrote: On Fri, May 26, 2017 at 3:54 AM, Konstantin Knizhnik wrote: Attached please find rebased version of the autoprepare patch based on Tom's proposal (perform analyze for tree with constant literals and then replace them with parameters). Also I subm

Re: [HACKERS] Secondary index access optimizations

2017-09-07 Thread Konstantin Knizhnik
On 07.09.2017 13:00, Thomas Munro wrote: On Sun, Sep 3, 2017 at 4:34 AM, Konstantin Knizhnik wrote: Thank you for review. I attached new version of the patch with remove_restrictions_implied_by_constraints() function. Concerning failed tests - this is actually result of this optimization

Re: [HACKERS] JIT compiling expressions/deform + inlining prototype v2.0

2017-09-05 Thread Konstantin Knizhnik
values, either mostly null values). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Secondary index access optimizations

2017-09-05 Thread Konstantin Knizhnik
easurement where extract(month from logdate) = 3; select * from measurement where logdate between '2006-03-01' AND '2006-03-31'; select * from measurement where logdate >= '2006-03-01' AND logdate < '2006-04-01'; Right now only for the last query opt

Re: [HACKERS] JIT compiling expressions/deform + inlining prototype v2.0

2017-09-04 Thread Konstantin Knizhnik
uot;alignto" and comment in the following branch else if (att->attnotnull) seems to be not related to this branch, because in this case attcuralign is expected to be less then zero wjhich means that previous attribute is varlen field. -- Konstantin Knizhnik Postgres Professional: ht

Re: [HACKERS] Secondary index access optimizations

2017-09-04 Thread Konstantin Knizhnik
On 04.09.2017 12:59, Amit Langote wrote: Hi Konstantin, On 2017/09/04 18:19, Konstantin Knizhnik wrote: On 04.09.2017 05:38, Amit Langote wrote: On 2017/09/02 12:44, Thomas Munro wrote: On Wed, Aug 16, 2017 at 9:23 PM, Konstantin Knizhnik wrote: postgres=# explain select * from bt where

Re: [HACKERS] Secondary index access optimizations

2017-09-04 Thread Konstantin Knizhnik
On 04.09.2017 05:38, Amit Langote wrote: On 2017/09/02 12:44, Thomas Munro wrote: On Wed, Aug 16, 2017 at 9:23 PM, Konstantin Knizhnik wrote: postgres=# explain select * from bt where k between 1 and 2 and v = 100; QUERY PLAN

Re: [HACKERS] JIT & function naming

2017-09-03 Thread Konstantin Knizhnik
text (es_sourceText from EText), replace all non-alphanumeric characters spaces with '_' and take first N (16?) characters of the result? It seems to me that in most cases it will help to identify the query... -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The

Re: [HACKERS] Secondary index access optimizations

2017-09-02 Thread Konstantin Knizhnik
On 09/02/2017 06:44 AM, Thomas Munro wrote: On Wed, Aug 16, 2017 at 9:23 PM, Konstantin Knizhnik wrote: postgres=# explain select * from bt where k between 1 and 2 and v = 100; QUERY PLAN

Re: [HACKERS] Surjective functional indexes

2017-09-01 Thread Konstantin Knizhnik
On 01.09.2017 09:25, Simon Riggs wrote: On 1 September 2017 at 05:40, Thomas Munro wrote: On Fri, Jun 9, 2017 at 8:08 PM, Konstantin Knizhnik wrote: Attached please find rebased version of the patch. Now "projection" attribute is used instead of surjective/injective. Hi Konstan

Re: [HACKERS] [PATCH] Push limit to sort through a subquery

2017-08-23 Thread Konstantin Knizhnik
On 22.08.2017 17:27, Konstantin Knizhnik wrote: On 18.08.2017 04:33, Robert Haas wrote: It seems like a somewhat ad-hoc approach; it supposes that we can take any query produced by deparseSelectStmtForRel() and stick a LIMIT clause onto the very end and all will be well. Maybe that&#

Re: [HACKERS] [PATCH] Push limit to sort through a subquery

2017-08-22 Thread Konstantin Knizhnik
all and non-invasive. And in principle, it can be used not only postgres_fdw, but also in other FDW implementations to push down information about LIMIT. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] [PATCH] Push limit to sort through a subquery

2017-08-17 Thread Konstantin Knizhnik
er by v limit 1 100 42 There is index for "u", so fetching records with smallest "u" values can be done without sorting, so times are similar. But in case of sorting by "v", pushing down limit allows to use TOP-1 instead of global sort and it reduces

Re: [HACKERS] Secondary index access optimizations

2017-08-16 Thread Konstantin Knizhnik
On 14.08.2017 19:33, Konstantin Knizhnik wrote: On 14.08.2017 12:37, Konstantin Knizhnik wrote: Hi hackers, I am trying to compare different ways of optimizing work with huge append-only tables in PostgreSQL where primary key is something like timestamp and queries are usually accessing

Re: [HACKERS] Secondary index access optimizations

2017-08-14 Thread Konstantin Knizhnik
On 14.08.2017 12:37, Konstantin Knizhnik wrote: Hi hackers, I am trying to compare different ways of optimizing work with huge append-only tables in PostgreSQL where primary key is something like timestamp and queries are usually accessing most recent data using some secondary keys. Size

[HACKERS] Secondary index access optimizations

2017-08-14 Thread Konstantin Knizhnik
s if specified interval is covered by their conditions. I wonder if someone is familiar with this part of optimizer ad can easily fix it. Otherwise I am going to spend some time on solving this problems (if community think that such optimizations will be useful). -- Konstantin Knizhnik Postgres

Re: [HACKERS] Error : undefined symbol : LWLockAssign in 9.6.3

2017-08-08 Thread Konstantin Knizhnik
o use IMCS with 9.6.3 and later Postgres versions. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] ASOF join

2017-06-21 Thread Konstantin Knizhnik
e join instead of nested loop) but also simplifying writing of such queries. Or do you think that nobody will be interested in non-standard SQL extensions? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ASOF join

2017-06-19 Thread Konstantin Knizhnik
On 16.06.2017 19:07, David Fetter wrote: On Fri, Jun 16, 2017 at 11:51:34AM +1200, Thomas Munro wrote: On Fri, Jun 16, 2017 at 4:20 AM, Konstantin Knizhnik wrote: I wonder if there were some discussion/attempts to add ASOF join to Postgres (sorry, may be there is better term for it, I am

Re: [HACKERS] WIP: Data at rest encryption

2017-06-16 Thread Konstantin Knizhnik
is better to allow user to make choice whether to encrypt temporary data or not. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/storage/file/Makefile b/src/backend/storage/file/Makefile index d2198f2..9492662 100644

[HACKERS] ASOF join

2017-06-15 Thread Konstantin Knizhnik
Trades ASOF JOIN EqTrades USING (underlyingSym,time); It seems to me that adding ASOF joins should not require huge amount of work and can be done with minimal number of changes in executor and optimizer. But may be there are some problems/challenges which I do not realize now? -- Konstantin Kniz

Re: [HACKERS] Surjective functional indexes

2017-06-09 Thread Konstantin Knizhnik
Attached please find rebased version of the patch. Now "projection" attribute is used instead of surjective/injective. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/sr

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-30 Thread Konstantin Knizhnik
astructure to do parallel load. We just need to spawn more than one background worker and specify separate block range for each worker. Do you think that such functionality (parallel autoprewarm) can be useful and be easily added? -- Konstantin Knizhnik Postgres Professional: http://www.po

Re: [HACKERS] Surjective functional indexes

2017-05-29 Thread Konstantin Knizhnik
to "projection" if it will be considered as better alternative. From my point of view, "projection" seems to be clearer for people without mathematical background, but IMHO this term is overloaded in DBMS context. The irony is that in Wikipedia "projection" is expla

Re: [HACKERS] Surjective functional indexes

2017-05-28 Thread Konstantin Knizhnik
On 05/27/2017 09:50 PM, Peter Eisentraut wrote: On 5/25/17 12:30, Konstantin Knizhnik wrote: Functions like (info->>'name') are named "surjective" ni mathematics. A surjective function is one where each value in the output type can be obtained by some input value

[HACKERS] Logical replication & corrupted pages recovery

2017-05-26 Thread Konstantin Knizhnik
seems to be much more challenged task. If somebody has already thought about this problem, have some plan or may be even ready solution for it, please share your thoughts. Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Surjective functional indexes

2017-05-25 Thread Konstantin Knizhnik
On 25.05.2017 19:37, Tom Lane wrote: Konstantin Knizhnik writes: My proposal is to check value of function for functional indexes instead of just comparing set of effected attributes. Obviously, for some complex functions it may have negative effect on update speed. This is why I have added

[HACKERS] Surjective functional indexes

2017-05-25 Thread Konstantin Knizhnik
is possible to explicitly disable it and make decision weather index needs to be updated or not only based on set of effected attributes. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/access/common/reloptions.c b/src/

Re: [HACKERS] Cached plans and statement generalization

2017-05-25 Thread Konstantin Knizhnik
On 10.05.2017 19:11, Konstantin Knizhnik wrote: Based on the Robert's feedback and Tom's proposal I have implemented two new versions of autoprepare patch. First version is just refactoring of my original implementation: I have extracted common code into prepare_cache

Re: [HACKERS] Cached plans and statement generalization

2017-05-18 Thread Konstantin Knizhnik
On 15.05.2017 18:31, Robert Haas wrote: On Wed, May 10, 2017 at 12:11 PM, Konstantin Knizhnik wrote: Robert, can you please explain why using TRY/CATCH is not safe here: This is definitely not a safe way of using TRY/CATCH. This has been discussed many, many times on this mailing list

Re: [HACKERS] Cached plans and statement generalization

2017-05-12 Thread Konstantin Knizhnik
On 12.05.2017 18:23, Bruce Momjian wrote: On Fri, May 12, 2017 at 10:50:41AM +0300, Konstantin Knizhnik wrote: Definitely changing session context (search_path, date/time format, ...) may cause incorrect behavior of cached statements. I wonder if we should clear the cache whenever any SET

Re: [HACKERS] Cached plans and statement generalization

2017-05-12 Thread Konstantin Knizhnik
On 12.05.2017 03:58, Bruce Momjian wrote: On Thu, May 11, 2017 at 10:41:45PM +0300, Konstantin Knizhnik wrote: This is why I have provided second implementation which replace literals with parameters after raw parsing. Certainly it is slower than first approach. But still provide significant

Re: [HACKERS] Cached plans and statement generalization

2017-05-11 Thread Konstantin Knizhnik
On 05/11/2017 10:52 PM, Andres Freund wrote: On 2017-05-11 22:48:26 +0300, Konstantin Knizhnik wrote: On 05/11/2017 09:31 PM, Tom Lane wrote: Bruce Momjian writes: Good point. I think we need to do some measurements to see if the parser-only stage is actually significant. I have a hunch

Re: [HACKERS] Cached plans and statement generalization

2017-05-11 Thread Konstantin Knizhnik
lane Yes, my results shows that pg_parse_query adds not so much overhead: 206k TPS for my first variant with string literal substitution and modified query text used as hash key vs. 181k. TPS for version with patching raw parse tree constructed by pg_parse_query. -- Konstantin Knizhnik Pos

Re: [HACKERS] Cached plans and statement generalization

2017-05-11 Thread Konstantin Knizhnik
On 05/11/2017 06:12 PM, Bruce Momjian wrote: On Wed, May 10, 2017 at 07:11:07PM +0300, Konstantin Knizhnik wrote: I am going to continue work on this patch I will be glad to receive any feedback and suggestions for its improvement. In most cases, applications are not accessing Postgres directly

Re: [HACKERS] Cached plans and statement generalization

2017-05-10 Thread Konstantin Knizhnik
be it is even more perspective approach, but definitely much more invasive and harder to implement. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 6e52eb7..f2eb0

Re: [HACKERS] Why type coercion is not performed for parameters?

2017-05-05 Thread Konstantin Knizhnik
iteral I have to set UNKNOWNOID type otherwise a lot of queries will not work. .m -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

[HACKERS] Why type coercion is not performed for parameters?

2017-05-05 Thread Konstantin Knizhnik
tions. The fix in func_get_detail functions solves the problem and doesn't cause some new issues: all regression tests are passed. So my question is whether it is possible to use the same rule for type coercion of parameters as for constant? -- Konstantin Knizhnik Postgres Professional:

Re: [HACKERS] Bug in prepared statement cache invalidation?

2017-05-02 Thread Konstantin Knizhnik
invalidation of all functions in case of DDL seems to be more acceptable solution. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Bug in prepared statement cache invalidation?

2017-05-02 Thread Konstantin Knizhnik
ot of changes, in all PL implementations. Looks like no easy fix is possible here. I am not sure how critical is this problem. Definitely it rarely happens, but lack of normal workarounds (restart backend, recreate function?) seems to be disappointing. -- Konstantin Knizhnik Postgres Professional

Re: [HACKERS] Cached plans and statement generalization

2017-05-02 Thread Konstantin Knizhnik
On 01.05.2017 18:52, Robert Haas wrote: On Fri, Apr 28, 2017 at 6:01 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Any comments and suggestions for future improvement of this patch are welcome. +PG_TRY(); +{ +

Re: [HACKERS] Cached plans and statement generalization

2017-04-28 Thread Konstantin Knizhnik
nt of this patch are welcome. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index cd39167..4fbc8b7 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/n

[HACKERS] Bug in prepared statement cache invalidation?

2017-04-28 Thread Konstantin Knizhnik
ERY p -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
during CREATE processing though, so that might not be too bad. But this is getting off topic - I just wanted to capture the idea while it was rumbling around. I think that it will be enough to handle modification of search path and invalidate prepared statements cache in this case. -- Konsta

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
On 26.04.2017 10:49, Konstantin Knizhnik wrote: On 26.04.2017 04:00, Tsunakawa, Takayuki wrote: Are you considering some upper limit on the number of prepared statements? In this case we need some kind of LRU for maintaining cache of autoprepared statements. I think that it is good idea

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
On 26.04.2017 04:00, Tsunakawa, Takayuki wrote: From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Konstantin Knizhnik Well, first of all I want to share results I already get: pgbench with default parameters, scale 10 and one connection: So

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
ched plans? As I already mentioned, the same problem can be reproduced with explicitly prepared statements. Greetings, Andres Freund -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
nction and are invalidated by PlanCacheRelCallback. - Andres -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 04/25/2017 11:40 PM, Serge Rielau wrote: On Apr 25, 2017, at 1:37 PM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: SELECT ‘hello’::CHAR(10) || ‘World’, 5 + 6; You can substitute ‘hello’, ‘World’, 5, and 6. But not 10. I am substituting only string literals.

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 04/25/2017 08:09 PM, Serge Rielau wrote: On Tue, Apr 25, 2017 at 9:45 AM, Konstantin Knizhnik wrote: On 25.04.2017 19:12, Serge Rielau wrote: On Apr 25, 2017, at 8:11 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Another problem is

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 04/25/2017 07:54 PM, David Fetter wrote: On Tue, Apr 25, 2017 at 06:11:09PM +0300, Konstantin Knizhnik wrote: On 24.04.2017 21:43, Andres Freund wrote: Hi, On 2017-04-24 11:46:02 +0300, Konstantin Knizhnik wrote: So what I am thinking now is implicit query caching. If the same query with

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 25.04.2017 19:12, Serge Rielau wrote: On Apr 25, 2017, at 8:11 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Another problem is caused by using integer literals in context where parameters can not be used, for example "order by 1”. You will also need t

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 24.04.2017 21:43, Andres Freund wrote: Hi, On 2017-04-24 11:46:02 +0300, Konstantin Knizhnik wrote: So what I am thinking now is implicit query caching. If the same query with different literal values is repeated many times, then we can try to generalize this query and replace it with

Re: [HACKERS] Cached plans and statement generalization

2017-04-24 Thread Konstantin Knizhnik
On 24.04.2017 13:24, Alexander Korotkov wrote: Hi, Konstantin! On Mon, Apr 24, 2017 at 11:46 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: There were a lot of discussions about query plan caching in hackers mailing list, but I failed to find some clear

[HACKERS] Cached plans and statement generalization

2017-04-24 Thread Konstantin Knizhnik
option was already considered and if it was for some reasons rejected: can you point me at this reasons? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] FDW and parallel execution

2017-04-11 Thread Konstantin Knizhnik
On 04.04.2017 13:29, Kyotaro HORIGUCHI wrote: Hi, At Sun, 02 Apr 2017 16:30:24 +0300, Konstantin Knizhnik wrote in <58e0fcf0.2070...@postgrespro.ru> Hi hackers and personally Robet (you are the best expert in both areas). I want to ask one more question concerning parallel executi

[HACKERS] FDW and parallel execution

2017-04-02 Thread Konstantin Knizhnik
76 rows=2284376 width=8) -> Hash (cost=32605.64..32605.64 rows=1500032 width=8) -> Foreign Scan on customer_fdw (cost=0.00..32605.64 rows=1500032 width=8) The plans look very similar, but first one is parallel and second - not. My FDW provides im

Re: [HACKERS] Parallel query execution with SPI

2017-03-31 Thread Konstantin Knizhnik
On 31.03.2017 13:48, Robert Haas wrote: On Fri, Mar 31, 2017 at 3:33 AM, Konstantin Knizhnik wrote: It is possible to execute query concurrently using SPI? If so, how it can be enforced? I tried to open cursor with CURSOR_OPT_PARALLEL_OK flag but it doesn't help: query is executed by s

[HACKERS] Parallel query execution with SPI

2017-03-31 Thread Konstantin Knizhnik
state->portal = SPI_cursor_open_with_args(NULL, fsstate->query, fsstate->numParams, argtypes, values, nulls, true, CURSOR_OPT_PARALLEL_OK); ... SPI_cursor_fetch(fsstate->portal, true, 1); Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespr

Re: [HACKERS] WIP: Faster Expression Processing v4

2017-03-13 Thread Konstantin Knizhnik
1171 ms Your patch 6420 ms 1034 ms VOPS 396 ms 249 ms VOPS + patch367 ms 233 ms -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

[HACKERS] Two questions about Postgres parser

2017-02-27 Thread Konstantin Knizhnik
::float8) from foo; select coalesce(c, 0.0::float8::complex) from foo; coalesce -- (0 rows) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] VOPS: vectorized executor for Postgres: how to speedup OLAP queries more than 10 times without changing anything in Postgres executor

2017-02-16 Thread Konstantin Knizhnik
and '1997-01-01'::date and l_discount between 0.08 and 0.1 and l_quantity < 24; On 13.02.2017 17:12, Konstantin Knizhnik wrote: Hello hackers, There were many discussions concerning possible ways of speeding-up Postgres. Different approaches were suggested:

Re: [HACKERS] Sum aggregate calculation for single precsion real

2017-02-14 Thread Konstantin Knizhnik
On 14.02.2017 16:59, Jim Nasby wrote: On 2/13/17 10:45 AM, Konstantin Knizhnik wrote: It is not true - please notice query execution time of this two queries: I bet you'd get even less difference if you simply cast to float8 instead of adding 0.0. Same result, no floating point add

  1   2   3   >