Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
On 02/27/2016 06:57 AM, Robert Haas wrote: On Sat, Feb 27, 2016 at 1:49 AM, Konstantin Knizhnik wrote: pg_tsdtm is based on another approach: it is using system time as CSN and doesn't require arbiter. In theory there is no limit for scalability. But differences in system time and nece

Re: [HACKERS] Relation cache invalidation on replica

2016-02-26 Thread Konstantin Knizhnik
On 02/27/2016 04:16 AM, Simon Riggs wrote: On 27 February 2016 at 00:33, Simon Riggs mailto:si...@2ndquadrant.com>> wrote: On 27 February 2016 at 00:29, Andres Freund mailto:and...@anarazel.de>> wrote: On 2016-02-26 18:05:55 +0300, Konstantin Knizhnik wrote: >

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Konstantin Knizhnik
On 02/27/2016 06:54 AM, Robert Haas wrote: On Fri, Feb 26, 2016 at 10:56 PM, Konstantin Knizhnik wrote: We do not have formal prove that proposed XTM is "general enough" to handle all possible transaction manager implementations. But there are two general ways of dealing with

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Konstantin Knizhnik
visibility check. On 02/27/2016 01:48 AM, Kevin Grittner wrote: On Fri, Feb 26, 2016 at 2:19 PM, Konstantin Knizhnik wrote: pg_tsdtm is based on another approach: it is using system time as CSN Which brings up an interesting point, if we want logical replication to be free of serialization ano

Re: [HACKERS] The plan for FDW-based sharding

2016-02-28 Thread Konstantin Knizhnik
have not (yet) reviewed the literature for it. The reference to the article is at our WiKi pages explaining our DTM: https://wiki.postgresql.org/wiki/DTM http://research.microsoft.com/en-us/people/samehe/clocksi.srds2013.pdf -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
Thank you very much for you comments. On 01.03.2016 18:19, Robert Haas wrote: On Sat, Feb 27, 2016 at 2:29 AM, Konstantin Knizhnik wrote: How do you prevent clock skew from causing serialization anomalies? If node receives message from "feature" it just needs to wait until this fut

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
ion manager API would end up similarly situated. IMHO non-stable API is better than lack of API. Just because it makes it possible to implement features in modular way. And refactoring of API is not so difficult thing... -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The R

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
pg_tsdtm provides mapping between local XIDs and global CSNs. Visibility checking rules looks on CSNs, not on XIDs. In both cases if system is for some reasons restarted and DTM plugin failed to be loaded, you can still access database locally. No data can be lost. -- Konstantin Knizhnik Po

Re: [HACKERS] PROPOSAL: Fast temporary tables

2016-03-01 Thread Konstantin Knizhnik
t's tables. Actually translating that into relcache and everything else would be a serious amount of work. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Konstantin Knizhnik
P and simple OLAP. For OLTP we definitely need transaction manager to provide global consistency. And we have actually prototype of integration postgres_fdw with out pg_dtm and pg_tsdtm transaction managers. The results are quite IMHO promising (see attached diagram). -- Konstantin Knizhnik Pos

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Konstantin Knizhnik
mprovements in this area to be part of core. None of that means I would support any particular hook proposal, of course. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Optimizer questions

2016-03-08 Thread Konstantin Knizhnik
On 03/08/2016 07:01 AM, Tom Lane wrote: Konstantin Knizhnik writes: Attached please find improved version of the optimizer patch for LIMIT clause. This patch isn't anywhere close to working after 3fc6e2d7f5b652b4. (TBH, the reason I was negative about this upthread is that I had that o

Re: [HACKERS] Optimizer questions

2016-03-09 Thread Konstantin Knizhnik
construction of the path instead of comparing costs of full paths. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.post

Re: [HACKERS] Optimizer questions

2016-03-10 Thread konstantin knizhnik
On Mar 10, 2016, at 1:56 AM, Tom Lane wrote: > Konstantin Knizhnik writes: >> I think that the best approach is to generate two different paths: >> original one, when projection is always done before sort and another one >> with postponed projection of non-trivial col

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread Konstantin Knizhnik
committable during this 'fest. I think we'd be well advised to boot it to the 2016-09 CF and focus our efforts on other stuff that has a better chance of getting finished this month. regards, tom lane -- Konstantin Knizhnik Postgres Professional: http://

Re: [HACKERS] Batch update of indexes

2016-03-14 Thread Konstantin Knizhnik
Hi David, Rebased patch is attached. On 14.03.2016 15:09, David Steele wrote: Hi Konstantin, On 2/3/16 11:47 AM, Konstantin Knizhnik wrote: Attached please find patch for "ALTER INDEX ... WHERE ..." clause. It is now able to handle all three possible situations: 1. Making index pa

[HACKERS] Applying logical replication changes by more than one process

2016-03-19 Thread Konstantin Knizhnik
done using current logical replication mechanism when changes of each slot are applied by more than one process? Or the only alternative is to write/read origin LSNs in WAL myself, for example using custom WAL records? Thanks in advance! -- Konstantin Knizhnik Postgres Professional: http://www.

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-21 Thread Konstantin Knizhnik
On 21.03.2016 15:10, Petr Jelinek wrote: Hi, On 19/03/16 11:46, Konstantin Knizhnik wrote: Hi, I am trying to use logical replication mechanism in implementation of PostgreSQL multimaster and faced with one conceptual problem. Originally logical replication was intended to support

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-21 Thread konstantin knizhnik
On Mar 21, 2016, at 4:30 PM, Petr Jelinek wrote: > On 21/03/16 14:25, Andres Freund wrote: >> On 2016-03-21 14:18:27 +0100, Petr Jelinek wrote: >>> On 21/03/16 14:15, Andres Freund wrote: > Only when the origin is actually setup for the current session. You > need > to call the replor

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-22 Thread konstantin knizhnik
On Mar 22, 2016, at 10:10 AM, Craig Ringer wrote: > On 22 March 2016 at 14:32, konstantin knizhnik > wrote: > >> Ah you mean because with wal_log=true the origin advance is in different WAL >> record than commit? OK yeah you might be one transaction behind then, true. &

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-22 Thread konstantin knizhnik
On Mar 22, 2016, at 11:14 AM, Petr Jelinek wrote: > > And each slot means connection with logical decoding attached to it so you > don't really want to have thousands of those anyway. I think you'll hit other > problems faster than loop over slots becomes problem if you plan to keep all > of t

[HACKERS] avg,first,last,median in one query

2016-03-24 Thread Konstantin Knizhnik
er-defined aggregate function which uses array_appendand so materialize all values in memory: https://wiki.postgresql.org/wiki/Aggregate_Median 3. Using percentile aggregate: http://blog.jooq.org/2015/01/06/how-to-emulate-the-median-aggregate-function-using-inverse-distribution-functions/ Th

[HACKERS] Very small patch for decode.c

2016-03-30 Thread Konstantin Knizhnik
i; -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Limit and inherited tables

2016-01-15 Thread Konstantin Knizhnik
er patch for it. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Limit and inherited tables

2016-01-15 Thread Konstantin Knizhnik
ows=1 loops=1) But if sort is performed by non-indexed fields, then current behaviour will be inefficient and can be significantly improved by pushing limits to remote hosts. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via p

Re: [HACKERS] Optimizer questions

2016-01-18 Thread Konstantin Knizhnik
s.mime_id = 904 OR s.mime_id = 908 ) AND m.mime_id = o.mime_id AND o.owner = uo.user_id AND o.creator = uc.user_id ORDER BY s.mtime LIMIT 9; -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The

[HACKERS] Batch update of indexes

2016-01-20 Thread Konstantin Knizhnik
ase: search condition is exactly the same as partial index condition. Optimal plan should be: Index Scan using idx1 on t (cost=0.00..4.13 rows=12263 width=0) Index Cond: (c1 < '10'::double precision) What do you think about this approach? Will it be useful to work in this direction? Or there are some better solutions for the problem? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Batch update of indexes

2016-01-20 Thread Konstantin Knizhnik
nk it'll be interesting to you. So small patch... Why it was not accepted? I do no see any problems with it... -- Anastasia Lubennikova Postgres Professional:http://www.postgrespro.com The Russian Postgres Company -- Konstantin Knizhnik Postgres Professional: http://www.postgrespr

Re: [HACKERS] Proposal for UPDATE: do not insert new tuple on heap if update does not change data

2016-01-20 Thread Konstantin Knizhnik
r any further help with testing and ideas. Attached is a script with minimal test case. Kind regards, Gasper Zejn -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Batch update of indexes

2016-01-20 Thread konstantin knizhnik
On Jan 21, 2016, at 5:14 AM, Simon Riggs wrote: > On 20 January 2016 at 14:55, Konstantin Knizhnik > wrote: > Hi, > > Hi, I glad to see that you interested in that too. > I think this is a good feature and I think it will be very useful to have. > I have already m

Re: [HACKERS] Batch update of indexes

2016-01-21 Thread Konstantin Knizhnik
On 21.01.2016 10:14, Simon Riggs wrote: On 21 January 2016 at 06:41, konstantin knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Certainly for B-Tree we can organize insert buffer (or pending list) as sorted array or also as a tree. But in both case complexity of search i

Re: [HACKERS] Batch update of indexes

2016-01-21 Thread Konstantin Knizhnik
es during index scan and provide proper recovery of main index in case of failure (assuming that pending list is maintained in memory and is lost after the fault). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers ma

Re: [HACKERS] Batch update of indexes

2016-01-26 Thread Konstantin Knizhnik
ed to be interesting by community, I will try to address these issues. On 20.01.2016 12:28, Konstantin Knizhnik wrote: Hi hackers, I want to know opinion of community about possible ways of solving quite common problem: increasing insert speed while still providing indexes for efficient executi

Re: [HACKERS] Optimizer questions

2016-01-30 Thread Konstantin Knizhnik
34 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=10) Index Cond: (t1.k = k) Planning time: 0.537 ms Execution time: 0.241 ms (7 rows) On 01/30/2016 01:01 AM, Alexander Korotkov wrote: On Fri, Jan 8, 2016 at 11:58 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.r

Re: [HACKERS] PATCH: index-only scans with partial indexes

2016-01-31 Thread Konstantin Knizhnik
this point I'm not sure if either Kyotaro or Tomas should be considered the patch author ... maybe both?) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] PATCH: index-only scans with partial indexes

2016-02-02 Thread Konstantin Knizhnik
On 01.02.2016 01:11, Alvaro Herrera wrote: Konstantin Knizhnik wrote: I am very interested in this patch because it allows to use partial indexes to ... speed up inserts. I have implemented "ALTER INDEX ... WHERE ..." construction which allows to change predicate of partial index without

Re: [HACKERS] Batch update of indexes

2016-02-03 Thread Konstantin Knizhnik
ether with "index-only scans with partial indexes" patch: http://www.postgresql.org/message-id/560c7213.3010...@2ndquadrant.com only in this case regression test will produce expected output. On 27.01.2016 23:15, Robert Haas wrote: On Wed, Jan 20, 2016 at 4:28 AM, Konstantin Kniz

Re: [HACKERS] Batch update of indexes

2016-02-03 Thread konstantin knizhnik
On Feb 4, 2016, at 2:00 AM, Jim Nasby wrote: > > My suspicion is that it would be useful to pre-order the new data before > trying to apply it to the indexes. Sorry, but ALTER INDEX is expected to work for all indexes, not only B-Tree, and for them sorting may not be possible... But for B-Tre

[HACKERS] eXtensible Transaction Manager API (v2)

2016-02-10 Thread Konstantin Knizhnik
age. The complete PostgreSQL branch with all our changes can be found here: https://github.com/postgrespro/postgres_cluster -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/contrib/pg_tsdtm/Makefile b/contrib/pg_tsdtm/Makefil

[HACKERS] Clock with Adaptive Replacement

2016-02-11 Thread Konstantin Knizhnik
ARC (CAR is inspired by ARC, but it is different algorithm). As far as I know there are several problems with current clock-sweep algorithm in PostgreSQL, especially for very large caches. May be CAR can address some of them? -- Konstantin Knizhnik Postgres Professional: http

Re: [HACKERS] Clock with Adaptive Replacement

2016-02-12 Thread Konstantin Knizhnik
T can address this problem because there are not counters - justs single bit per page. On 12.02.2016 18:55, Robert Haas wrote: On Thu, Feb 11, 2016 at 4:02 PM, Konstantin Knizhnik wrote: What do you think about improving cache replacement clock-sweep algorithm in PostgreSQL with adaptive ve

Re: [HACKERS] pglogical - logical replication contrib module

2016-02-17 Thread Konstantin Knizhnik
are a lot simpler than the sort of gymnastics done by ALTER TABLE, etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] pglogical - logical replication contrib module

2016-02-17 Thread Konstantin Knizhnik
. On 17.02.2016 12:16, Craig Ringer wrote: On 17 February 2016 at 16:24, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Thanks for your explanation. I have to agree with your arguments that in general case replication of DDL statement using logical decoding seems

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Konstantin Knizhnik
ostgres XC/XL features, Postgres XC/XL will probably remain a separate fork of Postgres. I don't think anyone knows the answer to this question, and I don't know how to find the answer except to keep going with our current FDW sharding approach. -- Konstantin Knizhnik Postgres Profess

Re: [HACKERS] Relation cache invalidation on replica

2016-02-26 Thread Konstantin Knizhnik
tgreSQL 9.5.1. --- Dmitry Vasilyev Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
n't think about distributed transactions support doesn't mean there no "other people", who has different ideas on postgres future. That's why we propose this patch, let's play the game ! I don't like to play games with the architecture of PostgreSQL. -- Konstantin

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
On 02/26/2016 09:30 PM, Alvaro Herrera wrote: Konstantin Knizhnik wrote: Yes, it is certainly possible to develop cluster by cloning PostgreSQL. But it cause big problems both for developers, which have to permanently synchronize their branch with master, and, what is more important, for

[HACKERS] Logical decoding restart problems

2016-08-19 Thread konstantin knizhnik
r recreation of the slot? Is there any mechanism in xlog which can enforce consistent decoding of transaction (so that no transaction records are missed)? May be I missed something but I didn't find any "record_number" or something else which can identify first record of transaction.

Re: [HACKERS] Logical decoding restart problems

2016-08-19 Thread Konstantin Knizhnik
ot;mtm_slot_1" LOGICAL 0/0 ("startup_params_format" '1', "max_proto_version" '1', "min_proto_version" '1', "forward_changesets" '1', "mtm_replication_mode" 'recovered'); I have als

Re: [HACKERS] Logical decoding restart problems

2016-08-20 Thread konstantin knizhnik
Thank you for answers. > No, you don't need to recreate them. Just advance your replication identifier > downstream and request a replay position in the future. Let the existing slot > skip over unwanted data and resume where you want to start replay. > > You can advance the replication origins

Re: [HACKERS] UPSERT strange behavior

2016-08-25 Thread Konstantin Knizhnik
e correct, what is the best workaround for the problem if we really need to have to separate indexes and want to enforce unique constraint for both keys? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list

[HACKERS] Handling dropped attributes in pglogical_proto

2016-09-28 Thread Konstantin Knizhnik
t->atttypid == InvalidOid) { + continue; + } + kind = pq_getmsgbyte(in); + switch (kind) { case 'n': /* null */ -- Konstantin Knizhnik Postgres Professional: http://www.postgr

[HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-06 Thread Konstantin Knizhnik
7ef9e2 in BackendRun (port=0x24dee00) at postmaster.c:4357 #28 0x007ef10c in BackendStartup (port=0x24dee00) at postmaster.c:4029 #29 0x007eb6cc in ServerLoop () at postmaster.c:1753 #30 0x007eacb8 in PostmasterMain (argc=3, argv=0x24bd660) at postmaster.c:1361 #31 0x000

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-06 Thread Konstantin Knizhnik
On 06.10.2017 15:29, Petr Jelinek wrote: On 06/10/17 12:16, Konstantin Knizhnik wrote: When creating logical replication slots we quite often get the following error: ERROR: cannot build an initial slot snapshot when MyPgXact->xmin already is valid which cause restart of WAL sender.

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-07 Thread Konstantin Knizhnik
at logical decoding snapshot is being built and using that to skip catalog access which does not seem very pretty. It is not quite clear from the comment why it is not possible to overwrite MyPgXact->xmin or just use existed value. -- Konstantin Knizhnik Postgres Professional: http://www.postgr

[HACKERS] Slow synchronous logical replication

2017-10-07 Thread konstantin knizhnik
In our sharded cluster project we are trying to use logical relication for providing HA (maintaining redundant shard copies). Using asynchronous logical replication has not so much sense in context of HA. This is why we try to use synchronous logical replication. Unfortunately it shows very bad p

Re: [HACKERS] Slow synchronous logical replication

2017-10-07 Thread Konstantin Knizhnik
On 10/07/2017 10:42 PM, Andres Freund wrote: Hi, On 2017-10-07 22:39:09 +0300, konstantin knizhnik wrote: In our sharded cluster project we are trying to use logical relication for providing HA (maintaining redundant shard copies). Using asynchronous logical replication has not so much sense

Re: [HACKERS] Slow synchronous logical replication

2017-10-09 Thread Konstantin Knizhnik
rarely than other or is not updated at all (for example because communication channels between this node is broken), then all backens will stuck. Also all backends are competing for the single SyncRepLock, which also can be a contention point. -- Konstantin Knizhnik Postgres Professional: http:/

Re: [HACKERS] Columnar storage support

2017-10-10 Thread Konstantin Knizhnik
us is probably what you are looking for. jD -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] Slow synchronous logical replication

2017-10-11 Thread Konstantin Knizhnik
On 11.10.2017 10:07, Craig Ringer wrote: On 9 October 2017 at 15:37, Konstantin Knizhnik wrote: Thank you for explanations. On 08.10.2017 16:00, Craig Ringer wrote: I think it'd be helpful if you provided reproduction instructions, test programs, etc, making it very clear when thing

Re: [HACKERS] Slow synchronous logical replication

2017-10-12 Thread Konstantin Knizhnik
On 12.10.2017 04:23, Craig Ringer wrote: On 12 October 2017 at 00:57, Konstantin Knizhnik wrote: The reason of such behavior is obvious: wal sender has to decode huge transaction generate by insert although it has no relation to this publication. It does. Though I wouldn't expect any

[HACKERS] Deadlock in ALTER SUBSCRIPTION REFRESH PUBLICATION

2017-10-24 Thread Konstantin Knizhnik
244d6e0) at main.c:228 The reason of this deadlock seems to be clear: ALTER SUBSCRIPTION starts transaction at one node and tries to create slot at other node, which waiting for completion of all active transaction while building scnapshpot. Is there any way to avoid this deadlock? The same de

Re: [HACKERS] Secondary index access optimizations

2017-11-06 Thread Konstantin Knizhnik
On 11/06/2017 04:27 AM, Thomas Munro wrote: On Fri, Sep 8, 2017 at 3:58 AM, Konstantin Knizhnik wrote: Updated version of the patch is attached to this mail. Also I added support of date type to operator_predicate_proof to be able to imply (logdate <= '2017-03-31') from (logdate

Re: [HACKERS] SQL procedures

2017-11-08 Thread Konstantin Knizhnik
have not done it yet, because there seems to be no chances to push this patch to community. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

[HACKERS] Aggregates push-down to partitions

2017-11-09 Thread Konstantin Knizhnik
estions about the best approach to implement this feature? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Aggregates push-down to partitions

2017-11-10 Thread Konstantin Knizhnik
->  Seq Scan on derived2  (cost=0.00..14425.00 rows=100 width=0) ->  Foreign Scan on derived_fdw  (cost=100.00..212.39 rows=3413 width=0) (6 rows) Are there some principle problems? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgre

Re: [HACKERS] Partition-wise aggregation/grouping

2017-11-11 Thread Konstantin Knizhnik
d approach is easier for implementation. But in case of sharded table, distributed query may need to traverse both remote and local shards and this approach doesn't allow to processed several local shards in parallel. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The R

Re: [HACKERS] Partition-wise aggregation/grouping

2017-11-13 Thread Konstantin Knizhnik
On 11.11.2017 23:29, Konstantin Knizhnik wrote: On 10/27/2017 02:01 PM, Jeevan Chalke wrote: Hi, Attached new patch-set here. Changes include: 1. Added separate patch for costing Append node as discussed up-front in the patch-set. 2. Since we now cost Append node, we don't

[HACKERS] On conflict update & hint bits

2016-09-30 Thread Konstantin Knizhnik
s content lock. and we get assertion failure in /* here, either share or exclusive lock is OK */ Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr))); So the question is whether it is correct that ExecOnConflictUpdate tries to access and update tuple without hol

Re: [HACKERS] On conflict update & hint bits

2016-09-30 Thread Konstantin Knizhnik
On 30.09.2016 19:37, Peter Geoghegan wrote: On Fri, Sep 30, 2016 at 5:33 PM, Konstantin Knizhnik wrote: Later we try to check tuple visibility: ExecCheckHeapTupleVisible(estate, &tuple, buffer); and inside HeapTupleSatisfiesMVCC try to set hint bit. So, you're using repeat

Re: [HACKERS] On conflict update & hint bits

2016-10-24 Thread Konstantin Knizhnik
te->mt_existing, buffer, false); +LockBuffer(buffer, BUFFER_LOCK_UNLOCK); + /* * Make tuple and any needed join variables available to ExecQual and * ExecProject. The EXCLUDED tuple is installed in ecxt_innertuple, while -- Konstantin Knizhnik Postgres Professional: htt

[HACKERS] Unlogged tables cleanup

2016-11-09 Thread Konstantin Knizhnik
res psql (10devel) Type "help" for help. postgres=# select * from foo1; x --- (0 rows) knizhnik@knizhnik:~/dtm-data$ ls -l fs/PG_10_201611041/12289/* -rw--- 1 knizhnik knizhnik 0 Nov 9 19:52 fs/PG_10_201611041/12289/32768 -rw--- 1 knizhnik knizhnik 0 Nov 9 19:52 fs/PG_10_2

Re: [HACKERS] Unlogged tables cleanup

2016-11-09 Thread konstantin knizhnik
On Nov 10, 2016, at 10:17 AM, Michael Paquier wrote: > > Hm.. I cannot reproduce what you see on Linux or macos. Perhaps you > have locally a standby pointing as well to this tablespace? No, it is latest sources from Postgres repository. Please notice that you should create new database and tab

[HACKERS] Two questions about Postgres parser

2017-02-27 Thread Konstantin Knizhnik
::float8) from foo; select coalesce(c, 0.0::float8::complex) from foo; coalesce -- (0 rows) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] WIP: Faster Expression Processing v4

2017-03-13 Thread Konstantin Knizhnik
1171 ms Your patch 6420 ms 1034 ms VOPS 396 ms 249 ms VOPS + patch367 ms 233 ms -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

[HACKERS] Parallel query execution with SPI

2017-03-31 Thread Konstantin Knizhnik
state->portal = SPI_cursor_open_with_args(NULL, fsstate->query, fsstate->numParams, argtypes, values, nulls, true, CURSOR_OPT_PARALLEL_OK); ... SPI_cursor_fetch(fsstate->portal, true, 1); Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespr

Re: [HACKERS] Parallel query execution with SPI

2017-03-31 Thread Konstantin Knizhnik
On 31.03.2017 13:48, Robert Haas wrote: On Fri, Mar 31, 2017 at 3:33 AM, Konstantin Knizhnik wrote: It is possible to execute query concurrently using SPI? If so, how it can be enforced? I tried to open cursor with CURSOR_OPT_PARALLEL_OK flag but it doesn't help: query is executed by s

[HACKERS] FDW and parallel execution

2017-04-02 Thread Konstantin Knizhnik
76 rows=2284376 width=8) -> Hash (cost=32605.64..32605.64 rows=1500032 width=8) -> Foreign Scan on customer_fdw (cost=0.00..32605.64 rows=1500032 width=8) The plans look very similar, but first one is parallel and second - not. My FDW provides im

Re: [HACKERS] FDW and parallel execution

2017-04-11 Thread Konstantin Knizhnik
On 04.04.2017 13:29, Kyotaro HORIGUCHI wrote: Hi, At Sun, 02 Apr 2017 16:30:24 +0300, Konstantin Knizhnik wrote in <58e0fcf0.2070...@postgrespro.ru> Hi hackers and personally Robet (you are the best expert in both areas). I want to ask one more question concerning parallel executi

Re: [HACKERS] ASOF join

2017-06-19 Thread Konstantin Knizhnik
On 16.06.2017 19:07, David Fetter wrote: On Fri, Jun 16, 2017 at 11:51:34AM +1200, Thomas Munro wrote: On Fri, Jun 16, 2017 at 4:20 AM, Konstantin Knizhnik wrote: I wonder if there were some discussion/attempts to add ASOF join to Postgres (sorry, may be there is better term for it, I am

Re: [HACKERS] ASOF join

2017-06-21 Thread Konstantin Knizhnik
e join instead of nested loop) but also simplifying writing of such queries. Or do you think that nobody will be interested in non-standard SQL extensions? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Error : undefined symbol : LWLockAssign in 9.6.3

2017-08-08 Thread Konstantin Knizhnik
o use IMCS with 9.6.3 and later Postgres versions. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

[HACKERS] Secondary index access optimizations

2017-08-14 Thread Konstantin Knizhnik
s if specified interval is covered by their conditions. I wonder if someone is familiar with this part of optimizer ad can easily fix it. Otherwise I am going to spend some time on solving this problems (if community think that such optimizations will be useful). -- Konstantin Knizhnik Postgres

Re: [HACKERS] Secondary index access optimizations

2017-08-14 Thread Konstantin Knizhnik
On 14.08.2017 12:37, Konstantin Knizhnik wrote: Hi hackers, I am trying to compare different ways of optimizing work with huge append-only tables in PostgreSQL where primary key is something like timestamp and queries are usually accessing most recent data using some secondary keys. Size

Re: [HACKERS] Secondary index access optimizations

2017-08-16 Thread Konstantin Knizhnik
On 14.08.2017 19:33, Konstantin Knizhnik wrote: On 14.08.2017 12:37, Konstantin Knizhnik wrote: Hi hackers, I am trying to compare different ways of optimizing work with huge append-only tables in PostgreSQL where primary key is something like timestamp and queries are usually accessing

Re: [HACKERS] [PATCH] Push limit to sort through a subquery

2017-08-17 Thread Konstantin Knizhnik
er by v limit 1 100 42 There is index for "u", so fetching records with smallest "u" values can be done without sorting, so times are similar. But in case of sorting by "v", pushing down limit allows to use TOP-1 instead of global sort and it reduces

[HACKERS] Cached plans and statement generalization

2017-04-24 Thread Konstantin Knizhnik
option was already considered and if it was for some reasons rejected: can you point me at this reasons? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Cached plans and statement generalization

2017-04-24 Thread Konstantin Knizhnik
On 24.04.2017 13:24, Alexander Korotkov wrote: Hi, Konstantin! On Mon, Apr 24, 2017 at 11:46 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: There were a lot of discussions about query plan caching in hackers mailing list, but I failed to find some clear

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 24.04.2017 21:43, Andres Freund wrote: Hi, On 2017-04-24 11:46:02 +0300, Konstantin Knizhnik wrote: So what I am thinking now is implicit query caching. If the same query with different literal values is repeated many times, then we can try to generalize this query and replace it with

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 25.04.2017 19:12, Serge Rielau wrote: On Apr 25, 2017, at 8:11 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Another problem is caused by using integer literals in context where parameters can not be used, for example "order by 1”. You will also need t

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 04/25/2017 07:54 PM, David Fetter wrote: On Tue, Apr 25, 2017 at 06:11:09PM +0300, Konstantin Knizhnik wrote: On 24.04.2017 21:43, Andres Freund wrote: Hi, On 2017-04-24 11:46:02 +0300, Konstantin Knizhnik wrote: So what I am thinking now is implicit query caching. If the same query with

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 04/25/2017 08:09 PM, Serge Rielau wrote: On Tue, Apr 25, 2017 at 9:45 AM, Konstantin Knizhnik wrote: On 25.04.2017 19:12, Serge Rielau wrote: On Apr 25, 2017, at 8:11 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Another problem is

Re: [HACKERS] Cached plans and statement generalization

2017-04-25 Thread Konstantin Knizhnik
On 04/25/2017 11:40 PM, Serge Rielau wrote: On Apr 25, 2017, at 1:37 PM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: SELECT ‘hello’::CHAR(10) || ‘World’, 5 + 6; You can substitute ‘hello’, ‘World’, 5, and 6. But not 10. I am substituting only string literals.

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
nction and are invalidated by PlanCacheRelCallback. - Andres -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
ched plans? As I already mentioned, the same problem can be reproduced with explicitly prepared statements. Greetings, Andres Freund -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
On 26.04.2017 04:00, Tsunakawa, Takayuki wrote: From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Konstantin Knizhnik Well, first of all I want to share results I already get: pgbench with default parameters, scale 10 and one connection: So

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
On 26.04.2017 10:49, Konstantin Knizhnik wrote: On 26.04.2017 04:00, Tsunakawa, Takayuki wrote: Are you considering some upper limit on the number of prepared statements? In this case we need some kind of LRU for maintaining cache of autoprepared statements. I think that it is good idea

Re: [HACKERS] Cached plans and statement generalization

2017-04-26 Thread Konstantin Knizhnik
during CREATE processing though, so that might not be too bad. But this is getting off topic - I just wanted to capture the idea while it was rumbling around. I think that it will be enough to handle modification of search path and invalidate prepared statements cache in this case. -- Konsta

[HACKERS] Bug in prepared statement cache invalidation?

2017-04-28 Thread Konstantin Knizhnik
ERY p -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cached plans and statement generalization

2017-04-28 Thread Konstantin Knizhnik
nt of this patch are welcome. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index cd39167..4fbc8b7 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/n

  1   2   3   >