[HACKERS] Cube extension improvement, GSoC

Hello, some time ago I started working on the data search system (about 100-200M of records) with queries consisted of several diapason and equality conditions, e.g.: WHERE dim1 BETWEEN 128 AND 137 AND WHERE dim2 BETWEEN 4815 AND 162342 AND WHERE dim3 = 42 ORDER BY dim1 ASC There are

Re: [HACKERS] Cube extension improvement, GSoC

in different records have different numbers of element? Stas Kelvich. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cube extension improvement, GSoC

:19 PM, Stas Kelvich stanc...@gmail.com wrote: I think we have at least 3 data types more or less similar to cube. 1) array of ranges 2) range of arrays 3) 2d arrays Semantically cube is most close to array or ranges. However array of ranges have huge storage overhead. Also we can

Re: [HACKERS] Cube extension improvement, GSoC

On May 14, 2013, at 4:53 PM, Alexander Korotkov wrote: Sounds promising. Did you examine how this technique can fit into GiST? In current GiST interface methods don't have access to parent entries. No, i didn't examine it yet. Anyway in this technique lots of changes should be performed to

[HACKERS] Cube extension kNN support

. This request selects rows ordered descending by 4th coordinate: SELECT * FROM objects ORDER BY objects.coord--4 LIMIT 10; Stas Kelvich. distances.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Cube extension point support // GSoC'13

option in both situations. I've also tested it on FreeBSD 9.0 and Ubuntu 12.04 with the same results. So is there some ideas how can I reproduce such results? Stas. points.diff Description: Binary data On Sep 16, 2013, at 10:48 AM, Heikki Linnakangas wrote: On 12.07.2013 14:57, Stas

[HACKERS] Cube extension types support

Hello, hackers. In this patch I've implemented support for different storage types for cubes. Now it supports float8, float4, int4, int2, int1. Type stored in the header of each cube, one for all coordinates. So for cubes with int1 coordinates it can save up to 8x disk space. Typed cubes can

[HACKERS] Cube extension split algorithm fix

Hello, hackers. I've fixed split algorithm that was implemented in cube extension. I've changed it according to the original Guttman paper (old version was more simple algorithm) and also ported Alexander Korotkov's algorithm from box datatype indexing that work faster and better on low

Re: [HACKERS] Cube extension kNN support

, kNN ordering 10 times slower than B-tree (on silly request for R-Tree, just as example), but still 100+ times faster than full scan on this table. Stas. On Sep 25, 2013, at 5:25 PM, Peter Eisentraut pete...@gmx.net wrote: On 9/22/13 7:38 PM, Stas Kelvich wrote: Here is the patch

Re: [HACKERS] Cube extension kNN support

selects rows ordered descending by 4th coordinate: SELECT * FROM objects ORDER BY objects.coord--4 LIMIT 10; Stas Kelvich. distances2.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Cube extension kNN support

Documentation along with style fix. distances2r3.patch Description: Binary data On 08 Feb 2015, at 00:32, Alexander Korotkov aekorot...@gmail.com wrote: Hi! On Sat, Feb 7, 2015 at 12:45 PM, Stas Kelvich stas.kelv...@gmail.com wrote: I had updated old patch with kNN operators for cube

Re: [HACKERS] Cube extension kNN support

...@2ndquadrant.com wrote: Sergey Konoplev wrote: On Thu, Mar 27, 2014 at 3:26 PM, Sergey Konoplev gray...@gmail.com wrote: On Sun, Sep 22, 2013 at 4:38 PM, Stas Kelvich stas.kelv...@gmail.com wrote: Here is the patch that introduces kNN search for cubes with euclidean, taxicab and chebyshev distances

Re: [HACKERS] Cube extension kNN support

2015, at 16:40, Alexander Korotkov <aekorot...@gmail.com> wrote: > > Hi! > > On Sat, May 9, 2015 at 6:53 AM, Stas Kelvich <stas.kelv...@gmail.com> wrote: > Patch is pretty ready, last issue was about changed extension interface, so > there should be migration script

[HACKERS] Tsvector editing functions

[] to_array(tsvector) converts tsvector to array of lexemes tsvector to_tsvector(text[]) converts array of lexemes to tsvector tsvector_funcs.diff Description: Binary data Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql

Re: [HACKERS] Cube extension kNN support

Hello, fixed. cube_distances.diff Description: Binary data --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company > On 01 Dec 2015, at 17:52, Teodor Sigaev <teo...@sigaev.ru> wrote: > > Patch looks good, but there ara some review not

Re: [HACKERS] Tsvector editing functions

Hello. Done with the list of suggestions. Also heavily edit delete function. tsvector_ops.diff Description: Binary data --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company > On 27 Nov 2015, at 15:13, Teodor Sigaev <teo...@sigaev.ru> wrote:

[HACKERS] Speedup twophase transactions

abalance = abalance + :delta WHERE aid = :to_aid; PREPARE TRANSACTION ':client_id.:scale'; COMMIT PREPARED ':client_id.:scale'; 2pc_xlog.diff Description: Binary data --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers

Re: [HACKERS] Speedup twophase transactions

Thanks, Kevin. > I assume that last one should have been *Patched master with 2PC”? Yes, this list should look like this: Current master without 2PC: ~42 ktps Current master with 2PC: ~22 ktps Patched master with 2PC: ~36 ktps And created CommitFest entry for this patch. -- Stas Kelv

Re: [HACKERS] Cube extension kNN support

Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > > Hi, > > On 12/16/2015 01:26 PM, Stas Kelvich wrote: >> Hi, thanks for the review. >> >>> 1) (nitpicking) There seem to be some minor whitespace issues, i.e. >>> trailing spaces, empty

Re: [HACKERS] Cube extension kNN support

patch attached. cube_distances.patch Description: Binary data > On 15 Dec 2015, at 21:46, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > > Hi, > > On 12/07/2015 03:47 PM, Stas Kelvich wrote: >> Hello, fixed. > > I've looked at the patch today, se

Re: [HACKERS] Speedup twophase transactions

r overflow problems. Hmm, that’s strange. Probably you set scale to big value, so that 10*:scale is bigger that int4? But i thought that pgbench will change aid columns to bigint if scale is more than 2. 2pc_xlog.v2.diff Description: Binary data --- Stas Kelvich Postgres Professional:

Re: [HACKERS] Tsvector editing functions

; 7) Some of the functions use intexterm that does not match the function > name. I see two such cases - to_tsvector and setweight. Is there a > reason for that? > Because sgml compiler wants unique indexterm. Both functions that you mentioned use overloading of arguments and have

Re: [HACKERS] Speedup twophase transactions

Description: Binary data Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company > On 08 Jan 2016, at 19:29, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > > Simon Riggs wrote: > >> I think we could do better still, but this l

Re: [HACKERS] Speedup twophase transactions

by > statefile re-creation serially, whereas on the master they'll usually be > executed in parallel. That’s interesting observation. Simon already pointed me to this problem in 2pc replay, but I didn’t thought that it is so slow. I’m now working on that. Stas Kelvich Postgres Professi

Re: [HACKERS] Speedup twophase transactions

My +1 for moving function to xlogutils.c too. Now call to this function goes through series of callbacks so it is hard to find it. Personally I found it only after I have implemented same function by myself (based on code in pg_xlogdump). Stas Kelvich Postgres Professional: http

Re: [HACKERS] Speedup twophase transactions

OLLBACK, so we can save transaction context and release it only if next command isn’t our designated COMMIT/ROLLBACK. But that is a big amount of work and requires changes to whole transaction pipeline in postgres. Anyway I suggest that we should consider that as a separate task. --- Stas Kelvich Pos

Re: [HACKERS] Tsvector editing functions

ny that you reintroduce the "unrecognized weight: %d" > (instead of %c) in tsvector_setweight_by_filter. > Ah, I was thinking about moving it to separate diff and messed. Fixed and attaching diff with same fix for old tsvector_setweight. tsvector_ops-v2.1.diff Description: Binary da

Re: [HACKERS] In-core regression tests for replication, cascading, archiving, PITR, etc.

l > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via p

Re: [HACKERS] Speedup twophase transactions

Agree, I had the same idea in my mind when was writing that script. I will migrate it to TAP suite and write a review for Michael Paquier's patch. Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company > On 26 Jan 2016, at 20:20, Alvaro Herrera <

Re: [HACKERS] Speedup twophase transactions

check.sh Description: Binary data --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company > On 12 Jan 2016, at 22:57, Simon Riggs <si...@2ndquadrant.com> wrote: > > On 12 January 2016 at 18:14, Andres Freund <and...@anarazel.de> wrote: > Hi,

Re: [HACKERS] Mac OS: invalid byte sequence for encoding "UTF8"

Hi. I tried that and confirm strange behaviour. It seems that problem with small cyrillic letter ‘х’. (simplest obscene language filter? =) That can be reproduced with simpler test Stas test.c Description: Binary data > On 27 Jan 2016, at 13:59, Artur Zakirov

Re: [HACKERS] tsearch_extras extension

wasn’t aware of you patch when started to work on that, so I think we can ask commiter to mention you in commit message (if it that will be commited). And how do you use ts_match_locs_array / ts_match_locs_array? To highlight search results? There is function called ts_headline that can mark matches with custo

Re: [HACKERS] Tsvector editing functions

ests. > -- > Teodor Sigaev E-mail: teo...@sigaev.ru > WWW: http://www.sigaev.ru/ > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.

[HACKERS] transam README small fix

changed it to actual call nesting. transam.readme.patch Description: Binary data --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

[HACKERS] 2PC support for pglogical

. pglogical_twophase.diff Description: Binary data --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] Speedup twophase transactions

ths. > * Add documentation for RecoverPreparedFromXLOG Done. --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company twophase_replay.v2.diff Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscript

Re: [HACKERS] Tsvector editing functions

essing. > > Thoughts? > Seems reasonable, done. tsvector_ops-v6.diff Description: Binary data --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: [HACKERS] async replication code

http://www.postgresql.org/docs/9.5/static/warm-standby.html#STREAMING-REPLICATION --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] eXtensible Transaction Manager API (v2)

es around zero. https://github.com/kelvich/postgres_xtm_docker --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Tsvector editing functions

> > On 11 Mar 2016, at 16:13, Stas Kelvich <s.kelv...@postgrespro.ru> wrote: > > >> On 10 Mar 2016, at 20:29, Teodor Sigaev <teo...@sigaev.ru> wrote: >> >> I would like to suggest rename both functions to array_to_tsvector and >> tsvector_to_a

Re: [HACKERS] Speedup twophase transactions

> On Apr 2, 2016, at 3:14 AM, Michael Paquier <michael.paqu...@gmail.com> wrote: > > On Fri, Apr 1, 2016 at 10:53 PM, Stas Kelvich <s.kelv...@postgrespro.ru> > wrote: >> I wrote: >>> While testing the patch, I found a bug in the recovery conflict

Re: [HACKERS] Speedup twophase transactions

> On 12 Apr 2016, at 15:47, Michael Paquier <michael.paqu...@gmail.com> wrote: > > On Mon, Apr 11, 2016 at 7:16 PM, Stas Kelvich wrote: >> Michael, it looks like that you are the only one person who can reproduce >> that bug. I’ve tried on bunch of OS’s and di

Re: [HACKERS] Speedup twophase transactions

ideas on why that can be caused by changing procedures of PREPARE replay. Just to keep things sane, here is my current diff: twophase_replay.v4.patch Description: Binary data -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers

Re: [HACKERS] Speedup twophase transactions

s just quick script to reproduce bug, that Michael faced. If there will be deterministic way to reproduce that bug, i'll rework it and move to 00X_twophase.pl -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pg

Re: [HACKERS] closing CommitFest 2016-03, feature freeze now in effect

ghtful decision, taking into account that absence of that patch in release can cause problems with replication in some cases as it was warned by Jesper[1] and Andres[2]. [1] http://www.postgresql.org/message-id/5707a8cc.6080...@redhat.com [2] http://www.postgresql.org/message-id/80856693-5065-4392

Re: [HACKERS] 'Create table if not exists as' breaks SPI_execute

> On 11 Apr 2016, at 18:41, Stas Kelvich <s.kelv...@postgrespro.ru> wrote: > > Hi. > > SPI_execute assumes that CreateTableAsStmt always have completionTag == > “completionTag”. > But it isn’t true in case of ‘IF NOT EXISTS’ present. > > > Sorry, I meant

[HACKERS] 'Create table if not exists as' breaks SPI_execute

Hi. SPI_execute assumes that CreateTableAsStmt always have completionTag == “completionTag”. But it isn’t true in case of ‘IF NOT EXISTS’ present. spi-cta.patch Description: Binary data -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent

Fwd: [HACKERS] Speedup twophase transactions

> On 08 Apr 2016, at 21:55, Jesper Pedersen <jesper.peder...@redhat.com> wrote: > > On 04/08/2016 02:42 PM, Robert Haas wrote: >> On Tue, Jan 26, 2016 at 7:43 AM, Stas Kelvich <s.kelv...@postgrespro.ru> >> wrote: >>> Hi, >>> >>>

Re: [HACKERS] Speedup twophase transactions

> On 08 Apr 2016, at 21:42, Robert Haas <robertmh...@gmail.com> wrote: > > On Tue, Jan 26, 2016 at 7:43 AM, Stas Kelvich <s.kelv...@postgrespro.ru> > wrote: >> Hi, >> >> Thanks for reviews and commit! > > I apologize for being clueless here, b

Re: [HACKERS] Speedup twophase transactions

> On 08 Apr 2016, at 16:09, Stas Kelvich <s.kelv...@postgrespro.ru> wrote: > > Tried on linux and os x 10.11 and 10.4. > > Still can’t reproduce, but have played around with your backtrace. > > I see in xlodump on slave following sequence of records: > > rmgr:

Re: [HACKERS] fd.c: flush data problems on osx

I don’t yet understand why are they happening. > > Greetings, > > Andres Freund > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers --- Stas Kelvich Postgres

[HACKERS] fd.c: flush data problems on osx

, is there any check that will guarantee that pg_flush_data will not end up with empty body on some platform? --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company # Minimal test testing streaming replication use strict; use warnings; use PostgresNode; use TestLib;

Re: [HACKERS] [PATCH] we have added support for box type in SP-GiST index

te to say we are treating a 2-dimensional box > as a point in 4-dimensional space? Or just say 4-d vector instead of 4-d point. Look like it will be enough rigorous. Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing

Re: [HACKERS] [PATCH] we have added support for box type in SP-GiST index

im indexing with kNN, but I think it is different problem and should be solved in a different way. Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 2PC support for pglogical

> On 24 Mar 2016, at 17:03, Robert Haas <robertmh...@gmail.com> wrote: > > On Wed, Mar 23, 2016 at 1:44 AM, Craig Ringer <cr...@2ndquadrant.com> wrote: >> On 10 March 2016 at 22:50, Stas Kelvich <s.kelv...@postgrespro.ru> wrote: >>> Hi. >>>

Re: [HACKERS] fd.c: flush data problems on osx

> On 18 Mar 2016, at 14:45, Stas Kelvich <s.kelv...@postgrespro.ru> wrote: >> >>> One possible solution for that is just fallback to pg_fdatasync in case >>> when offset = nbytes = 0. >> >> Hm, that's a bit heavyweight. I'd rather do an lseek(SEEK

Re: [HACKERS] fd.c: flush data problems on osx

tion to support the > offset = 0, nbytes = 0 case (via fseek(SEEK_END). It is already in this diff. I’ve added this few messages ago. flushdata.v4.patch Description: Binary data --- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-ha

Re: [HACKERS] transam README small fix

Thanks. Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company > On 04 Mar 2016, at 22:14, Robert Haas <robertmh...@gmail.com> wrote: > > On Tue, Mar 1, 2016 at 4:31 AM, Stas Kelvich <s.kelv...@postgrespro.ru> wrote: >> Transac

Re: [HACKERS] Speedup twophase transactions

On Mar 29, 2016, at 6:04 PM, David Steele wrote:It looks like you should post a new patch or respond to Michael's comments.  Marked as "waiting on author".Yep, here it is.On Mar 22, 2016, at 4:20 PM, Michael Paquier wrote:Looking at this

Re: [HACKERS] Speedup twophase transactions

rame #9: 0x000107e70f93 > postgres`LockAcquireExtended(locktag=0x7fff581f0238, lockmode=8, > sessionLock='\x01', dontWait='\0', reportMemoryError='\0') + 2819 at > lock.c:998 >frame #10: 0x000107e6a9a6 > postgres`StandbyAcquireAccessExclusiveLock(xid=

Re: [HACKERS] Speedup twophase transactions

> On 13 Apr 2016, at 01:04, Michael Paquier <michael.paqu...@gmail.com> wrote: > > On Wed, Apr 13, 2016 at 1:53 AM, Stas Kelvich <s.kelv...@postgrespro.ru> > wrote: >>> On 12 Apr 2016, at 15:47, Michael Paquier <michael.paqu...@gmail.com> wrote: >>

Re: [HACKERS] Naming of new tsvector functions

That would be a good convention if we were able to easily rename old functions. But now that will just create another pattern on top of three existing (no prefix, ts_*, tsvector_*). Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-h

Re: [HACKERS] Naming of new tsvector functions

), numnode(), strip() Recent commit added setweight(), delete(), unnest(), tsvector_to_array(), array_to_tsvector(), filter(). Last bunch can be painlessly renamed, for example to ts_setweight, ts_delete, ts_unnest, ts_filter. The question is what to do with old ones? Leave them as is? Rename to ts_

[HACKERS] tsvector filter problem

Hi. As discovered by Oleg Bartunov, current filter() function for tsvector can crash backend. Bug was caused erroneous usage of char type in memmove argument. tsvector_bugfix_type.diff Description: Binary data -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian

Re: [HACKERS] Naming of new tsvector functions

> On 04 May 2016, at 20:15, Tom Lane <t...@sss.pgh.pa.us> wrote: > > Stas Kelvich <s.kelv...@postgrespro.ru> writes: >>> On 04 May 2016, at 16:58, Tom Lane <t...@sss.pgh.pa.us> wrote: >>> The other ones are not so problematic because they do not conf

Re: [HACKERS] Naming of new tsvector functions

> On 04 May 2016, at 16:58, Tom Lane <t...@sss.pgh.pa.us> wrote: > > Stas Kelvich <s.kelv...@postgrespro.ru> writes: >>> On 03 May 2016, at 00:59, David Fetter <da...@fetter.org> wrote: >>> I suspect that steering that ship would be a good idea startin

Re: [HACKERS] Parser extensions (maybe for 10?)

ier to allow extensions to define custom parameters for WITH, than to extend parser. -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] Logical Replication WIP

lass: 6102 * Several time i’ve run in a situation where provider's postmaster ignores Ctrl-C until subscribed node is switched off. * Patch with small typos fixed attached. I’ll do more testing, just want to share what i have so far. typos.diff Description: Binary data -- Stas Kelvich Postg

Re: [HACKERS] Logical Replication WIP

--- 0/1530F30 | 7FFF/5E7F6A30 | 7FFF/5E7F6A30 | 7FFF/5E7F6A30 (1 row) -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Speedup twophase transactions

e segments should be re-read after cache drop) master, without constant cache_drop: time to recover 35 segments: 2h 25m (after that i tired to wait) expected total recovery time: 4.5 hours -- Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] logical decoding of two-phase transactions

MIT/ABORT 4. COMMIT/ABORT decoded and sent After step 3 there is no more memory state associated with that prepared tx, so if will fail between 3 and 4 then we can’t know GID unless we wrote it commit record (or table). -- Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian

Re: [HACKERS] logical decoding of two-phase transactions

we should invent something more nasty like writing them into a table. > That should eliminate Simon's > objection re the cost of tracking GIDs and still let us have access to > them when we want them, which is the best of both worlds really. Having 2PC decoding in core is a good thing an

Re: [HACKERS] Speedup twophase transactions

gt; KnownPreparedList in the code path that follows as well as elsewhere. Thanks Nikhil, now I got that. Since we are talking about promotion we are on different timescale and 1-10 second lag matters a lot. I think I have in my mind realistic scenario when proposed recovery code path will hit

[HACKERS] Walsender crash

14: 0x00010e76371f postgres`main(argc=3, argv=0x7fbcabc02b90) + 751 at main.c:228 frame #15: 0x7fffa951c255 libdyld.dylib`start + 1 frame #16: 0x7fffa951c255 libdyld.dylib`start + 1 Patch with lacking initStringInfo() attached. init_reply_message.diff Description: Binary

Re: [HACKERS] logical decoding of two-phase transactions

= 0/D8B43E28 and with 194-byte GID’s difference in WAL size is about 18% So using big GID’s (as J2EE does) can cause notable WAL bloat, while small GID’s are almost unnoticeable. May be we can introduce configuration option track_commit_gid by analogy with track_commit_timestamp and make tha

Re: [HACKERS] logical decoding of two-phase transactions

form prepare decoding with some kind of copied-end-edited snapshot. I’ll have a look at this. -- Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] logical decoding of two-phase transactions

points about this topic. Or, maybe, I’m failing to understand some points. Can we maybe setup skype call to discuss this and post summary here? Craig? Peter? -- Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mai

Re: [HACKERS] Logical decoding restart problems

decoding output plugin, and current postgres master). -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Speedup twophase transactions

> On 06 Sep 2016, at 04:41, Michael Paquier <michael.paqu...@gmail.com> wrote: > > On Sat, Sep 3, 2016 at 10:26 PM, Michael Paquier > <michael.paqu...@gmail.com> wrote: >> On Fri, Sep 2, 2016 at 5:06 AM, Simon Riggs <si...@2ndquadrant.com> wrote: >>

Re: [HACKERS] Speedup twophase transactions

> On 06 Sep 2016, at 12:09, Simon Riggs <si...@2ndquadrant.com> wrote: > > On 6 September 2016 at 09:58, Stas Kelvich <s.kelv...@postgrespro.ru> wrote: >> >> I'll check it against my failure scenario with subtransactions and post >> results or updated pat

Re: [HACKERS] Speedup twophase transactions

> On 07 Sep 2016, at 03:09, Michael Paquier <michael.paqu...@gmail.com> wrote: > >>> On 06 Sep 2016, at 12:03, Michael Paquier <michael.paqu...@gmail.com> wrote: >>> >>> On Tue, Sep 6, 2016 at 5:58 PM, Stas Kelvich <s.kelv...@postgrespro.ru&g

[HACKERS] Bug in two-phase transaction recovery

at StandbyRecoverPreparedTransactions() i’ve noticed that buffer for 2pc file is allocated in TopMemoryContext but never freed. That probably exists for a long time. gidlen_fixes.diff Description: Binary data standby_recover_pfree.diff Description: Binary data -- Stas Kelvich Postgres

Re: [HACKERS] Suggestions for first contribution?

handler.html [2] https://www.postgresql.org/docs/current/static/gist-extensibility.html -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Logical decoding restart problems

> On 31 Aug 2016, at 03:28, Craig Ringer <cr...@2ndquadrant.com> wrote: > > On 25 Aug. 2016 20:03, "Stas Kelvich" <s.kelv...@postgrespro.ru> wrote: > > > > Thanks for clarification about how restart_lsn is working. > > > > Digging slightly

Re: [HACKERS] Speedup twophase transactions

> On 21 Sep 2016, at 10:32, Michael Paquier <michael.paqu...@gmail.com> wrote: > > On Tue, Sep 20, 2016 at 11:13 PM, Stas Kelvich <s.kelv...@postgrespro.ru> > wrote: >> >> Putting that before actual WAL replay is just following historical order of >> e

Re: [HACKERS] Speedup twophase transactions

t is possible even without DSM, it possible to allocate static sized array storing some info about tx, whether it is in the WAL or in file, xid, gid. Some sort of PGXACT doppelganger only for replay purposes instead of using normal one. So taking into account my comments what do you think? Sh

Re: [HACKERS] Speedup twophase transactions

> On 07 Sep 2016, at 11:07, Stas Kelvich <s.kelv...@postgrespro.ru> wrote: > >> On 07 Sep 2016, at 03:09, Michael Paquier <michael.paqu...@gmail.com> wrote: >> >>>> On 06 Sep 2016, at 12:03, Michael Paquier <michael.paqu...@gmail.com> >>>

[HACKERS] assert violation in logical messages serialization

-- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

. Tried to generate Xcode project out of cmake, build fails on genbki.pl: can't locate Catalog.pm (which itself lives in src/backend/catalog/Catalog.pm) -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Stats sender and 2pc minor problem

of 0. Fix along with test is attached. 2pc-stats.patch Description: Binary data -- Stas Kelvich Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Speedup twophase transactions

> It must have been stripped by our email system. You were a direct CC so > you received it. > Then, probably, my mail client did something strange. I’ll check. -- Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers

Re: [HACKERS] logical decoding of two-phase transactions

d GID to any WAL records, nor to any in-memory structures. Other part of the story is how to find GID during decoding of commit prepared record. I did that by adding GID field to the commit WAL record, because by the time of decoding all memory structures that were holding xid<->gid cor

Re: [HACKERS] logical decoding of two-phase transactions

GID on all of the child nodes then we don't need to add the > GID. Yes, that’s also possible but seems to be less flexible restricting us to some specific GID format. Anyway, I can measure WAL space overhead introduced by the GID’s inside commit records to know exactly what will be th

[HACKERS] logical decoding of two-phase transactions

logical replication. [1] https://www.postgresql.org/message-id/EE7452CA-3C39-4A0E-97EC-17A414972884%40postgrespro.ru logical_twophase.diff Description: Binary data -- Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers

Re: [HACKERS] Speedup twophase transactions

filesystem cache applies here as well, but just without spending time on file creation. -- Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Speedup twophase transactions

m to reduce the overall noise between > two measurements though. Okay, i’ll perform such testing. -- Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Speedup twophase transactions

On 27 Sep 2016, at 03:30, Michael Paquier wrote:OK. I am marking this patch as returned with feedback then. Lookingforward to seeing the next investigations.. At least this review hastaught us one thing or two.So, here is brand new implementation of the same thing.Now

Re: [HACKERS] logical decoding of two-phase transactions

t useful for the main case when commit/abort is generated after receiver side will answer to prepares. Also that two-pass scan is a massive change in relcache.c and genam.c (FWIW there were no problems with cache, but some problems with index scan and handling one-to-many queries to catalog, e.g

Re: [HACKERS] GSOC'17 project introduction: Parallel COPY execution with errors handling

> On 23 Mar 2017, at 15:53, Craig Ringer wrote: > > On 23 March 2017 at 19:33, Alexey Kondratov > wrote: > >> (1) Add errors handling to COPY as a minimum program > > Huge +1 if you can do it in an efficient way. > > I think the main

Re: [HACKERS] logical decoding of two-phase transactions

ements logic i’ve just described. There is runtest.sh script that setups postgres, runs python logical consumer in background and starts regression test. Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company logical_twophase_v5.diff Description: Binary data

  1   2   >