Re: [HACKERS] GSoC 2017 : Patch for predicate locking in Gist index

On Mon, Oct 2, 2017 at 8:00 PM, Alexander Korotkov wrote: > What happen if exactly this "continue" fires? > >> if (GistFollowRight(stack->page)) >> { >> if (!xlocked) >> { >> LockBuffer(stack->buffer, GIST_UNLOCK); >> LockBuffer(stack->buffer, GIST_EXCLUSIVE); >>

Re: [HACKERS] GSoC 2017 : Patch for predicate locking in Gist index

Hi, Alexander! Thanks for looking into the patch! On Thu, Sep 28, 2017 at 3:59 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > > > In gistdoinsert() you do CheckForSerializableConflictIn() only if page > wasn't exclusively locked before (xlocked is false). > > if (!xlocked) >> { >>

[HACKERS] Re: [HACKERS] 答复: GiST API Adancement

2017-06-16 17:06 GMT+03:00 Tom Lane <t...@sss.pgh.pa.us>: > Yuan Dong <doff...@hotmail.com> writes: >> ·¢¼þÈË: Andrew Borodin <boro...@octonica.com> >>> I think there is one more aspect of development: backward >>> compatibility: it's

Re: [HACKERS] GiST API Adancement

Hi, Dong! 2017-06-15 21:19 GMT+05:00 Yuan Dong <doff...@hotmail.com>: > I'm going to hack on my own. With the help of Andrew Borodin, I want to > start the project with adding a third state to collision check. The third > state is that: > subtree is totally within the query.

Re: [HACKERS] GSoC 2017 weekly progress reports (week 2)

2017-06-13 18:00 GMT+05:00 Shubham Barai : > > Project: Explicitly support predicate locks in index AMs besides b-tree > Hi, Shubham Good job! So, in current HEAD test predicate_gist_2.spec generate false positives, but with your patch, it does not? I'd suggest keeping

Re: [HACKERS] Range Merge Join v1

2017-06-02 19:42 GMT+05:00 Jeff Davis <pg...@j-davis.com>: > On Tue, May 30, 2017 at 11:44 PM, Andrew Borodin <boro...@octonica.com> wrote: >> 1. Are there any types, which could benefit from Range Merge and are >> not covered by this patch? > > I thought about thi

Re: [HACKERS] GSoC 2017 : Proposal for predicate locking in gist index

Hi, hackers! 2017-06-01 1:50 GMT+05:00 Kevin Grittner : > > The main difference between b-tree and gist index while searching for a > > target tuple is that in gist index, we can determine if there is a match or > > not at any level of the index. > > Agreed. A gist scan can

Re: [HACKERS] Range Merge Join v1

Hi, Jeff! Sorry for being late. Actually, I had several unsuccessful attempts to find something wrong with the patch. Here's my review. in pathkey.c ecs = (EquivalenceClass **) palloc(nClauses * sizeof(EquivalenceClass *)); scores = (int *) palloc(nClauses * sizeof(int)); range_ecs =

Re: [HACKERS] Allow GiST opcalsses without compress\decompres functions

2017-05-29 0:00 GMT+05:00 Tom Lane : > But the opclass's consistent function might expect to be always given an > untoasted input, in which case you'd need the decompress function to fix > that up. In cube data is detoasted at least 4 times before going to

Re: [HACKERS] Allow GiST opcalsses without compress\decompres functions

28 мая 2017 г. 11:15 PM пользователь "Tom Lane" <t...@sss.pgh.pa.us> написал: Andrew Borodin <boro...@octonica.com> writes: > 2017-05-28 22:22 GMT+05:00 Tom Lane <t...@sss.pgh.pa.us>: >> 2. If compress/decompress are omitted, then we could support index-only &

Re: [HACKERS] Allow GiST opcalsses without compress\decompres functions

Thanks, Tom! 2017-05-28 22:22 GMT+05:00 Tom Lane <t...@sss.pgh.pa.us>: > Andrew Borodin <boro...@octonica.com> writes: >> Maybe we should make compress\decompress functions optional? > > 1. You'll need to adjust the documentation (gist.sgml) not just the code.

[HACKERS] Allow GiST opcalsses without compress\decompres functions

Hi, hackers! Currently, GiST stores each attribute in a compressed form. That is, each time attribute is written it's calling compress function, and when the attribute is accessed the decompress functions is called. Some types can't get any advantage out of this technique since the context of one

[HACKERS] Index Only Scan support for cube

Hi, hackers! Here's a small patch that implements fetch function necessary for Index Only Scans that use cube data type. I reuse function g_cube_decompress() instead of creating new function g_cube_fetch(). Essentially, they both have to detoast data. How do you think, is it better to create a

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

Hi, Kang and everyone in this thread. I'm planning to present the online course "Hacking PostgreSQL: data access methods in action and under the hood" on edX on June 1st. It's not announced yet, links will be available later. This course I'm describing information that was crucial for me to start

Re: [HACKERS] Merge join for GiST

Hi, hackers! I've adapted crossmatch join from pgSphere to cube for performance tests. I've placed spatial join code here https://github.com/Octonica/postgres/blob/spatialjoin/contrib/cube/spatialjoin.c and node code here

Re: [HACKERS] PG 10 release notes

Hi, Bruce! 2017-04-25 6:31 GMT+05:00 Bruce Momjian : > The only unusual thing is that this release has ~180 items while most > recent release have had ~220. The pattern I see that there are more > large features in this release than previous ones. I'm not sure, but, probably,

Re: [HACKERS] Range Merge Join v1

Hi, Jeff! I'm planning to provide the review for this patch in future. I've done some performance tesing (see attachment). All in all, nothing important, everything works fine. Tests were executed under current HEAD on Ubuntu 16 over MacBook Air. I observe that: 1. Patch brings performance

Re: [HACKERS] Merge join for GiST

2017-04-13 11:30 GMT+05:00 Jeff Davis : > I don't quite follow. I don't think any of these proposals uses btree, > right? Range merge join doesn't need any index, your proposal uses > gist, and PgSphere's crossmatch uses gist. Merge join will use presorted data, B-tree provides

Re: [HACKERS] Merge join for GiST

2017-04-13 7:01 GMT+05:00 Jeff Davis : > On Tue, Apr 11, 2017 at 8:35 AM, Alexander Korotkov > wrote: >> On Tue, Apr 11, 2017 at 5:46 PM, Jeff Davis wrote: >>> Do you have a sense of how this might compare with range merge join? >>

Re: [HACKERS] Merge join for GiST

2017-04-11 14:17 GMT+05:00 Alexander Korotkov : > FYI, I've implemented this algorithm for pgsphere. See following branch. > https://github.com/akorotkov/pgsphere/tree/experimental > It's implemented as crossmatch() function which takes as arguments names of > two

Re: [HACKERS] Merge join for GiST

2017-04-10 20:38 GMT+05:00 Robert Haas <robertmh...@gmail.com>: > On Mon, Apr 10, 2017 at 7:53 AM, Andrew Borodin <boro...@octonica.com> wrote: >> I think this idea is somewhat related to this patch [2], but as for >> now cannot describe how exactly GiST merge and Ran

[HACKERS] Merge join for GiST

Hello, hackers! ==Spatial joins== Scientific papers from the dawn of R-trees and multidimensional indexes feature a lot of algorithms for spatial joins. I.e. you have two sets of geometries s1 and s2, you need to produce all colliding pairs (p1,p2) where p1 in s1 and p2 in s2. For 2 R-trees of

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-03-22 22:48 GMT+05:00 Teodor Sigaev : > hasEmptyChild? and hasNonEmptyChild (BTW, isAnyNonempy has missed 't') Yes, I think this naming is good. It's clear what's in common in these flags and what's different. > And if the whole posting tree is empty,then we could mark

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

Hi, Teodor! 2017-03-21 20:32 GMT+05:00 Teodor Sigaev : > I had a look on patch That's great, thanks! > > /* > * All subtree is empty - just return TRUE to indicate that parent > must > * do a cleanup. Unless we are ROOT an there is way to go upper. >

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-03-16 23:55 GMT+05:00 Peter Geoghegan <p...@bowt.ie>: > On Thu, Mar 16, 2017 at 11:53 AM, Andrew Borodin <boro...@octonica.com> wrote: >> 2. Thus, L fully concurrent vacuum is possible, indeed, and >> furthermore Theodor suggested that I should implement not only

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-03-16 21:27 GMT+05:00 David Steele : > This patch applies cleanly and compiles at cccbdde. > > Jeff, any thoughts on Andrew's responses? Hi, David! I've got some updates on the matter of this patch, since the understanding of the B-tree bothered me much. Currently, I'm

Re: [HACKERS] [GSoC] Self introduction and question to mentors

Hi, Kirill! 2017-03-06 12:41 GMT+05:00 Кирилл Бороздин : > My name is Kirill Borozdin. I am a student of Ural Federal University from > Yekaterinburg, Russia. That's fine. I'm the associate professor at Ural Federal University. But not in your institution, though. > I

Re: [HACKERS] [GSoC] Personal presentation and request for clarification

Hi! It's great that you are interested in PostgreSQL! I'll answer the question on the matter of GSoC project proposed by me. I hope someone else will handle questions on your primary objective. 2017-03-03 4:15 GMT+05:00 João Miguel Afonso : > My second choice

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

Hi, Jeff! 2017-02-05 3:45 GMT+05:00 Jeff Davis <pg...@j-davis.com>: > On Sun, Jan 22, 2017 at 10:32 PM, Jeff Davis <pg...@j-davis.com> wrote: >> On Sat, Jan 21, 2017 at 4:25 AM, Andrew Borodin <boro...@octonica.com> wrote: >> One idea I had that might be s

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-01-25 12:09 GMT+05:00 Andrew Borodin <boro...@octonica.com>: > 2017-01-24 22:29 GMT+05:00 Jeff Davis <pg...@j-davis.com>: >> By the way, can you show me where the Lehman and Yao paper addresses >> page recycling? > > Here J. Hellerstein comments L

Re: [HACKERS] pg_background contrib module proposal

2017-01-27 19:14 GMT+05:00 Peter Eisentraut : > I suppose we should decide first whether we want pg_background as a > separate extension or rather pursue extending dblink as proposed elsewhere. > > I don't know if pg_background allows any use case that dblink

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-01-24 22:29 GMT+05:00 Jeff Davis <pg...@j-davis.com>: > On Tue, Jan 24, 2017 at 3:18 AM, Andrew Borodin <boro...@octonica.com> wrote: >> Technically, approach of locking a subtree is not novel. Lehman and >> Yao focused on "that any process for manipulati

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

Hello! I see your point on alternatives. I will do my best to generate more ideas on how vacuum can be done withing existing index structure, this will take a day or two. In this message I'll answer some questions. 2017-01-23 22:45 GMT+05:00 Jeff Davis : > 1. I haven't seen

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

Hi! 2017-01-23 11:32 GMT+05:00 Jeff Davis : > I have some reservations about the complexity and novelty of the > approach. I don't think I've seen an approach quite like this one > before. It seems like the main reason you are using this approach is > because the btree

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

Hello Jeff! >Review of the code itself: How do you think, is there anything else to improve in that patch or we can mark it as 'Ready for committer'? I've checked the concurrency algorithm with original work of Lehman and Yao on B-link-tree. For me everything looks fine (safe, deadlock free and

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

Hi, Jeff! Thanks for review! Here's a patch corrected according to your suggestions. 2017-01-19 11:48 GMT+05:00 Jeff Davis : > https://www.postgresql.org/message-id/CAJEAwVE4UAmm8fr%2BNW8XTnKV6M--ACoNhL3ES8yoKL2sKhbaiw%40mail.gmail.com > > Let me re-summarize what's been done

Re: [HACKERS] background sessions

2017-01-12 9:01 GMT+05:00 Peter Eisentraut : > On 1/10/17 10:58 AM, Robert Haas wrote: >> On Thu, Dec 29, 2016 at 5:18 PM, Peter Eisentraut >> wrote: >>> For additional entertainment, I include patches that integrate >>>

Re: [HACKERS] GSoC 2017

2017-01-10 14:53 GMT+05:00 Alexander Korotkov : > 1. What project ideas we have? I have one more project of interest which I can mentor. Topic. GiST API advancement ===Idea=== GiST API was designed at the beginning of 90th to reduce boilerplate code around data access

Re: [HACKERS] GSoC 2017

2017-01-10 14:53 GMT+05:00 Alexander Korotkov : > 1. What project ideas we have? Hi! I'd like to propose project on sorting algorithm research. I’m ready to be a mentor on this project. ===Topic=== Sorting algorithms benchmark and implementation. ===Idea=== Currently

Re: [HACKERS] pg_background contrib module proposal

Hi! 2017-01-07 11:44 GMT+05:00 amul sul : > Changes: > 1. pg_background_launch renamed to pg_background_start > 2. pg_background_detach renamed to pg_background_close > 3. Added new api to display previously launch background worker & its > result count waiting to be read.

Re: [HACKERS] background sessions

2017-01-04 10:23 GMT+05:00 amul sul : > One more query, can we modify > BackgroundSessionStart()/BackgroundSession struct to get background > worker PID as well? I think since session always has a PID it's absoultley reasonable to return PID. > I can understand this requirement

Re: [HACKERS] background sessions

2017-01-03 19:39 GMT+05:00 Peter Eisentraut : > On 1/3/17 1:26 AM, amul sul wrote: > > One more requirement for pg_background is session, command_qh, > > response_qh and worker_handle should be last longer than current > > memory context, for that we might need

Re: [HACKERS] pg_background contrib module proposal

2016-12-21 20:42 GMT+05:00 Robert Haas : > This whole subthread seems like a distraction to me. I find it hard > to believe that this test case would be stable enough to survive the > buildfarm where, don't forget, we have things like > CLOBBER_CACHE_ALWAYS machines where

Re: [HACKERS] pg_background contrib module proposal

2016-12-21 4:55 GMT+05:00 David Fetter : > I see. > > I find the following a little easier to follow, and the sleeps still > work even when very short. Now I know a little more SQL :) Thank you. I'm not sure every platform supports microsecond sleeps. But it will work anyway.

Re: [HACKERS] pg_background contrib module proposal

2016-12-19 4:21 GMT+05:00 David Fetter : > Couldn't it sleep in increments smaller than a second? Like maybe > milliseconds? Also, it's probably cleaner (or at least more > comprehensible) to write something using format() and dollar quoting > than the line with the double 's.

Re: [HACKERS] background sessions

2016-12-16 20:17 GMT+05:00 Peter Eisentraut : >> And one more thing... Can we have BackgroundSessionExecute() splitted >> into two parts: start query and wait for results? >> It would allow pg_background to reuse bgsession's code. > > Yes, I will look into that.

Re: [HACKERS] background sessions

2016-12-15 0:30 GMT+05:00 Peter Eisentraut : TryBeginSession()? >>> >>> What exactly would that do? >> Return status (success\failure) and session object, if a function succeeded. >> >> If there is max_connections exceeded, then (false,null). >> >> I'm not

Re: [HACKERS] background sessions

2016-12-14 20:45 GMT+05:00 Peter Eisentraut <peter.eisentr...@2ndquadrant.com>: > On 12/11/16 5:38 AM, Andrew Borodin wrote: >> 2. From my point of view the interface of the feature is the strong >> point here (compared to pg_background). But it does not help >> program

Re: [HACKERS] pg_background contrib module proposal

2016-12-13 12:55 GMT+05:00 amul sul : > I think background-session code is not that much deviated from > pg_background code, It is not derived, though it is not much deviated. background sessions code do not have detailed exceptions and code comments, but it is doing somewhat

Re: [HACKERS] pg_background contrib module proposal

Hi! Just in case you'd like to include sleepsort as a test, here it is wrapped as a regression test(see attachment). But it has serious downside: it runs no less than 5 seconds. Also I'll list here every parallelism feature I managed to imagine. It is not to say that I suggest having some of

Re: [HACKERS] background sessions

Hi! I'm planning to review this patch. I have some questions, maybe answers could help me understand patch better. 1. As far as I can see, we connot use COPY FROM STDIN in bg session? Since one of purposes is to orchestrate transactions, may be that would be valuable. 2. From my point of view the

Re: [HACKERS] pg_background contrib module proposal

2016-12-09 18:00 GMT+05:00 Robert Haas : > It looks like this could be reworked as a client of Peter Eisentraut's > background sessions code, which I think is also derived from > pg_background: > >

Re: [HACKERS] pg_background contrib module proposal

I've read the code and now I have more suggestions. 1. Easy one. The case of different natts of expected and acutal result throws same errmsg as the case of wrong types. I think we should assist the user more here if (natts != tupdesc->natts) ereport(ERROR,

Re: [HACKERS] pg_background contrib module proposal

This is simply wonderful! Finaly, I can implement my favorite sleepsort in postgres: create table input as select round(random()*20) x from generate_series(1,5,1); create table output(place int,value int); create sequence s start 1; create table handles as select pg_background_launch('select

Re: [HACKERS] GIN non-intrusive vacuum of posting tree

Here is v1 of the patch. Now it has changes for README and contains more comments clarifying changes of locking model. Also I will elaborate some more on what is patched. Main portion of changes is made to function ginVacuumPostingTreeLeaves(). Before the patch it was traversing tree in

[HACKERS] GIN non-intrusive vacuum of posting tree

Hi hackers! Here is a patch with more concurrency-friendly vacuum of GIN. ===Short problem statement=== Currently GIN vacuum during cleaning posting tree can lock this tree for a long time without real need. ===Problem statement=== Vacuum of posting tree (function ginVacuumPostingTree() in

Re: [HACKERS] Fractal tree indexing

Hi hackers! Here is prototype of procrastinating GiST. Or lazy GiST. Or buffered GiST. Indeed, there is nothing fractal about it. The concept is leaf tuples stall on internal pages. From time to time they are pushed down in batches. This code is far from perfect, I've coded this only for the

Re: [HACKERS] GiST penalty functions [PoC]

> This thread has basically died, so marking as returned with feedback. Well, not exactly died, but it's kind of in blind lead. I'll summarize a bit for future references: 1. cube extension for indexing lowered dimensionality data (2d in 3d) is broken. Queries effectively will degrade to full

Re: [HACKERS] GiST penalty functions [PoC]

Hi Heikki! > Got a link for a description of the RR*-tree? Would be good to reference it > in the patch comments, too. Well, as for now, the best way to reach the paper is DOI 10.1145/1559845.1559929 http://sci-hub.cc/ Authors in conversations clearly stated that they endorse (not sure in correct

Re: [HACKERS] GiST: interpretation of NaN from penalty function

> Certainly, "NaN means infinity", as your patch proposes, has no more basis to > it than "NaN means zero". You are absolutley right. Now I see that best interpretation is "NaN means NaN". Seems like we need only drop a check. Nodes with NaN penalties will be considered even worser than those

[HACKERS] GiST: interpretation of NaN from penalty function

Hi hackers! Currently GiST treats NaN penalty as zero penalty, in terms of generalized tree this means "perfect fit". I think that this situation should be considered "worst fit" instead. Here is a patch to highlight place in the code. I could not construct test to generate bad tree, which would

Re: [HACKERS] GiST penalty functions [PoC]

Here is v6 of cube penalty patch. There was a warning about unused edge function under systems without __STDC_IEC_559__ defined, this patch fixes it. Regards, Andrey Borodin. diff --git a/contrib/cube/cube.c b/contrib/cube/cube.c index 3feddef..ad868ac 100644 --- a/contrib/cube/cube.c +++

Re: [HACKERS] GiST penalty functions [PoC]

Hi hackers! Here is gist cube penaly patch v5. Changes: 1. union version of pack_float() is used, without warings and with minimum asm commands emitted 2. test for __STDC_IEC_559__ is added, this is a standard way of C99 to determine float compliance with IEEE754. ANSI-only compiler+libs, along

Re: [HACKERS] GiST penalty functions [PoC]

>Personally I wouldn't be very happy about an IEEE754 assumption. Ok, so let's avoid autoconf man. There is no real explanations of the ground for this assumption, just a reference to paper of David Goldberg (and there is no metion about IEEE754 is absoulte everywhere). BTW, may be we can ask

Re: [HACKERS] GiST penalty functions [PoC]

>BTW, would someone explain to me why using a float here will not fail >catastrophically for inputs outside the range of float? Indeed, it will. And that's one of the reason I'm considering advancing GiST API instead of advancing extensions. It won't crash, but will produce terrible index, not

Re: [HACKERS] GiST penalty functions [PoC]

>autoconf check for IEEE 754 floats Autoconf man says folowing: >it is safe to assume IEEE-754 in most portable code these days https://www.gnu.org/software/autoconf/manual/autoconf.html#Floating-Point-Portability > A union might be more readable Here is union version of the patch. It's slower

Re: [HACKERS] GiST penalty functions [PoC]

Well, arithmetics is too fragile. This version of float packing with arithmetical packaging static float pack_float(float actualValue, int realm) { double max,min; max = FLT_MAX / ( 8 >> realm ); min = FLT_MAX / ( 16 >> realm ); if( realm == 0 ) min = 0; /* squeeze the actual value between min

Re: [HACKERS] GiST penalty functions [PoC]

Oh, sorry, made one more attemp and now I see your algorithm differently. So you propose to use oE and oV as a markers of borders for what I call Realm. But there may be very little significant bits in one of this ranges. pg_sphere and PostGiS extensions tried to use 1 as a marker, with alike

Re: [HACKERS] GiST penalty functions [PoC]

Hi Heikki! Thank you for reviewing the code, it's always inspiring when a work is noted (: >Looking at the code, by "margin", you mean the sum of all "edges", i.e. of each dimension, of the cube. Indeed. As far as I remember, this is a terminology of old R*-tree paper. Now they use the word

Re: [HACKERS] GiST penalty functions [PoC]

Here is new version of the patch. Now function pack_float is commented better. All inline keywords are removed. I haven't found any performance loss for that. Remove of static's showed 1%-7% performance loss in SELECT computation (3 test runs), so I left statics where they are. Actually, to avoid

Re: [HACKERS] GiST penalty functions [PoC]

Thank you for your coments, Tom. > Modern compilers are generally able to make their own decisions about it, and > trying to put your thumb on the scales this heavily is not likely to improve > the code. Well, I have tested that combination of "static inline" affets performance of index build

Re: [HACKERS] GiST penalty functions [PoC]

Hi hackers! Here is the new patch version. With the help of Mikhail Bakhterev I've optimized subroutines of the penalty function. Index build time now is roughly equivalent to build time before patch (test attached to thread start). Time of SELECT statement execution is reduced by 40%. Changes in

[HACKERS] GiST penalty functions [PoC]

Hi, hackers! Some time ago I put a patch to commitfest that optimizes slightly GiST inserts and updates. It's effect in general is quite modest (15% perf improved), but for sorted data inserts are 3 times quicker. This effect I attribute to mitigation of deficiencies of old split algorithm.

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

tonica & Ural Federal University. 2016-08-16 20:23 GMT+05:00 Anastasia Lubennikova <a.lubennik...@postgrespro.ru>: > 09.08.2016 19:45, Andrew Borodin: >> >> Here is new version of the patch, now it includes recommendations from >> Anastasia Lubennikova. >> >> &

Re: [HACKERS] WIP: Covering + unique indexes.

> That was a bug caused by high key truncation, that occurs when index has more > than 3 levels. Fixed. Affirmative. I've tested index construction with 100M rows and subsequent execution of select queries using index, works fine. Best regards, Andrey Borodin, Octonica & Ural Federal University.

Re: [HACKERS] Btree Index on PostgreSQL and Wiredtiger (MongoDB3.2)

>So on average in a large randomly filled index, pages spend more time nearer 50% full than 100% full. I think we can make this number more...controllable. Before split we can check whether left and right pages both are in shared buffer and if they are seriously under certain fillfactor, say

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

Here is new version of the patch, now it includes recommendations from Anastasia Lubennikova. I've investigated anamalous index size decrease. Most probable version appeared to be true. Cube extension, as some others, use Guttman's polynomial time split algorithm. It is known to generate

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

Anastasia , thank you for your attentive code examine. 2016-08-05 21:19 GMT+05:00 Anastasia Lubennikova : > First of all, shouldn't we use MAXALIGN(oldsize) instead of oldsize? > Although, I'm quite sure that it was already aligned somewhere before. > I doubt that

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

Here is BRIN-compatible version of patch. Now function PageIndexTupleOverwrite consumes tuple size as a parameter, this behavior is coherent with PageAddItem. Also, i had to remove asserstion that item has a storage in the loop that adjusts line pointers. It would be great if someone could check

Re: [HACKERS] Optimizing numeric SUM() aggregate

I've tested patch with this query https://gist.github.com/x4m/fee16ed1a55217528f036983d88771b4 Test specs were: Ubuntu 14 LTS VM, dynamic RAM, hypervisor Windows Server 2016, SSD disk, core i5-2500. Configuration: out of the box master make. On 10 test runs timing of select statement: AVG 3739.8

Re: [HACKERS] Optimizing numeric SUM() aggregate

hat value 1 is hardcoded somewhere else, any other value is not recommended and a bunch of code is left for historical reasons. Best regards, Andrey Borodin, Octonica & Ural Federal University. 2016-07-27 13:57 GMT+05:00 Dean Rasheed <dean.a.rash...@gmail.com>: > On 27 July 2016 at

Re: [HACKERS] Optimizing numeric SUM() aggregate

>I think we could do carry every 0x7FF / 1 accumulation, couldn't we? I feel that I have to elaborate a bit. Probably my calculations are wrong. Lets assume we already have accumulated INT_MAX of digits in previous-place accumulator. That's almost overflow, but that's not overflow.

Re: [HACKERS] Optimizing numeric SUM() aggregate

Hi! I like the idea and implementation, but I have one suggestion. > Instead of propagating carry after each new value, it's done only every > values (or at the end). I think we could do carry every 0x7FF / 1 accumulation, couldn't we? Best regards, Andrey Borodin, Octonica & Ural

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

>Can you please patch BRIN to use this new function too? On my machine replacement of both BRIN update cases (see https://github.com/x4m/pggistopt/commit/a6d301ff79104b977619339d53aebf748045418a ) showed no performance changes on folowing update query (6 seconds of updates avg): create table

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

>Can you please patch BRIN to use this new function too? AFAIK Sergey Mirvoda was going to implement and test it. It is easy to do this optimization for brin_samepage_update (see patch draft in attachment, make check passes), but WAL of regular BRIN update is not clear for me. Best regards,

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

>If a feature changes the shape of WAL records, XLOG_PAGE_MAGIC is bumped to >prevent any problems. I've attached patch with a bump, but, obviously, it'll be irrelevant after any other commit changing WAL shape. Here is a patch with updated GiST README. I haven't found apropriate place to

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

One more thing came to my mind: WAL records done by the GiST before patch will corrupt GiST after patch if replayed. Is it a problem? May be it's prevented on some higher level? Best regards, Andrey Borodin, Octonica & Ural Federal University. 2016-07-06 22:11 GMT+05:00 Andrew Borodin &

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

ign size instead. > > Also I suspect that PageIndexTupleOverwrite() should make a call to > ItemIdSetNormal() to pretend it is generic function and not just a > private part of GiST. > > Best regards, Andrey Borodin, Octonica & Ural Federal University. > > 2016-07

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

() should make a call to ItemIdSetNormal() to pretend it is generic function and not just a private part of GiST. Best regards, Andrey Borodin, Octonica & Ural Federal University. 2016-07-04 22:59 GMT+05:00 Andrew Borodin <boro...@octonica.com>: > Thank you, Alvaro. > > Yes, now

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

t;alvhe...@2ndquadrant.com>: > Andrew Borodin wrote: >> Thank you, Amit. >> >> Currently, if WAL reconstruct page in another order it won't be a problem. > > We require that replay of WAL leads to identical bytes than the > original. Among other things, this enable

Re: [HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

spaces later. I also found some more missing spaces in PageIndexTupleOverwrite call and typo in comments ("on-place"). Best regards, Andrey Borodin, Octonica & Ural Federal University. 2016-07-04 18:58 GMT+05:00 Amit Kapila <amit.kapil...@gmail.com>: > On Mon, Jul 4, 2016 at

[HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

Here is a patch with corrections from Alexander Korotkov. My tests, check and check-world show no problems against Ubuntu LTS 14 x64 VM. Best regards, Andrey Borodin, Octonica & Ural Federal University. 2016-07-04 10:41 GMT+05:00 Andrew Borodin <boro...@octonica.com>: > Hi! >>

[HACKERS] Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

checks copied from PageIndexTupleDelete). Next weekend I'll do more testing, and then add it to commitfest. Best regards, Andrey Borodin, Octonica & Ural Federal University. 2016-07-03 15:21 GMT+05:00 Alexander Korotkov <a.korot...@postgrespro.ru>: > Hi! > > On Sun, Jul 3,

[HACKERS] GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]

Hi, hackers! I think there is some room for improving GiST inserts. Following is the description of what I think is performance problem. Function gistplacetopage in file /src/backend/access/gist/gist.c is responsible for updating or appending new tuple on GiST page. Currently, after checking

[HACKERS] [Proposal] Improvement of GiST page layout

Hi, hackers! I want to propose improvement of GiST page layout. GiST is optimized to reduce disk operations on search queries, for example, windows search queries in case of R-tree. I expect that more complicated page layout will help to tradeoff some of CPU efficiency for disk efficiency.