Re: PANIC during crash recovery of a recently promoted standby

2018-05-14 Thread Pavan Deolasee
On Fri, May 11, 2018 at 8:39 PM, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Michael Paquier wrote: > > On Thu, May 10, 2018 at 10:52:12AM +0530, Pavan Deolasee wrote: > > > I propose that we should always clear the minRecoveryPoint after > promotion > >

PANIC during crash recovery of a recently promoted standby

2018-05-09 Thread Pavan Deolasee
int after promotion to ensure that crash recovery always run to the end if a just-promoted standby crashes before completing its first regular checkpoint. A WIP patch is attached. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Trai

Faster inserts with mostly-monotonically increasing values

2017-12-30 Thread Pavan Deolasee
comes higher. Patch credit: this work is based on Simon Riggs's original ideas and research. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services pg_btree_target_block_v1.patch Description: Binary data

Re: Faster inserts with mostly-monotonically increasing values

2018-01-04 Thread Pavan Deolasee
On Thu, Jan 4, 2018 at 6:05 PM, Alvaro Herrera <alvhe...@alvh.no-ip.org> wrote: > Pavan Deolasee wrote: > > On Tue, Jan 2, 2018 at 7:15 PM, Tels <nospam-pg-ab...@bloodgate.com> > wrote: > > > > > Just a question trying to understand how btree indexe

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-01-31 Thread Pavan Deolasee
it sees. There were some concerns about bit-flipping, which may inadvertently SegID stored in the carried over WAL record so that it now matches with the current WAL files' SegID, but TBH I don't see why we can't trust CRC to detect that. Because if we can't, then there would be othe

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-07 Thread Pavan Deolasee
or v11. > TBH I did not consider partitioning any less complex and it was indeed very complex, requiring at least 3 reworks by me. And from what I understood, it would have been a blocker too. So is subquery handling and RLS. That's why I focused on addressing those items while you and Simon were

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-02-12 Thread Pavan Deolasee
e file system crash can survive many renames and still resurrect old data several renames before, then we shall have the same problem. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-12 Thread Pavan Deolasee
ate WHEN AND conditions separately, outside the EPQ. The problem arises when the join qual returns a different result with the updated tuple. I listed down those cases in my earlier email in the day. To me (and I assume to Peter and Simon too), those are the more interesting cases. Thanks, Pava

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-11 Thread Pavan Deolasee
On Fri, Feb 9, 2018 at 6:53 AM, Peter Geoghegan <p...@bowt.ie> wrote: > On Wed, Feb 7, 2018 at 7:51 PM, Pavan Deolasee <pavan.deola...@gmail.com> > wrote: > > I understand getting EPQ semantics right is very important. Can you > please > > (once again) summaris

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-13 Thread Pavan Deolasee
INSERT. Similarly, INSERT policies will be applied when MERGE attempts to INSERT a row into the table and error will be thrown if the row does not satisfy INSERT policies. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-05 Thread Pavan Deolasee
nQual might need some treatment because when the plan is re-executed, it will expect to the find the updated tuple in the slot of the underlying query's RTE and not in the resultRelation's RTE, which does not participate in the join at all. Anything else I could be missing out completely? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-15 Thread Pavan Deolasee
are other issues detected by sqlsmith or otherwise. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Faster inserts with mostly-monotonically increasing values

2018-01-02 Thread Pavan Deolasee
On Sun, Dec 31, 2017 at 4:36 PM, Peter Geoghegan <p...@bowt.ie> wrote: > On Sun, Dec 31, 2017 at 6:44 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > Here is a patch that implements the idea. If the last insert happens to > be > > in the rightmost

Re: Faster inserts with mostly-monotonically increasing values

2018-01-04 Thread Pavan Deolasee
ight page will get the remaining. The next split will happen when the right page is full i.e. when another N/2 entries are added. Thus there will be a split at every N/2 inserts, creating an index with half filled pages. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: PATCH: backtraces for error messages

2018-06-21 Thread Pavan Deolasee
bly written and has a bad UI, but those can be improved if there is interest. Given the lack of response, I suspect there is enough interest in the feature though. Thanks, Pavan [1] https://www.postgresql.org/message-id/CABOikdMvx_Kr_b4ELhJEoeGcLTZKrDma%2BfPZpoZVdpL7Zc0bVw%40mail.gmail.com -- Pavan Deolasee

Re: PATCH: backtraces for error messages

2018-06-21 Thread Pavan Deolasee
h more likely to get nowhere. > > Sorry, I did not mean to mix up two patches. I brought it up just in case it provides another idea about when and how to log the backtrace. So yeah, let's discuss that patch separately. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: PANIC during crash recovery of a recently promoted standby

2018-06-21 Thread Pavan Deolasee
+ */ s/that/this Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Accidental removal of a file causing various problems

2018-08-24 Thread Pavan Deolasee
ide effects are well understood and known? IMHO even if we accept that we can't do much about a missing file, it seems quite odd that both 1 and 3 happens. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: PANIC during crash recovery of a recently promoted standby

2018-07-05 Thread Pavan Deolasee
e complete fix, verifying all the cases, in various back branches. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Locking considerations of REINDEX

2018-07-04 Thread Pavan Deolasee
y we would have ability so that the existing queries can continue to read from the old physical index, new queries will shift to the new index and eventually the old index's storage will be dropped when nobody can see it. Thanks, Pavan [1] https://www.postgresql.org/docs/11/static/sql-reindex.html

Re: [HACKERS] Restrict concurrent update/delete with UPDATE of partition key

2018-03-07 Thread Pavan Deolasee
ts a thought. May be it was discussed somewhere else and ruled out. I happened to notice this patch because of the bug I encountered. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Restrict concurrent update/delete with UPDATE of partition key

2018-03-07 Thread Pavan Deolasee
On Thu, Mar 8, 2018 at 12:31 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Thu, Mar 8, 2018 at 11:04 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > > > On Tue, Feb 13, 2018 at 12:41 PM, amul sul <sula...@gmail.com> wrote: > >> &g

Re: [HACKERS] Restrict concurrent update/delete with UPDATE of partition key

2018-03-07 Thread Pavan Deolasee
for partition key updates are either missing (I haven't checked) or they need more work. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-12 Thread Pavan Deolasee
, but that would most likely mean complete re-writing of the UPDATEs/DELETEs for partition/inheritance tables. The challenges would just be the same in both cases. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-27 Thread Pavan Deolasee
ks to me irrespective of what we choose, our implementation would be much superior to what Oracle offers. BTW I've sent v17a of the patch, which is very close to being complete from my perspective (except some documentation fixes/improvements). The only thing pending is the decision to accept or change the currently implemented concurrency semantics. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-01 Thread Pavan Deolasee
y doubtful that we want to go down that path and whether it's even feasible. Our regular UPDATE .. FROM does not do that either. Given that, it seems better to just throw an error (even when no NOT MATCHED action exists) and explain to the users that MERGE will work as long as concurrent updates do

Re: Faster inserts with mostly-monotonically increasing values

2018-03-13 Thread Pavan Deolasee
On Sun, Mar 11, 2018 at 9:18 PM, Claudio Freire <klaussfre...@gmail.com> wrote: > On Sun, Mar 11, 2018 at 2:27 AM, Pavan Deolasee > > > > > Yes, I will try that next - it seems like a good idea. So the idea would > be: > > check if the block is still the righ

Re: Faster inserts with mostly-monotonically increasing values

2018-03-13 Thread Pavan Deolasee
ry ad-hoc script.. But here it is.. It assumes a presence of a branch named "btree_rightmost" with the patched code. You will need to make necessary adjustments of course. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, T

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-15 Thread Pavan Deolasee
Hi Stephen, On Fri, Mar 16, 2018 at 7:28 AM, Stephen Frost <sfr...@snowman.net> wrote: > Greetings Pavan, all, > > * Pavan Deolasee (pavan.deola...@gmail.com) wrote: > > On 9 March 2018 at 08:29, Peter Geoghegan <p...@bowt.ie> wrote: > > > My #1 concern ha

Re: ON CONFLICT DO UPDATE for partitioned tables

2018-03-16 Thread Pavan Deolasee
s there some rule that we follow? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Restrict concurrent update/delete with UPDATE of partition key

2018-03-08 Thread Pavan Deolasee
ther ideas too. And even if we can't find one, my vote would be to settle for #1 instead of trying to do #2. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-10 Thread Pavan Deolasee
side of an outer join. That's probably a big complication for > using one target RTE. Your approach to implementing partitioning [1] > seems to benefit from having two different RTEs, in a way -- you > sidestep the restriction. Right. The entire purpose of having two different RTEs is t

Re: Faster inserts with mostly-monotonically increasing values

2018-03-10 Thread Pavan Deolasee
On Sat, Mar 10, 2018 at 12:11 AM, Claudio Freire <klaussfre...@gmail.com> wrote: > On Fri, Mar 9, 2018 at 2:54 PM, Pavan Deolasee <pavan.deola...@gmail.com> > wrote: > > > > > > > > > So yes, the benefits of the patch go down with higher number of c

Re: Faster inserts with mostly-monotonically increasing values

2018-03-09 Thread Pavan Deolasee
On Tue, Mar 6, 2018 at 10:10 AM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > On Tue, Mar 6, 2018 at 7:29 AM, Peter Geoghegan <p...@bowt.ie> wrote: > >> On Mon, Mar 5, 2018 at 5:48 PM, Claudio Freire <klaussfre...@gmail.com> >> wrote: &

Re: Faster inserts with mostly-monotonically increasing values

2018-03-14 Thread Pavan Deolasee
On Wed, Mar 14, 2018 at 4:53 PM, Simon Riggs <simon.ri...@2ndquadrant.com> wrote: > On 14 March 2018 at 04:36, Pavan Deolasee <pavan.deola...@gmail.com> > wrote: > > I wonder if the shortened code path actually leads to > > heavier contention for EXCLUSIVE loc

Re: Faster inserts with mostly-monotonically increasing values

2018-03-14 Thread Pavan Deolasee
On Wed, Mar 14, 2018 at 7:21 PM, Simon Riggs <simon.ri...@2ndquadrant.com> wrote: > On 14 March 2018 at 13:33, Pavan Deolasee <pavan.deola...@gmail.com> > wrote: > > > > > > On Wed, Mar 14, 2018 at 4:53 PM, Simon Riggs < > simon.ri...@2ndquadrant.com>

Re: Missing parse_merge.h?

2018-04-03 Thread Pavan Deolasee
oks like Simon forgot to add new files in that commit. I am trying to get in touch with him so that he can add the missing files and correct the situation. Sorry for the trouble. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Pavan Deolasee
rt/Update etc to a new file as I suggested, but still use the ModifyTable to run Merge. There are many things common between them. ModifyTable executes all DMLs and MERGE is just another DML which can run all three. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.

Re: pgsql: New files for MERGE

2018-04-06 Thread Pavan Deolasee
He said this on 29th March: On Thu, Mar 29, 2018 at 3:20 PM, Simon Riggs <si...@2ndquadrant.com> wrote: > On 28 March 2018 at 12:00, Pavan Deolasee <pavan.deola...@gmail.com> > wrote: > > > v27 attached, though review changes are in > > the add-on 0005 patch.

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-06 Thread Pavan Deolasee
On Fri, Apr 6, 2018 at 1:30 AM, Andres Freund <and...@anarazel.de> wrote: > Hi, > > On 2018-04-05 11:31:48 +0530, Pavan Deolasee wrote: > > > +/*- > > > > > > + * > > > +

Bugs in TOAST handling, OID assignment and redo recovery

2018-04-10 Thread Pavan Deolasee
st upto 9.1 and the bugs exist there too. In fact, these bugs probably existed forever, though I did not check very old releases. Attached is a simple reproducer and a proposed fix to address both the bugs. We should consider backpatching it all supported releases. Thanks, Pavan -- Pavan Deolase

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-10 Thread Pavan Deolasee
nline checkpoint record's > nextOid value seem fine to me. > Ok. Thanks for checking. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Faster inserts with mostly-monotonically increasing values

2018-04-10 Thread Pavan Deolasee
well. It slipped the review. > Yes, looks like an oversight :-( I will fix it along with the other changes that Peter requested. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Add support for printing/reading MergeAction nodes

2018-04-05 Thread Pavan Deolasee
se flags. No problem there too. This now also enforces single VALUES clause in the grammar itself instead of doing that check at parse-analyse time. So that's a net improvement too. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services 0001-Separate-raw-parse-representation-from-rest.patch Description: Binary data

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Pavan Deolasee
nk the code will compile that way. We should either rewrite that assertion or put it inside a #ifdef ASSERT_CHECKING block or simple remove that assertion because we already check for relkind in parse_merge.c. Will check. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Restrict concurrent update/delete with UPDATE of partition key

2018-04-05 Thread Pavan Deolasee
ON CONFLICT DO UPDATE to move a row to a different partition, but otherwise it works now. See 555ee77a9668e3f1b03307055b5027e13bf1a715. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: pgsql: New files for MERGE

2018-04-04 Thread Pavan Deolasee
On Thu, Apr 5, 2018 at 12:16 AM, Andres Freund <and...@anarazel.de> wrote: > Hi, > > On 2018-04-05 00:02:06 +0530, Pavan Deolasee wrote: > > Apologies from my end. Simon checked with me regarding your referenced > > email. I was in the middle of responding to it (wi

Re: pgsql: New files for MERGE

2018-04-04 Thread Pavan Deolasee
ch to take care of your review comments), but got side tracked by some high priority customer escalation. I shall respond soon. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-11 Thread Pavan Deolasee
On Tue, Apr 10, 2018 at 7:24 PM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > Hi Heikki, > > On Tue, Apr 10, 2018 at 7:07 PM, Heikki Linnakangas <hlinn...@iki.fi> > wrote: > >> >>> >> It would seem more straightforward to add a s

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-11 Thread Pavan Deolasee
On Wed, Apr 11, 2018 at 8:20 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > Pavan Deolasee <pavan.deola...@gmail.com> writes: > > Or may be we simply err on the side of caution and scan the toast table > > with SnapshotAny while looking for a duplicate? That might prevent u

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-12 Thread Pavan Deolasee
future tests for catching such issues. This was a very specialised test case written after getting a full grasp on the bug. And it just tests the thing that I knew is broken based on code reading. Also, with OID duplicate issue fixed, hitting more bugs in this area is going to be ev

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-12 Thread Pavan Deolasee
ked at the code, I couldn't find a case that is broken. I even tried a few test cases with DDLs etc. But I think what you did is fine and more bullet proof. So +1 to that. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2018-04-18 Thread Pavan Deolasee
ay be we use some heuristic based on available free space in the table prior to the truncation point? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: VM map freeze corruption

2018-04-18 Thread Pavan Deolasee
"select pg_check_frozen('t');" | $p # See if a vacuum freeze scanning all pages corrects the problem echo "vacuum (verbose, freeze, disable_page_skipping) t;" | $p echo "select pg_check_frozen('t');" | $p Thanks, Pavan [1] https://www.postgresql.org/m

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2018-04-18 Thread Pavan Deolasee
K blocks, remember those K buffers, discard those K buffers, truncate the relation and then try for next K blocks. If another backend requests lock on the table, we give up or retry after a while. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Developm

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-18 Thread Pavan Deolasee
On Thu, Apr 12, 2018 at 5:53 AM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 4/10/18 06:29, Pavan Deolasee wrote: > > One of our 2ndQuadrant support customers recently reported a sudden rush > > of TOAST errors post a crash recovery, nearly causing an ou

Re: Toast issues with OldestXmin going backwards

2018-04-21 Thread Pavan Deolasee
- > Andrew (irc:RhodiumToad) > > -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: VM map freeze corruption

2018-04-19 Thread Pavan Deolasee
hem now. I come up with attached. Not fully polished (and there is a XXX that I left for comments), but hopefully enough to tell what I am thinking. Do you think it's any improvement or actually makes the problem worse? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.c

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Pavan Deolasee
On Tue, Mar 27, 2018 at 1:54 PM, Simon Riggs <si...@2ndquadrant.com> wrote: > On 26 March 2018 at 17:06, Simon Riggs <si...@2ndquadrant.com> wrote: > > On 26 March 2018 at 15:39, Pavan Deolasee <pavan.deola...@gmail.com> > wrote: > > > > > That's all

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Pavan Deolasee
On Wed, Mar 28, 2018 at 8:28 AM, Peter Geoghegan <p...@bowt.ie> wrote: > On Tue, Mar 27, 2018 at 2:28 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > (Version 26) > > I have some feedback on this version: > > * ExecMergeMatched() needs to determin

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-27 Thread Pavan Deolasee
any more WAL than what we do today. The idea is to just use a different mechanism to find the prior checkpoint. But we should surely find the latest prior checkpoint. In the rare scenario that Tom showed, we should just throw an error and fix the patch if it's not doing that already. Thanks, Pava

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-27 Thread Pavan Deolasee
So in the worst case, we might read a WAL segment extra before we find the checkpoint record. That's not ideal but not too bad given that only pg_rewind needs this and that too only once. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] A design for amcheck heapam verification

2018-03-27 Thread Pavan Deolasee
thus holding OldestXmin back. Is there anything we can do to lessen that burden like telling other backends to ignore our xmin while computing OldestXmin (like vacuum does)? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] A design for amcheck heapam verification

2018-03-28 Thread Pavan Deolasee
On Wed, Mar 28, 2018 at 2:48 AM, Peter Geoghegan <p...@bowt.ie> wrote: > On Tue, Mar 27, 2018 at 6:48 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > + * When index-to-heap verification is requested, a Bloom filter is used > to > > + * fingerpri

Re: [HACKERS] A design for amcheck heapam verification

2018-03-28 Thread Pavan Deolasee
s not necessary since IndexBuildHeapScan() does that internally. I stumbled upon that while looking for any potential leaks. I know at least one other caller of IndexBuildHeapScan() doesn't bother to say anything either, but it's helpful. FWIW I also looked at the 0001 patch and it looks fine to me. Tha

Re: PATCH: Exclude unlogged tables from base backups

2018-03-26 Thread Pavan Deolasee
On Mon, Mar 26, 2018 at 1:03 PM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > On Fri, Mar 23, 2018 at 9:51 PM, David Steele <da...@pgmasters.net> wrote: > >> On 3/23/18 12:14 PM, Teodor Sigaev wrote: >> > >> > Thank you, pushed >

Re: Faster inserts with mostly-monotonically increasing values

2018-03-26 Thread Pavan Deolasee
On Sun, Mar 25, 2018 at 6:00 AM, Andrew Dunstan < andrew.duns...@2ndquadrant.com> wrote: > On Fri, Mar 23, 2018 at 8:27 PM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > >> > >> > >> I would probably just have a few regression lines that

Re: PATCH: Exclude unlogged tables from base backups

2018-03-26 Thread Pavan Deolasee
e check" and thus fails the test. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-29 Thread Pavan Deolasee
nce send that row for NOT MATCHED processing, thus inserting it into the table again. I am not saying there is no scope for improvement. But we need to be careful about what can be pushed down to the join and what must be applied after the join. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: PATCH: Exclude unlogged tables from base backups

2018-03-26 Thread Pavan Deolasee
On Mon, Mar 26, 2018 at 5:16 PM, Masahiko Sawada <sawada.m...@gmail.com> wrote: > On Mon, Mar 26, 2018 at 4:52 PM, Pavan Deolasee > > >> > > > > This one-liner patch fixes it for me. > > > > Isn't this issue already fixed by commit > d0c0c894533f9

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-02 Thread Pavan Deolasee
erns about the two RTE stuff which was necessitated when we added support for partitioned table. We discussed that at some length, with your inputs and agreed that it's not necessarily a bad thing and probably the only way to deal with partitioned tables. Personally, I don't see why an inte

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-18 Thread Pavan Deolasee
On Mon, Mar 12, 2018 at 5:43 PM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > On Sun, Mar 11, 2018 at 11:18 AM, Peter Geoghegan <p...@bowt.ie> wrote: > >> >> >> As you know, there is an ON CONFLICT DO UPDATE + partitioning patch in >>

Re: Faster inserts with mostly-monotonically increasing values

2018-03-19 Thread Pavan Deolasee
-25% with smaller number of client and 10-15% with larger number of clients. I haven't been able to establish why it's happening, but since it's a different AWS instance (though of the same type), I am inclined to blame it on that for now. Thanks, Pavan -- Pavan Deolasee ht

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-22 Thread Pavan Deolasee
count(*) FROM pg_class), (SELECT relname FROM pg_class LIMIT 1)); MERGE 1 postgres=# SELECT * FROM target; a | b -+ 755 | pgbench_source (1 row) Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: ON CONFLICT DO UPDATE for partitioned tables

2018-03-22 Thread Pavan Deolasee
Info as I wrote above. > > Thanks. It's looking much better now. I think we can possibly move all ON CONFLICT related members to a separate structure and just copy the pointer to the structure if (map == NULL). That might make the code a bit more tidy. Is there anything that needs to be do

Re: ON CONFLICT DO UPDATE for partitioned tables

2018-03-21 Thread Pavan Deolasee
indexes and the tuple desc to ResultRelInfo, as suggested above, I think we can avoid making any API changes to ExecPrepareTupleRouting and ExecInsert. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-23 Thread Pavan Deolasee
= #clients #tps 1 24.128004 2 12.326135 4 8.334143 8 16.035699 16 8.502794 So that's pretty good improvement across the spectrum. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services pg_reduce_wal_contention_v2.patch Description: Binary data

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-22 Thread Pavan Deolasee
On Fri, Mar 23, 2018 at 10:00 AM, Amit Langote < langote_amit...@lab.ntt.co.jp> wrote: > On 2018/03/23 3:42, Pavan Deolasee wrote: > > A slightly improved version attached. Apart from doc cleanup based on > > earlier feedback, fixed one assertion failure bas

Re: Faster inserts with mostly-monotonically increasing values

2018-03-23 Thread Pavan Deolasee
regression lines that should be sure > to exercise the code path and leave it at that. > > I changed the regression tests to include a few more scenarios, basically using multi-column indexes in different ways and they querying rows by ordering rows in different ways. I did

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-23 Thread Pavan Deolasee
s because it increases my overall > confidence in the design. If it was genuinely hard to add WITH clause > support, then that would probably tell us something about the overall > design that likely creates problems elsewhere. It's easy to say that > it isn't worth holding

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-23 Thread Pavan Deolasee
On Fri, Mar 23, 2018 at 4:43 AM, Peter Geoghegan <p...@bowt.ie> wrote: > On Thu, Mar 22, 2018 at 11:42 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > A slightly improved version attached. > > You still need to remove this change: > > > dif

Re: ON CONFLICT DO UPDATE for partitioned tables

2018-03-16 Thread Pavan Deolasee
On Fri, Mar 16, 2018 at 5:13 PM, Etsuro Fujita <fujita.ets...@lab.ntt.co.jp> wrote: > (2018/03/16 19:43), Pavan Deolasee wrote: > >> On Fri, Mar 2, 2018 at 9:06 PM, Alvaro Herrera <alvhe...@2ndquadrant.com >> <mailto:alvhe...@2ndquadrant.com>> wrote: >&

Re: Faster inserts with mostly-monotonically increasing values

2018-03-22 Thread Pavan Deolasee
ed some cleanup. > Here is an updated patch. I removed the last offset caching thing completely and integrated your changes for conditional lock access. Some other improvements to test cases too. I realised that we must truncate and re-insert to test index fastpath correctly. Thanks, Pavan -- Pavan Deol

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-24 Thread Pavan Deolasee
it would > at least alleviate the spinlock. This gives us the same level of guarantee that xl_prev used to offer, yet help us use atomic operations. I'll be happy if we can look at that particular change and see if there are any holes there. Thanks, Pavan -- Pavan Deolasee

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-24 Thread Pavan Deolasee
Plan then gets plan_id 2. I can investigate further, but given that we see a similar behaviour with regular UPDATE, I don't think it's worth. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: ON CONFLICT DO UPDATE for partitioned tables

2018-03-05 Thread Pavan Deolasee
having converted the existing ones to use partition relation's varno? May be that works because missing attributes are already added during planning and expand_targetlist() here only adds dropped columns, which are just NULL constants. Thanks, Pavan -- Pavan Deolasee http:

Re: Faster inserts with mostly-monotonically increasing values

2018-03-05 Thread Pavan Deolasee
n that case. I haven't yet figured out if DESC indexes are actually stored in the reverse order. I am gonna look at that too. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Accidental removal of a file causing various problems

2018-09-03 Thread Pavan Deolasee
e, a more interesting question is: should this be automatic or require administrative action? Does either of the ideas sound interesting enough for further work? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: MERGE SQL statement for PG12

2018-11-21 Thread Pavan Deolasee
Hi Tomas, Sorry for a delayed response. On Mon, Oct 29, 2018 at 4:59 PM Tomas Vondra wrote: > Hi Pavan, > > On 10/29/2018 10:23 AM, Pavan Deolasee wrote: > > > > ... > > > > Thanks for keeping an eye on the patch. I've rebased the patch > > against the cu

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-01-08 Thread Pavan Deolasee
e PROGRESS_CREATEIDX_PARTITIONS_TOTAL 3 +#define PROGRESS_CREATEIDX_PARTITIONS_DONE 4 + Is there a reason to leave those reserve placeholders, only to fill them a few lines down? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: MERGE SQL statement for PG12

2019-01-13 Thread Pavan Deolasee
because nothing has changed since then. The purpose of keeping it up-to-date is to solicit feedback and directions and to show that my interest in the patch is still intact. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-02 Thread Pavan Deolasee
On Wed, Mar 27, 2019 at 9:47 AM Masahiko Sawada wrote: > > The patch looks good to me. There is no comment from me. > > Thanks for your review! Updated patch attached since patch failed to apply after recent changes in the master. Thanks, Pavan -- Pavan Deolasee

Re: Re: A separate table level option to control compression

2019-04-03 Thread Pavan Deolasee
n though toast_tuple_target is set to 256. But with the new code, the tuple will get toasted. So that's a change, but in the right direction as far as I can see. Also, this is a bit unrelated to what this patch is trying to achieve. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-03-26 Thread Pavan Deolasee
as is. This code is borrowed from vacuumlazy.c and at some point we can even move it to some common location. > Perhaps we can add some tests for this feature to pg_visibility module. > > That's a good idea. Please see if the tests included in the attached patch are enough. Thanks, Pav

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Pavan Deolasee
required then it's probably too late to consider this for PG12, even though it's more of a bug fix and a candidate for back-patching too. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Pavan Deolasee
he proposed code does not introduce any new behaviour AFAICS. But I might be missing something. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Caveats from reloption toast_tuple_target

2019-04-04 Thread Pavan Deolasee
t_tuple_target is specified at CREATE TABLE, but we would still have problem with ALTER TABLE, no? But there might be side effects of changing the lower limit for pg_dump/pg_restore. So we would need to think about that too. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.c

COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-02-20 Thread Pavan Deolasee
] https://www.postgresql.org/message-id/CAMkU%3D1w3osJJ2FneELhhNRLxfZitDgp9FPHee08NT2FQFmz_pQ%40mail.gmail.com -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services copy_freeze_v3.patch Description: Binary data

Re: [HACKERS] COPY FREEZE and PD_ALL_VISIBLE

2019-02-20 Thread Pavan Deolasee
] for consideration to include in PG12. I started a new thread because the patch is completely new and this thread was a bit too old. Thanks, Pavan [1] https://www.postgresql.org/message-id/CABOikdN-ptGv0mZntrK2Q8OtfUuAjqaYMGmkdU1dCKFtUxVLrg%40mail.gmail.com -- Pavan Deolasee http:

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-02-26 Thread Pavan Deolasee
ed block in BulkInsertState and if the relcache invalidation flushes smgr, start inserting again from the last saved block. In fact, we already track the last used buffer in BulkInsertState and that's enough to know the last used block. Thanks, Pavan -- Pavan Deolasee

  1   2   >