Re: [HACKERS] checkpointer continuous flushing

2016-01-12 Thread Amit Kapila
> the sorting all a relation's early pages are going to be in "in a row". > Not sure, what is best way to tackle this problem, but I think one way could be to perform sorting at flush requests level rather than before writing to OS buffers. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] checkpointer continuous flushing

2016-01-12 Thread Amit Kapila
On Tue, Jan 12, 2016 at 5:52 PM, Andres Freund wrote: > > On 2016-01-12 17:50:36 +0530, Amit Kapila wrote: > > On Tue, Jan 12, 2016 at 12:57 AM, Andres Freund wrote:> > > > > > > My theory is that this happens due to the sorting: pgbench is an update > > &g

Re: [HACKERS] checkpointer continuous flushing

2016-01-12 Thread Amit Kapila
On Tue, Jan 12, 2016 at 7:24 PM, Andres Freund wrote: > On 2016-01-12 19:17:49 +0530, Amit Kapila wrote: > > Why can't we do it at larger intervals (relative to total amount of writes)? > > To explain, what I have in mind, let us assume that checkpoint interval > > is l

Re: [HACKERS] ExecGather() + nworkers

2016-01-13 Thread Amit Kapila
On Mon, Jan 11, 2016 at 9:16 AM, Amit Kapila wrote: > On Mon, Jan 11, 2016 at 3:14 AM, Peter Geoghegan wrote: > > > > On Sun, Jan 10, 2016 at 9:13 AM, Robert Haas wrote: > > >> I'm not sure why the test for nworkers following the > > >> LaunchP

Re: [HACKERS] POC: Cache data in GetSnapshotData()

2016-01-15 Thread Amit Kapila
compare and swap operation. > > After above fix memory corruption is not visible. But I see some more > failures at higher client sessions(128 it is easily reproducible). > > Don't you think we need to update the snapshot fields like count, subcount before saving i

Re: [HACKERS] POC: Cache data in GetSnapshotData()

2016-01-15 Thread Amit Kapila
to a snapshot, so these xids will not be reported as "running" by this function. This is OK for current uses, because we always check TransactionIdIsCurrentTransactionId first, except for known-committed XIDs which could not be ours anyway. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-16 Thread Amit Kapila
gress in the server by checking buffers dirtied/written (we can refer that information using pgBufferUsage) since last time we log this record in bgwriter. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-16 Thread Amit Kapila
On Sat, Jan 16, 2016 at 5:08 PM, Michael Paquier wrote: > > On Sat, Jan 16, 2016 at 7:10 PM, Amit Kapila wrote: > > On Sun, Dec 20, 2015 at 6:44 PM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> On Sun, Nov 8, 2015 at 9:50 PM, Michael Paquier

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-16 Thread Amit Kapila
On Sat, Jan 16, 2016 at 6:37 PM, Michael Paquier wrote: > > On Sat, Jan 16, 2016 at 9:07 PM, Amit Kapila wrote: > > On Sat, Jan 16, 2016 at 5:08 PM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> > >> On Sat, Jan 16, 2016 at 7:10 PM, Amit

Re: [HACKERS] WIP: Rework access method interface

2016-01-16 Thread Amit Kapila
x27;s handler and validate into am specific new file can make this arrangement closer to what we have for PL's (ex. we have plpgsql_validator and plpgsql_call_ handler in pl_handler.c and similar handler and validator functions for other languages in their corresponding modules). With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-18 Thread Amit Kapila
On Mon, Jan 18, 2016 at 10:54 AM, Michael Paquier wrote: > > On Sun, Jan 17, 2016 at 1:37 PM, Amit Kapila wrote: > > On Sat, Jan 16, 2016 at 6:37 PM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> > >> > >> > > > >

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-01-18 Thread Amit Kapila
On Mon, Jan 18, 2016 at 11:06 PM, Robert Haas wrote: > > On Mon, Jan 18, 2016 at 11:09 AM, Alvaro Herrera > wrote: > > Amit Kapila wrote: > > > >> The reason for not updating the patch related to this thread is that it is > >> dependent on the work for

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-18 Thread Amit Kapila
nything relevant happened in any backend since the > last time we performed a checkpoint/logged a running xacts snapshot. > Sounds to be a more robust way of dealing with this problem. Michael, if you don't disagree with above proposal, then we can mark this patch as Waiting on Author? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-19 Thread Amit Kapila
On Tue, Jan 19, 2016 at 12:41 PM, Michael Paquier wrote: > > On Mon, Jan 18, 2016 at 10:19 PM, Amit Kapila wrote: > > On Mon, Jan 18, 2016 at 10:54 AM, Michael Paquier > > wrote: > >> Yes, the thing is that, as mentioned at the beginning of the thread, a > >>

Re: [HACKERS] Re: [JDBC] 9.4-1207 behaves differently with server side prepared statements compared to 9.2-1102

2016-01-19 Thread Amit Kapila
dvocating any such mechanism, but rather sharing an information, I came across. [1] - http://dsl.serc.iisc.ernet.in/publications/conference/bouquet.pdf With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-01-19 Thread Amit Kapila
On Wed, Jan 20, 2016 at 12:41 AM, Robert Haas wrote: > > On Mon, Jan 18, 2016 at 10:41 PM, Amit Kapila wrote: > > Second important and somewhat related point is whether we should save > > this information in PGPROC as 4 bytes or keep it in pgBackendStatus. > > I think

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-01-20 Thread Amit Kapila
7;t been reviewed at all, but the other two have seen a > bit of discussion and evolution. > The last one "PGProc" has been reviewed and all the reported problems have been fixed. It is "Ready For Committer". With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] checkpointer continuous flushing

2016-01-20 Thread Amit Kapila
minute that part of the cause for the > > slowdown is slowness in pg_clog, maybe we should reconsider the initial > > decision to flush as quickly as possible (i.e. adopt a strategy where > > walwriter sleeps a bit between two flushes) in light of the group-update > > feature

Re: [HACKERS] Releasing in September

2016-01-21 Thread Amit Kapila
ess. > > Another idea popping to my mind in order to fix the CF manager burnout > and actually motivate people into becoming CF managers: say that one a > CF manager is done with a commit fest, folks on -hackers discuss if > the CFM has done a good job or not. If yes, he gets a travel paid to > the conference of his choice. > I also think there should be some way to give credit to CFM, if it is difficult to do anything related to money, then we can enforce that if CFM submits any patches for next CF, then those should be prioritised. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Relation extension scalability

2016-01-22 Thread Amit Kapila
On Tue, Jan 12, 2016 at 2:41 PM, Dilip Kumar wrote: > On Thu, Jan 7, 2016 at 4:53 PM, Andres Freund wrote: > >> On 2016-01-07 16:48:53 +0530, Amit Kapila wrote: >> >> I think it's a worthwhile approach to pursue. But until it actually >> fixes the problem of l

Re: [HACKERS] insert/update performance

2016-01-22 Thread Amit Kapila
isible to any active transaction), it won't reduce the size which is already extended, unless the empty space is at end of relation. Are you updating any index column? I think if you should once try with higher fill factor as you have lot of updates. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Releasing in September

2016-01-22 Thread Amit Kapila
L, wherever that occurs. > > Thats a valid point and I think one way to retain such an evidence without adding name of author/reviewer/committer is to add a link to commit/'s after each feature, something like whats done in some of the other release notes [1]. [1] - http://kerne

Re: [HACKERS] insert/update performance

2016-01-23 Thread Amit Kapila
erstand why the recycle ratio is not 50%. > At the moment, I am also not able to see why it is so. You might want to first try with a simple test (Can you extract insert/update statements from application and run it manually for couple of times and then run Vacuum to see the result). By anychanc

Re: [HACKERS] Relation extension scalability

2016-01-23 Thread Amit Kapila
On Sat, Jan 23, 2016 at 12:19 PM, Amit Kapila wrote: > On Tue, Jan 12, 2016 at 2:41 PM, Dilip Kumar > wrote: > >> On Thu, Jan 7, 2016 at 4:53 PM, Andres Freund wrote: >> >>> On 2016-01-07 16:48:53 +0530, Amit Kapila wrote: >>> >>> I think it

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-01-25 Thread Amit Kapila
On Wed, Jan 20, 2016 at 6:39 PM, Robert Haas wrote: > > On Tue, Jan 19, 2016 at 11:49 PM, Amit Kapila wrote: > >> On the topic of the UI, I understand that redefining > >> pg_stat_activity.waiting might cause some short-term annoyance. But I > >> think in th

Re: [HACKERS] Releasing in September

2016-01-26 Thread Amit Kapila
e original Author even though the effort put by reviewer is comparable to Author especially for somewhat complex patches. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-01-26 Thread Amit Kapila
On Tue, Jan 26, 2016 at 1:40 PM, and...@anarazel.de wrote: > > On 2016-01-26 13:22:09 +0530, Amit Kapila wrote: > > @@ -633,9 +633,11 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser > > Time when the state was last changed > > >

Re: [HACKERS] CustomScan under the Gather node?

2016-01-26 Thread Amit Kapila
formation with other set of workers, then it will require such an enhancement. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

[HACKERS] WAL Re-Writes

2016-01-27 Thread Amit Kapila
ncept stage, rather than real implementation, but I think it won't need too much effort to improve it, if we find any particular approach as an acceptable approach. [1] - http://www.postgresql.org/message-id/CA+TgmobWdBcbuipWPsbHSbf+-KDmatnYQYZ=akaju6alb5m...@mail.gmail.com With Regard

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-01-27 Thread Amit Kapila
here if existing users of pg_stat_activity.waiting are still relying on it, they can get wrong information or if they are aware of the recent change, then they need to add additional check like (waiting = true && wait_event_type = 'HWLock')). So I think it is better to go wi

Re: [HACKERS] WAL Re-Writes

2016-01-27 Thread Amit Kapila
On Thu, Jan 28, 2016 at 1:34 AM, james wrote: > On 27/01/2016 13:30, Amit Kapila wrote: > >> >> Thoughts? >> >> Are the decreases observed with SSD as well as spinning rust? > > The test is done with WAL on SSD and data on spinning rust, but I think the result

Re: [HACKERS] New committer

2016-01-28 Thread Amit Kapila
On Thu, Jan 28, 2016 at 8:07 PM, Magnus Hagander wrote: > Hello! > > The PostgreSQL core team would like to welcome Dean Rasheed as a new > committer for the PostgreSQL project. > > Many Congratulation Dean! With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-01-29 Thread Amit Kapila
On Fri, Jan 29, 2016 at 6:21 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > On Mon, Jan 4, 2016 at 5:58 PM, Robert Haas wrote: > >> On Mon, Jan 4, 2016 at 1:20 AM, Amit Kapila >> wrote: >> > If we do that way, then user of API needs to maintain

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-01-29 Thread Amit Kapila
in the sentence below. > > + from _PG_init. Tranche repersents an array of LWLocks >> and >> + can be accessed by it's name. First parameter >> tranche_name > > > Right, will change. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-01-29 Thread Amit Kapila
on: Is there a reason why we should not assign ReplicationOrigins a fixed tranche id and then we might want to even get away with LWLockRegisterTranche()? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-01-30 Thread Amit Kapila
On Thu, Jan 28, 2016 at 9:16 AM, Amit Kapila wrote: > > On Thu, Jan 28, 2016 at 2:12 AM, Robert Haas wrote: > > > > On Tue, Jan 26, 2016 at 3:10 AM, and...@anarazel.de wrote: > > > I do think there's a considerable benefit in improving the > > > in

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-01-31 Thread Amit Kapila
On Sat, Jan 30, 2016 at 12:23 PM, Amit Kapila wrote: > On Fri, Jan 29, 2016 at 6:55 PM, Alexander Korotkov < > a.korot...@postgrespro.ru> wrote: >> >> Also couple of minor comments from me. >> >> I think this >> >> + StrNCpy(LWLockTrancheReque

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-01-31 Thread Amit Kapila
On Sat, Jan 30, 2016 at 2:15 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > > On Sat, Jan 30, 2016 at 10:58 AM, Amit Kapila wrote: >> >> On Fri, Jan 29, 2016 at 6:47 PM, Robert Haas wrote: >> > >> > On Fri, Jan 29, 2016 at 6:59 AM, Alexande

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-01 Thread Amit Kapila
will be updated on each XLOGInsert apart from the XLOGInsert for standby snapshots and use that in a patch somewhat close to what you have in hs-checkpoints-v1. One related point is don't we need to bother about activity on unlogged relations for checkpoints to occur, considering the case when the only activity happened after last checkpoint is on unlogged relations? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-02 Thread Amit Kapila
the second only one line. > Thank you! With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-02-02 Thread Amit Kapila
On Mon, Feb 1, 2016 at 7:10 PM, Alexander Korotkov wrote: > On Sun, Jan 31, 2016 at 6:55 AM, Amit Kapila > wrote: > >> On Thu, Jan 28, 2016 at 9:16 AM, Amit Kapila >> wrote: >> > >> > On Thu, Jan 28, 2016 at 2:12 AM, Robert Haas >> wrote: >&

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-02-02 Thread Amit Kapila
d) into two-bytes which sounds reasonable to me apart from the fact that it might add operation or two extra in this path. Do you or anyone else have any preference over this point? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] WAL Re-Writes

2016-02-02 Thread Amit Kapila
On Mon, Feb 1, 2016 at 8:05 PM, Jim Nasby wrote: > On 1/31/16 3:26 PM, Jan Wieck wrote: > >> On 01/27/2016 08:30 AM, Amit Kapila wrote: >> >>> operation. Now why OS couldn't find the corresponding block in >>> memory is that, while closing the WA

Re: [HACKERS] WAL Re-Writes

2016-02-03 Thread Amit Kapila
On Wed, Feb 3, 2016 at 11:12 AM, Amit Kapila wrote: > > On Mon, Feb 1, 2016 at 8:05 PM, Jim Nasby wrote: >> >> On 1/31/16 3:26 PM, Jan Wieck wrote: >>> >>> On 01/27/2016 08:30 AM, Amit Kapila wrote: >>>> >>>> operation. Now why OS

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-04 Thread Amit Kapila
On Tue, Feb 2, 2016 at 7:19 PM, Robert Haas wrote: > > On Mon, Feb 1, 2016 at 12:27 AM, Amit Kapila wrote: > > Fixed. > > This patch doesn't build: > > ./xfunc.sgml:int lwlock_count = 0; > Tabs appear in SGML/XML files > Changed

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-04 Thread Amit Kapila
On Wed, Feb 3, 2016 at 3:08 PM, Andres Freund wrote: > > On 2016-02-02 10:12:25 +0530, Amit Kapila wrote: > > @@ -8239,7 +8262,7 @@ CreateCheckPoint(int flags) > > if ((flags & (CHECKPOINT_IS_SHUTDOWN | > > CHECKPOINT_END_OF_RECOVERY | > >CHECKPO

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-04 Thread Amit Kapila
On Thu, Feb 4, 2016 at 6:40 PM, Andres Freund wrote: > > On 2016-02-04 18:21:41 +0530, Amit Kapila wrote: > > I think generally it is good idea, but one thing worth a thought is that > > by doing so, we need to acquire all WAL Insertion locks every > > LOG_SNAPSHOT_

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-04 Thread Amit Kapila
On Fri, Feb 5, 2016 at 3:17 AM, Robert Haas wrote: > > On Thu, Feb 4, 2016 at 7:00 AM, Amit Kapila wrote: > > [ new patch ] > > I've committed this after heavy rewriting. In particular, I merged > two loops, used local variables to get rid of the very long and quite &

Re: [HACKERS] Relation extension scalability

2016-02-05 Thread Amit Kapila
e relation without checking if it can get a page with space from FSM. It seems to me that we should re-check the availability of page because while one backend is waiting on extension lock, other backend might have added pages. To re-check the availability we might want to use something similar to LWLockAcquireOrWait() semantics as used during WAL writing. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] "using previous checkpoint record at" maybe not the greatest idea?

2016-02-06 Thread Amit Kapila
ght not be full-proof, but OTOH we should provide some way to allow user to start database and dump the existing contents. Some of the options that comes to mind are provide some way to get the last checkpoint record from WAL or provide a way to compute max-lsn from data-pages and use that with pg_resetxlog utility to allow user to start database. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] "using previous checkpoint record at" maybe not the greatest idea?

2016-02-06 Thread Amit Kapila
On Sun, Feb 7, 2016 at 10:54 AM, Amit Kapila wrote: > > On Tue, Feb 2, 2016 at 5:28 AM, Andres Freund wrote: > > > > Hi, > > > > currently if, when not in standby mode, we can't read a checkpoint > > record, we automatically fall back to the previous ch

Re: [HACKERS] WAL Re-Writes

2016-02-07 Thread Amit Kapila
On Wed, Feb 3, 2016 at 7:12 PM, Robert Haas wrote: > > On Wed, Feb 3, 2016 at 7:28 AM, Amit Kapila wrote: > > On further testing, it has been observed that misaligned writes could > > cause reads even when blocks related to file are not in-memory, so > > I think what Ja

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-08 Thread Amit Kapila
ten locked section */ > > Check. > What is the need of holding locks one-by-one during checkpoint when we anyway are going to take lock on all the insertion slots. + * to not rely on taking an exclusive lock an all the WAL insertion locks, /an all/on all With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] WAL Re-Writes

2016-02-08 Thread Amit Kapila
On Mon, Feb 8, 2016 at 8:11 PM, Robert Haas wrote: > > On Mon, Feb 8, 2016 at 12:08 AM, Amit Kapila wrote: > > I think deciding it automatically without user require to configure it, > > certainly has merits, but what about some cases where user can get > > benefits b

Re: [HACKERS] WAL Re-Writes

2016-02-08 Thread Amit Kapila
On Mon, Feb 8, 2016 at 8:16 PM, Andres Freund wrote: > > On 2016-02-08 10:38:55 +0530, Amit Kapila wrote: > > I think deciding it automatically without user require to configure it, > > certainly has merits, but what about some cases where user can get > > benefits by con

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-08 Thread Amit Kapila
On Tue, Feb 9, 2016 at 5:37 AM, Michael Paquier wrote: > > On Mon, Feb 8, 2016 at 11:24 PM, Amit Kapila wrote: > > On Mon, Feb 8, 2016 at 12:28 PM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> > >> > >> >> /* >

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-09 Thread Amit Kapila
to me after new implementation of named tranches, so I have removed that as well. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com fixed_locks_tranche_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chang

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Amit Kapila
On Tue, Feb 9, 2016 at 6:08 PM, Michael Paquier wrote: > > On Tue, Feb 9, 2016 at 2:24 PM, Amit Kapila wrote: > > Do you see any benefit in allowing checkpoints for such cases considering > > bgwriter will anyway take care of logging standby snapshot for such > > cases?

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Amit Kapila
On Wed, Feb 10, 2016 at 7:17 AM, Michael Paquier wrote: > > On Tue, Feb 9, 2016 at 10:42 PM, Amit Kapila wrote: > > On Tue, Feb 9, 2016 at 6:08 PM, Michael Paquier wrote: > >> Well, the idea is to improve the system responsiveness. Imagine that > >> the call to GetP

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-02-09 Thread Amit Kapila
On Tue, Feb 9, 2016 at 7:26 PM, Thom Brown wrote: > > On 7 January 2016 at 05:24, Amit Kapila wrote: > > On Fri, Dec 25, 2015 at 6:36 AM, Robert Haas wrote: > >> > >> On Wed, Dec 23, 2015 at 1:16 AM, Amit Kapila > >> wrote: > >> >> >

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-09 Thread Amit Kapila
On Tue, Feb 9, 2016 at 11:05 PM, Robert Haas wrote: > > On Tue, Feb 9, 2016 at 7:53 AM, Amit Kapila wrote: > > On Fri, Feb 5, 2016 at 3:17 AM, Robert Haas wrote: > >> I think we ought to move the buffer mapping, lock manager, and > >> predicate lock manager lock

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-09 Thread Amit Kapila
On Wed, Feb 10, 2016 at 12:16 PM, Michael Paquier wrote: > > > On Wed, Feb 10, 2016 at 12:41 PM, Amit Kapila > wrote: > > On Wed, Feb 10, 2016 at 7:17 AM, Michael Paquier < > michael.paqu...@gmail.com> > > wrote: > >> > > > > Consider below

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-10 Thread Amit Kapila
On Wed, Feb 10, 2016 at 1:01 PM, Michael Paquier wrote: > > > On Wed, Feb 10, 2016 at 4:11 PM, Amit Kapila > wrote: > >> >> >> >>> >>> >> > - last_snapshot_lsn != GetXLogInsertRecPtr()) >>> >> > + >>> >>

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-10 Thread Amit Kapila
On Wed, Feb 10, 2016 at 1:42 PM, Michael Paquier wrote: > > On Wed, Feb 10, 2016 at 5:00 PM, Amit Kapila wrote: > > Okay, but isn't it better that we remove the snapshot taken > > at checkpoint time in the main branch or till where this code is > > getting back-patc

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-11 Thread Amit Kapila
On Wed, Feb 10, 2016 at 8:51 PM, Robert Haas wrote: > > On Wed, Feb 10, 2016 at 1:32 AM, Amit Kapila wrote: > > The reason for using centralized way is that we need to request > > named tranches before initialization of shared memory and as far as > > I can see, currentl

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-11 Thread Amit Kapila
On Thu, Feb 11, 2016 at 6:45 PM, Robert Haas wrote: > > On Thu, Feb 11, 2016 at 3:15 AM, Amit Kapila wrote: > > Sounds sensible, however after changes, CreateLWLocks() started > > looking unreadable, so moved initialization and registration of tranches > > to separate f

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-11 Thread Amit Kapila
On Fri, Feb 12, 2016 at 12:39 AM, Robert Haas wrote: > > On Thu, Feb 11, 2016 at 12:10 PM, Amit Kapila wrote: > > Okay, changed as per suggestion. > > Looks good to me; committed. > Thanks! Attached patch changes the name of some of the existing tranches as suggested by

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-02-11 Thread Amit Kapila
of multiple Clog-page updates in same group on the basis that such cases will be less and even if it happens, it won't effect the transaction status update. Do you have anything else in mind? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] [COMMITTERS] pgsql: Code cleanup in the wake of recent LWLock refactoring.

2016-02-12 Thread Amit Kapila
se + RequestAddinLWLocks(1); +#endif + + RequestAddinShmemSpace(SHMEMMSGSZ); + It seems you have moved request for shared memory (RequestAddinShmemSpace()) after requesting locks, any reason for same? You don't need to change the request for shared memory. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

[HACKERS] Prepared Statement support for Parallel query

2016-02-17 Thread Amit Kapila
am expects some data to be generated before-hand and the information of same is added in file-header. Thoughts? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com prepared_stmt_parallel_query_v1.patch Description: Binary data /* * Test case for PreparedStmt * Need first to r

Re: [HACKERS] [COMMITTERS] pgsql: Introduce group locking to prevent parallel processes from deadl

2016-02-18 Thread Amit Kapila
s.objid AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND blocking_locks.pid != blocked_locks.pid JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid WHERE NOT blocked_locks.GRANTED; With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Typo in bufmgr.c that result in waste of memory

2016-02-18 Thread Amit Kapila
HASH_ELEM | HASH_BLOBS); > } > > Your proposed change seems right to me. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-19 Thread Amit Kapila
itional locking has any impact on performance. I think taking locks at intervals of 15s or 30s should not matter much, but it is better to be safe. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Spurious standby query cancellations

2016-02-20 Thread Amit Kapila
;t have any other outstanding timeouts than what + * this function + * uses. If that stops being true, we could cancel the timeouts + * individually, but that'd be slower. + */ Comment seems to be misaligned. 3. + void + StandbyLockTimeoutHandler(void) + { + /* forget any pending STANDBY_DEADLOCK_TIMEOUT

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-02-20 Thread Amit Kapila
On Sat, Feb 13, 2016 at 10:10 AM, Robert Haas wrote: > > On Fri, Feb 12, 2016 at 12:55 AM, Amit Kapila wrote: > > Very Good Catch. I think if we want to address this we can detect > > the non-group leader transactions that tries to update the different > > CLOG page (dif

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-02-20 Thread Amit Kapila
On Sun, Feb 21, 2016 at 12:02 PM, Robert Haas wrote: > On Sun, Feb 21, 2016 at 10:27 AM, Amit Kapila > wrote: > >> >> Client_Count/Patch_Ver 1 64 128 256 >> HEAD(481725c0) 963 28145 28593 26447 >> Patch-1 938 28152 31703 29402 >> >> >> We

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-02-21 Thread Amit Kapila
On Sun, Feb 21, 2016 at 2:28 PM, Robert Haas wrote: > On Sun, Feb 21, 2016 at 12:24 PM, Amit Kapila > wrote: > >> On Sun, Feb 21, 2016 at 12:02 PM, Robert Haas >> wrote: >> >>> On Sun, Feb 21, 2016 at 10:27 AM, Amit Kapila >>> wrote: >>> &

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-02-21 Thread Amit Kapila
On Wed, Feb 3, 2016 at 8:59 AM, Robert Haas wrote: > > On Tue, Feb 2, 2016 at 10:27 PM, Amit Kapila wrote: > > So, let's leave adding any additional column, but Alexander has brought up > > a good point about storing the wait_type and actual wait_event > > informatio

Re: [HACKERS] Prepared Statement support for Parallel query

2016-02-24 Thread Amit Kapila
On Wed, Feb 24, 2016 at 7:27 PM, Robert Haas wrote: > On Wed, Feb 17, 2016 at 6:41 PM, Amit Kapila > wrote: > > Commit d1b7c1ffe72e86932b5395f29e006c3f503bc53d has added > > the support for passing bind parameters to parallel workers, however > > prepared statement wh

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-02-24 Thread Amit Kapila
On Wed, Feb 24, 2016 at 7:14 PM, Robert Haas wrote: > On Mon, Feb 22, 2016 at 10:05 AM, Amit Kapila > wrote: > >> I wouldn't bother tinkering with it at this point. The value isn't > >> going to be recorded on disk anywhere, so it will be easy to change > &

[HACKERS] Performance degradation in commit 6150a1b0

2016-02-24 Thread Amit Kapila
is as to why the commit 6150a1b08a9fe7ead2b25240be46dddeae9d98e1 lead to degradation, but any ideas are welcome. [1] - http://www.postgresql.org/message-id/CAB-SwXZh44_2ybvS5Z67p_CDz=XFn4hNAD=cnmef+qqkxwf...@mail.gmail.com With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Performance degradation in commit 6150a1b0

2016-02-25 Thread Amit Kapila
On Thu, Feb 25, 2016 at 11:38 PM, Simon Riggs wrote: > On 24 February 2016 at 23:26, Amit Kapila wrote: > >> From past few weeks, we were facing some performance degradation in the >> read-only performance bench marks in high-end machines. My colleague >> Mithun, has t

Re: [HACKERS] Prepared Statement support for Parallel query

2016-02-26 Thread Amit Kapila
On Fri, Feb 26, 2016 at 4:37 PM, Robert Haas wrote: > > On Thu, Feb 25, 2016 at 1:09 PM, Robert Haas wrote: > > On Thu, Feb 25, 2016 at 8:53 AM, Amit Kapila wrote: > >>> But if the user says > >>> they want to PREPARE the query, they are probab

Re: [HACKERS] parallel.c is not marked as test covered

2016-08-17 Thread Amit Kapila
le file suggesting that adding more scripts to that group >> would be bad. But yes, perhaps putting this test into its own standalone >> group would be enough of a fix. > > Maybe now would be a good time to address this by applying the attached > patch to master and seeing wh

Re: [HACKERS] [PATCH] bigint txids vs 'xid' type, new txid_recent(bigint) => xid

2016-08-18 Thread Amit Kapila
xact_commit_timestamp > I think there is a value in exposing such a variant which takes bigint and internally converts it to xid. I am not sure the semantics for the other proposal txid_recent() is equivalent to what we have for txid_current(). One thing which is different is that txid_curren

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Amit Kapila
alog information, they need to be aware of HeapTuple and other required stuff like syscache? Again, if they need to update some stats or something like that, they need to be aware of heap tuple format. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-ha

Re: [HACKERS] LWLocks in DSM memory

2016-08-19 Thread Amit Kapila
h difference in performance, > small variance in perf can be attributed to variance in probability of > drawing the particular built-in script. > > Can you specify the m/c details as Andres wants tests to be conducted on some high socket m/c? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Should we cacheline align PGXACT?

2016-08-19 Thread Amit Kapila
ws huge benefit in this case. > For sure, we should validate that it doesn't cause performance regression in > other cases. At least we should test read-write and smaller machines. > Any other ideas? > may be test on Power m/c as well. -- With Regards, Amit Kapila. Enterp

Re: [HACKERS] Should we cacheline align PGXACT?

2016-08-19 Thread Amit Kapila
On Fri, Aug 19, 2016 at 8:24 PM, Alexander Korotkov wrote: > On Fri, Aug 19, 2016 at 3:19 PM, Amit Kapila > wrote: >> >> On Fri, Aug 19, 2016 at 11:42 AM, Alexander Korotkov >> wrote: >> > Hackers, >> > >> > originally this idea was prop

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-19 Thread Amit Kapila
to use it. [1] - https://www.postgresql.org/message-id/caa4ek1lkq_udism-z2dq6cuvjh3db5fnfnnezzbpsrjw0ha...@mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-19 Thread Amit Kapila
CPU intensive is probably a good bet to push down to workers (provided it is safe). -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-20 Thread Amit Kapila
On Sat, Aug 20, 2016 at 10:23 AM, Claudio Freire wrote: > On Sat, Aug 20, 2016 at 1:05 AM, Amit Kapila wrote: >> On Thu, Aug 18, 2016 at 8:24 AM, Claudio Freire >> wrote: >>> >>> A couple of points make me uneasy about this patch, yet I can think of >

Re: [HACKERS] dsm_unpin_segment

2016-08-20 Thread Amit Kapila
should be set to false after dsm_impl_unpin_segment(). Do you need a check if (control_slot >= 0)? In the code just above you have as Assert to ensure that it is >=0. 2. + if (dsm_control->item[seg->control_slot].pinned) + elog(ERROR, "cannot pin a segment that is already pinned&q

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-20 Thread Amit Kapila
On Sat, Aug 20, 2016 at 9:58 PM, Claudio Freire wrote: > On Sat, Aug 20, 2016 at 4:27 AM, Amit Kapila wrote: >> >> That makes sense, but this means there is a chance that the searches >> could lead to different buffers in case of uniqueness checks (the >> search wi

Re: [HACKERS] dsm_unpin_segment

2016-08-20 Thread Amit Kapila
On Sat, Aug 20, 2016 at 6:13 PM, Robert Haas wrote: > On Sat, Aug 20, 2016 at 7:37 AM, Amit Kapila wrote: >> 2. >> + if (dsm_control->item[seg->control_slot].pinned) >> + elog(ERROR, "cannot pin a segment that is already pinned"); >> >> Shou

Re: [HACKERS] dsm_unpin_segment

2016-08-22 Thread Amit Kapila
On Mon, Aug 22, 2016 at 5:24 AM, Thomas Munro wrote: > On Sat, Aug 20, 2016 at 11:37 PM, Amit Kapila wrote: >> On Tue, Aug 9, 2016 at 10:07 AM, Thomas Munro >> wrote: >>> On Tue, Aug 9, 2016 at 12:53 PM, Tom Lane wrote: >>>> The larger picture here is that

Re: [HACKERS] Re: [sqlsmith] FailedAssertion("!(XLogCtl->Insert.exclusiveBackup)", File: "xlog.c", Line: 10200)

2016-08-22 Thread Amit Kapila
sically in similar situation, the count for nonExclusiveBackups will be decremented and if it hits pg_start_backup_callback(), the following Assertion can fail. pg_start_backup_callback() { .. Assert(XLogCtl->Insert.nonExclusiveBackups > 0); .. } -- With Regards, Amit Kapila. EnterpriseDB: http://www.e

Re: [HACKERS] dsm_unpin_segment

2016-08-22 Thread Amit Kapila
On Mon, Aug 22, 2016 at 6:06 PM, Thomas Munro wrote: > On Tue, Aug 23, 2016 at 12:07 AM, Amit Kapila wrote: >> + int control_slot = -1; >> ... >> + if (control_slot == -1) >> + elog(ERROR, "cannot unpin unknown segment handle"); >> >> Isn't

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-08-22 Thread Amit Kapila
On Tue, Aug 23, 2016 at 8:54 AM, Amit Kapila wrote: > $SUBJECT will make hash indexes reliable and usable on standby. > AFAIU, currently hash indexes are not recommended to be used in > production mainly because they are not crash-safe and with this patch, > I hope we can address tha

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Amit Kapila
nager list as defined in rmgrlist.h? Another thing that needs some thoughts is the UI of this patch, currently it is using integer mask which might not be best way, but again as it is intended to be mainly used for tests, it might be okay. Do we want to enable some tests in the regression

<    1   2   3   4   5   6   7   8   9   10   >