A cost issue in ORDER BY + LIMIT

2022-08-06 Thread Paul Guo
Hello, Postgres seems to always optimize ORDER BY + LIMIT as top-k sort. Recently I happened to notice that in this scenario the output tuple number of the sort node is not the same as the LIMIT tuple number. See below, postgres=# explain analyze verbose select * from t1 order by a limit 10;

Two small issues related to table_relation_copy_for_cluster() and CTAS with no data.

2022-04-21 Thread Paul Guo
Hello hackers, While reading the latest master branch code, I found something that we may be able to improve. 1. The am table_relation_copy_for_cluster() interface. static inline void table_relation_copy_for_cluster(Relation OldTable, Relation NewTable,

Re: Two patches to speed up pg_rewind.

2021-08-19 Thread Paul Guo
Thanks for reviewing, please see the replies below. On Tue, Aug 17, 2021 at 3:47 PM Michael Paquier wrote: > > On Thu, Aug 05, 2021 at 06:18:03PM +0800, Paul Guo wrote: > > I modified the copy_file_range() patch using the below logic: > > > > If the first call of

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-08-11 Thread Paul Guo
On Wed, Aug 11, 2021 at 4:56 AM Robert Haas wrote: > > On Thu, Aug 5, 2021 at 6:20 AM Paul Guo wrote: > > Rebased. > > The commit message for 0001 is not clear enough for me to understand > what problem it's supposed to be fixing. The code comments aren't > really either.

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-08-05 Thread Paul Guo
Rebased. v12-0002-Tests-to-replay-create-database-operation-on-sta.patch Description: v12-0002-Tests-to-replay-create-database-operation-on-sta.patch v12-0001-Support-node-initialization-from-backup-with-tab.patch Description: v12-0001-Support-node-initialization-from-backup-with-tab.patch

Re: Two patches to speed up pg_rewind.

2021-08-05 Thread Paul Guo
On Tue, Jun 22, 2021 at 11:08 AM Paul Guo wrote: > > On Thu, Jun 17, 2021 at 3:19 PM Michael Paquier wrote: > > > > On Wed, Jun 02, 2021 at 05:02:10PM +0900, Michael Paquier wrote: > > > On Wed, Jun 02, 2021 at 06:20:30PM +1200, Thomas Munro wrote: > > > >

Re: Two patches to speed up pg_rewind.

2021-06-21 Thread Paul Guo
On Thu, Jun 17, 2021 at 3:19 PM Michael Paquier wrote: > > On Wed, Jun 02, 2021 at 05:02:10PM +0900, Michael Paquier wrote: > > On Wed, Jun 02, 2021 at 06:20:30PM +1200, Thomas Munro wrote: > > > The main thing I noticed was that Linux < 5.3 can fail with EXDEV if > > > you cross a filesystem

Re: Two patches to speed up pg_rewind.

2021-06-17 Thread Paul Guo
No worry I’m work on this. On 2021/6/17, 3:18 PM, "Michael Paquier" wrote: On Wed, Jun 02, 2021 at 05:02:10PM +0900, Michael Paquier wrote: > On Wed, Jun 02, 2021 at 06:20:30PM +1200, Thomas Munro wrote: > > The main thing I noticed was that Linux < 5.3 can fail with EXDEV if > > you cross a

Re: Should wal receiver reply to wal sender more aggressively?

2021-06-16 Thread Paul Guo
[ Resending the mail since I found my previous email has a very bad format that is hard to read]. While working on some related issues I found that the wal receiver tries to call walrcv_receive() loop before replying the write/flush/apply LSN to wal senders in XLogWalRcvSendReply(). It is

Should wal receiver reply to wal sender more aggressively?

2021-06-15 Thread Paul Guo
tartup process and primary server know about * them. */ XLogWalRcvFlush(false); -- Paul Guo (Vmware)

Re: Two patches to speed up pg_rewind.

2021-05-27 Thread Paul Guo
> On 2021/2/19, 10:33 AM, "Paul Guo" wrote: > Refactored the code a bit along with fixes. Manually tested them on centos > & Ubuntu (the later has copy_file_range()) > For the first patch, actually I have some concerns. My assumption is that > the target pg_data

Re: sync request forward function ForwardSyncRequest() might hang for some time in a corner case?

2021-05-27 Thread Paul Guo
On Thu, May 27, 2021 at 10:22 PM Julien Rouhaud wrote: > > On Thu, May 27, 2021 at 10:05 PM Paul Guo wrote: > > > > Also note that ForwardSyncRequest() does wake up the checkpointer if > > it thinks the requests in shared memory are "too full", but doe

Re: sync request forward function ForwardSyncRequest() might hang for some time in a corner case?

2021-05-27 Thread Paul Guo
> You said "trigger a checkpoint", which sounded more like forcing a > checkpointer rather than waking up the checkpointer so that it can > absorb the pending requests, so it seems worth to mention what it > would really do. Yes it is not accurate. Thanks for the clarif

Re: sync request forward function ForwardSyncRequest() might hang for some time in a corner case?

2021-05-27 Thread Paul Guo
On Thu, May 27, 2021 at 9:59 PM Paul Guo wrote: > > On Thu, May 27, 2021 at 7:11 PM Julien Rouhaud wrote: > > > > On Tue, May 25, 2021 at 4:39 PM Paul Guo wrote: > > > > > > Hi hackers, > > > > > > I found this when reading th

Re: sync request forward function ForwardSyncRequest() might hang for some time in a corner case?

2021-05-27 Thread Paul Guo
On Thu, May 27, 2021 at 7:11 PM Julien Rouhaud wrote: > > On Tue, May 25, 2021 at 4:39 PM Paul Guo wrote: > > > > Hi hackers, > > > > I found this when reading the related code. Here is the scenario: > > > > bool > > RegisterSyncR

Re: pg_rewind fails if there is a read only file.

2021-05-27 Thread Paul Guo
file) in case users really want to skip some files (probably not just read only files). Presumably the people who run pg_rewind should be DBA or admin that have basic knowledge of this so we should not worry too much about that the user skips some important files (we could even set a deny list for this). Also this solution works fine for a read only file system since pg_rewind soon quits when adding the write permission for any read only file. -- Paul Guo (Vmware)

Re: pg_rewind fails if there is a read only file.

2021-05-25 Thread Paul Guo
> You seem to have missed my point completely. The answer to this problem > IMNSHO is "Don't put a read-only file in the data directory". Oh sorry. Well, if we really do not want this we may want to document this and keep educating users, but meanwhile probably the product should be more user

sync request forward function ForwardSyncRequest() might hang for some time in a corner case?

2021-05-25 Thread Paul Guo
rSyncRequest(), trigger a checkpoint for the scenario. // checkpointer_triggered: variable for one trigger only. if (!ret && retryOnError && ProcGlobal->checkpointerLatch && !checkpointer_triggered) SetLatch(ProcGlobal->checkpointerLatch); Any comments? Regards, Paul Guo (Vmware)

Re: pg_rewind fails if there is a read only file.

2021-05-20 Thread Paul Guo
> Presumably the user has a reason for adding the file read-only to the > data directory, and we shouldn't lightly ignore that. > > Michael's advice is reasonable. This seems like a case of: I agree. Attached is a short patch to handle the case. The patch was tested in my dev environment.

pg_rewind fails if there is a read only file.

2021-05-19 Thread Paul Guo
or thoughts? Thanks, -- Paul Guo (Vmware)

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-03-30 Thread Paul Guo
On 2021/3/27, 10:23 PM, "Alvaro Herrera" wrote: >Hmm, can you post a rebased set, where the points under discussion > are marked in XXX comments explaining what the issue is? This thread is >long and old ago that it's pretty hard to navigate the whole thing in >order to find out

Re: fdatasync performance problem with large number of DB files

2021-03-18 Thread Paul Guo
About the syncfs patch, my first impression on the guc name sync_after_crash is that it is a boolean type. Not sure about other people's feeling. Do you guys think It is better to rename it to a clearer name like sync_method_after_crash or others?

Re: fdatasync performance problem with large number of DB files

2021-03-17 Thread Paul Guo
filesystem might be shared by other applications (this is not unusual and happens in my customer's environment for example) so syncfs() is (probably much) slower than skipping the sync step. -- Paul Guo (Vmware)

Re: fdatasync performance problem with large number of DB files

2021-03-16 Thread Paul Guo
On Tue, Mar 16, 2021 at 4:29 PM Fujii Masao wrote: > > > > On 2021/03/16 8:15, Thomas Munro wrote: > > On Tue, Mar 16, 2021 at 3:30 AM Paul Guo wrote: > >> By the way, there is a usual case that we could skip fsync: A fsync-ed > >> already standby

Re: Freeze the inserted tuples during CTAS?

2021-03-15 Thread Paul Guo
> to set visibility map bits on materialized views. I'll start a new >thread to discuss that. Thanks. Also I withdrew the patch.

Re: fdatasync performance problem with large number of DB files

2021-03-15 Thread Paul Guo
> On 2021/3/15, 7:34 AM, "Thomas Munro" wrote: >> On Mon, Mar 15, 2021 at 11:52 AM Thomas Munro wrote: >> Time being of the essence, here is the patch I posted last year, this >> time with a GUC and some docs. You can set sync_after_crash to >> "fsync" (default) or "syncfs"

Re: Freeze the inserted tuples during CTAS?

2021-03-09 Thread Paul Guo
> On Mar 3, 2021, at 1:35 PM, Masahiko Sawada wrote: >> On Sun, Feb 21, 2021 at 4:46 PM Paul Guo wrote: >> Attached is the v2 version that fixes a test failure due to plan change >> (bitmap index scan -> index only scan). > I think this is a good idea. > BTW, h

Re: Freeze the inserted tuples during CTAS?

2021-02-20 Thread Paul Guo
Attached is the v2 version that fixes a test failure due to plan change (bitmap index scan -> index only scan). v2-0001-Freeze-the-tuples-during-CTAS.patch Description: v2-0001-Freeze-the-tuples-during-CTAS.patch

Re: Freeze the inserted tuples during CTAS?

2021-02-18 Thread Paul Guo
arking as visible will help. Thanks for letting me know there is such a real case in production environment. I attached the short patch. If no more other concerns, I will log the patch on commitfest. -Paul On Wed, Jan 27, 2021 at 12:29 PM Paul Guo mailto:gu...@vmware.com>> wrote: Here is the

Re: Two patches to speed up pg_rewind.

2021-02-18 Thread Paul Guo
Refactored the code a bit along with fixes. Manually tested them on centos & Ubuntu (the later has copy_file_range()) For the first patch, actually I have some concerns. My assumption is that the target pg_data directory should be fsync-ed already. This should be correct normally but there is one

Re: Two patches to speed up pg_rewind.

2021-02-02 Thread Paul Guo
On Jan 28, 2021, at 3:31 PM, Michael Paquier mailto:mich...@paquier.xyz>> wrote: On Wed, Jan 27, 2021 at 09:18:48AM +, Paul Guo wrote: Second one is use copy_file_range() for the local rewind case to replace read()+write(). This introduces copy_file_range() check and HAVE_COPY_FILE

Freeze the inserted tuples during CTAS?

2021-01-27 Thread Paul Guo
Here is the simple patch, diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..0391699423 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -552,7 +552,7 @@ intorel_startup(DestReceiver *self, int operation,

Two patches to speed up pg_rewind.

2021-01-27 Thread Paul Guo
While reading pg_rewind code I found two things could speed up pg_rewind. Attached are the patches. First one: pg_rewind would fsync the whole pgdata directory on the target by default, but that is a waste since usually just part of the files/directories on the target are modified. Other files

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-01-27 Thread Paul Guo
Thanks for the review, please see the replies below. > On Jan 5, 2021, at 9:07 AM, Kyotaro Horiguchi wrote: > > At Wed, 8 Jul 2020 12:56:44 +0000, Paul Guo wrote in >> On 2020/01/15 19:18, Paul Guo wrote: >> I further fixed the last test failure (due to a

Re: Multi Inserts in CREATE TABLE AS - revived patch

2020-11-16 Thread Paul Guo
> On Nov 13, 2020, at 7:21 PM, Bharath Rupireddy > wrote: > > On Tue, Nov 10, 2020 at 3:47 PM Paul Guo wrote: >> >> Thanks for doing this. There might be another solution - use raw insert >> interfaces (i.e. raw_heap_insert()). >> Attached is the t

Re: Multi Inserts in CREATE TABLE AS - revived patch

2020-11-10 Thread Paul Guo
> On Nov 9, 2020, at 6:41 PM, Bharath Rupireddy > wrote: > > On Tue, Nov 3, 2020 at 4:54 PM Bharath Rupireddy > wrote: >> >> If the approach followed in the patch looks okay, I can work on a separate >> patch for multi inserts in refresh materialized view cases. >> > > Hi, I'm attaching a v2

Re: Parallelize stream replication process

2020-09-15 Thread Paul Guo
> On Sep 16, 2020, at 11:15 AM, Li Japin wrote: > > > >> On Sep 15, 2020, at 3:41 PM, Fujii Masao wrote: >> >> >> >> On 2020/09/15 13:41, Bharath Rupireddy wrote: >>> On Tue, Sep 15, 2020 at 9:27 AM Li Japin wrote: For now, postgres use single process to send, receive and

Some two phase optimization ideas

2020-08-27 Thread Paul Guo
Hello hackers, While working on two phase related issues, I found something related to two phase could be optimized. 1. The current implementation decouples PREPRE and COMMIT/ABORT PREPARE a lot. This is flexible, but if PREPARE & COMMIT/ABORT mostly happens on the same backend we could

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-07-08 Thread Paul Guo
, at 10:18 AM, Paul Guo mailto:gu...@vmware.com>> wrote: Thanks for the review. I’m now re-picking up the work. I modified the code following the comments. Besides, I tweaked the test code a bit. There are several things I’m not 100% sure. Please see my replies below. On Jan 27, 2020, at

Re: Two fsync related performance issues?

2020-05-18 Thread Paul Guo
Thanks for the replies. On Tue, May 12, 2020 at 2:04 PM Michael Paquier wrote: > On Tue, May 12, 2020 at 12:55:37PM +0900, Fujii Masao wrote: > > On 2020/05/12 9:42, Paul Guo wrote: > >> 1. StartupXLOG() does fsync on the whole data directory early in > >> the cr

Two fsync related performance issues?

2020-05-11 Thread Paul Guo
Hello hackers, 1. StartupXLOG() does fsync on the whole data directory early in the crash recovery. I'm wondering if we could skip some directories (at least the pg_log/, table directories) since wal, etc could ensure consistency. Here is the related code. if (ControlFile->state !=

Re: Batch insert in CTAS/MatView code

2020-01-16 Thread Paul Guo
I took some time on digging the issue yesterday so the main concern of the patch is to get the tuple length from ExecFetchSlotHeapTuple(). + ExecCopySlot(batchslot, slot); + tuple = ExecFetchSlotHeapTuple(batchslot, true, NULL); + + myState->mi_slots_num++; + myState->mi_slots_size +=

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-01-15 Thread Paul Guo
I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result. v9-0001-Support-node-initialization-from-backup-with-tabl.patch Description: Binary data

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-01-13 Thread Paul Guo
On Fri, Jan 10, 2020 at 9:43 PM Alvaro Herrera wrote: > On 2020-Jan-09, Alvaro Herrera wrote: > > > I looked at this a little while and was bothered by the perl changes; it > > seems out of place to have RecursiveCopy be thinking about tablespaces, > > which is way out of its league. So I

Questions about SyncRepWaitForLSN()

2019-12-15 Thread Paul Guo
Hello pg hackers, This is the definition of the function: SyncRepWaitForLSN(XLogRecPtr lsn, bool commit) 1. In the code, it emits ereport(WARNING) for the ProcDiePending/QueryCancelPending case like this: ereport(WARNING, (errcode(ERRCODE_ADMIN_SHUTDOWN),

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-30 Thread Paul Guo
> > BTW in the future if you have two separate patches, please post them in > separate threads and use separate commitfest items for each, even if > they have minor conflicts. > Sure. Thanks.

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-30 Thread Paul Guo
> > > I went through the remaining two patches and they seem to be very clear > and concise. However, there are two points I could complain about: > > 1) Maybe I've missed it somewhere in the thread above, but currently > pg_rewind allows to run itself with -R and --source-pgdata. In that case >

Re: Batch insert in CTAS/MatView code

2019-09-29 Thread Paul Guo
> > > > > However, I can also see that there is no better alternative. We need > to > > > compute the size of accumulated tuples so far, in order to decide > whether > > > to stop accumulating tuples. There is no convenient way to obtain the > > > length of the tuple, given a slot. How about

Re: Batch insert in CTAS/MatView code

2019-09-29 Thread Paul Guo
On Sat, Sep 28, 2019 at 5:49 AM Andres Freund wrote: > Hi, > > On 2019-09-09 18:31:54 +0800, Paul Guo wrote: > > diff --git a/src/backend/access/heap/heapam.c > b/src/backend/access/heap/heapam.c > > index e9544822bf..8a844b3b5f 100644 > > --- a/src/backend/acce

Re: Batch insert in CTAS/MatView code

2019-09-26 Thread Paul Guo
On Thu, Sep 26, 2019 at 9:43 PM Alvaro Herrera wrote: > On 2019-Sep-25, Asim R P wrote: > > > I reviewed your patch today. It looks good overall. My concern is that > > the ExecFetchSlotHeapTuple call does not seem appropriate. In a generic > > place such as createas.c, we should be using

Re: Batch insert in CTAS/MatView code

2019-09-26 Thread Paul Guo
Asim Thanks for the review. On Wed, Sep 25, 2019 at 6:39 PM Asim R P wrote: > > > > On Mon, Sep 9, 2019 at 4:02 PM Paul Guo wrote: > > > > So in theory > > we should not worry about additional tuple copy overhead now, and then I > tried the patch without s

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-26 Thread Paul Guo
> > > > Note in the 2nd patch, the long option is changed as below. Both the > option > > and description > > now seems to be more concise since we want db state as either > DB_SHUTDOWNED > > or > > DB_SHUTDOWNED_IN_RECOVERY. > > > > "-s, --no-ensure-shutdowned do not auto-fix unclean

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-26 Thread Paul Guo
> > > Yes, -R is already used in pg_basebackup for the same functionality, so > it seems natural to use it here as well for consistency. > > I will review options naming in my own patch and update it accordingly. > Maybe -w/-W or -a/-A options will be good, since it's about WALs > retrieval from

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-26 Thread Paul Guo
On Thu, Sep 26, 2019 at 1:48 AM Alvaro Herrera wrote: > CC Alexey for reasons that become clear below. > > On 2019-Sep-25, Paul Guo wrote: > > > > Question about 0003. If we specify --skip-clean-shutdown and the > cluter > > > was not cleanly shut down, shouldn

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-24 Thread Paul Guo
> > On 2019-Sep-20, Paul Guo wrote: > > > The patch series failed on Windows CI. I modified the Windows build file > to > > fix that. See attached for the v7 version. > > Thanks. > > Question about 0003. If we specify --skip-clean-shutdown and the cluter >

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-20 Thread Paul Guo
The patch series failed on Windows CI. I modified the Windows build file to fix that. See attached for the v7 version. v7-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch Description: Binary data v7-0001-Extact-common-recovery-related-functions-from-pg_.patch Description: Binary

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-19 Thread Paul Guo
I've updated the patch series following the suggestions. Thanks. > > v6-0001-Extact-common-functions-from-pg_basebackup-into-s.patch Description: Binary data v6-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch Description: Binary data

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-09 Thread Paul Guo
> > Thank for rebasing. > > I didn't like 0001 very much. > > * It seems now would be the time to stop pretending we're using a file > called recovery.conf; I know we still support older server versions that > use that file, but it sounds like we should take the opportunity to > rename the

Re: Batch insert in CTAS/MatView code

2019-09-09 Thread Paul Guo
On Fri, Aug 2, 2019 at 2:55 AM Heikki Linnakangas wrote: > On 17/06/2019 15:53, Paul Guo wrote: > > I noticed that to do batch insert, we might need additional memory copy > > sometimes comparing with "single insert" > > (that should be the reason that we previ

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-05 Thread Paul Guo
> > It seems there's minor breakage in the build, per CFbot. Can you > please rebase this? > > There is a code conflict. See attached for the new version. Thanks. v5-0001-Extact-common-functions-from-pg_basebackup-into-s.patch Description: Binary data

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-09-05 Thread Paul Guo
On Tue, Sep 3, 2019 at 11:58 PM Alvaro Herrera wrote: > On 2019-Aug-22, Anastasia Lubennikova wrote: > > > 22.08.2019 16:13, Paul Guo wrote: > > > Thanks. I updated the patch to v5. It passes install-check testing and > > > recovery testing. > > Hi, &

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-08-22 Thread Paul Guo
Thanks. I updated the patch to v5. It passes install-check testing and recovery testing. On Fri, Aug 2, 2019 at 6:38 AM Thomas Munro wrote: > On Mon, Jul 15, 2019 at 10:52 PM Paul Guo wrote: > > Please see the attached v4 patch. > > While moving this to the next CF, I noticed

Re: Possible race condition in pg_mkdir_p()?

2019-07-18 Thread Paul Guo
On Thu, Jul 18, 2019 at 4:57 PM Michael Paquier wrote: > On Thu, Jul 18, 2019 at 04:17:22PM +0800, Ning Yu wrote: > > This is still wrong with current code logic, because when the statusdir > is > > a file the errno is also EEXIST, but it can pass the check here. Even if > > we fix pg_mkdir_p()

How to reclaim the space of dropped columns of a table?

2019-07-15 Thread Paul Guo
Hello hackers, I have been having a question about this with no answer from various sources . As known after dropping a column using 'alter table', table is not rewritten and vacuum full does not remove them also (still see the dropped column in pg_attribute). PG document says:

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-07-15 Thread Paul Guo
On Mon, Jul 8, 2019 at 11:16 AM Thomas Munro wrote: > On Wed, Jun 19, 2019 at 7:22 PM Paul Guo wrote: > > I updated the patch to v3. In this version, we skip the error if copydir > fails due to missing src/dst directory, > > but to make sure the ignoring is legal, I add a

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-15 Thread Paul Guo
On Wed, Jul 10, 2019 at 3:28 PM Michael Paquier wrote: > On Tue, Jul 09, 2019 at 10:48:49PM +0800, Paul Guo wrote: > > Yes, the patches changed Makefile so that pg_rewind and pg_basebackup > could > > use some common code, but for Windows build, I'm not sure where are thos

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-09 Thread Paul Guo
Yes, the patches changed Makefile so that pg_rewind and pg_basebackup could use some common code, but for Windows build, I'm not sure where are those window build files. Does anyone know about that? Thanks. On Tue, Jul 9, 2019 at 6:55 AM Thomas Munro wrote: > On Tue, Jul 2, 2019 at 5:46 PM P

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-01 Thread Paul Guo
On Tue, Jul 2, 2019 at 12:35 AM Alvaro Herrera wrote: > On 2019-Apr-19, Paul Guo wrote: > > > The below patch runs single mode Postgres if needed to make sure the > target > > is cleanly shutdown. A new option is added (off by default). > > v2-0001-Ensure-target-

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-01 Thread Paul Guo
Rebased, aligned with recent changes in pg_rewind/pg_basebackup and then retested. Thanks. On Mon, Jul 1, 2019 at 7:35 PM Thomas Munro wrote: > On Fri, Apr 19, 2019 at 3:41 PM Paul Guo wrote: > > I updated the patches as attached following previous discussions. > > Hi Paul, >

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-06-19 Thread Paul Guo
On Mon, May 27, 2019 at 9:39 PM Paul Guo wrote: > > > On Tue, May 14, 2019 at 11:06 AM Kyotaro HORIGUCHI < > horiguchi.kyot...@lab.ntt.co.jp> wrote: > >> Hello. >> >> At Mon, 13 May 2019 17:37:50 +0800, Paul Guo wrote in < >> caeet0zf9yn4daxyuf

Re: Batch insert in CTAS/MatView code

2019-06-18 Thread Paul Guo
On Mon, Jun 17, 2019 at 8:53 PM Paul Guo wrote: > Hi all, > > I've been working other things until recently I restarted the work, > profiling & refactoring the code. > It's been a long time since the last patch was proposed. The new patch has > now been

Re: Batch insert in CTAS/MatView code

2019-06-17 Thread Paul Guo
. v2 patch is attached. On Thu, Mar 7, 2019 at 4:54 PM Heikki Linnakangas wrote: > On 06/03/2019 22:06, Paul Guo wrote: > > The patch also modifies heap_multi_insert() a bit to do a bit further > > code-level optimization by using static memory, instead of using memory > > c

Re: undefined symbol: PQgssEncInUse

2019-05-29 Thread Paul Guo
Have you used the correct libpq library? If yes, you might want to check the build logs and related files to see where is wrong. In my environment, it's ok with both gssapi enabled or disabled. On Thu, May 30, 2019 at 9:22 AM Donald Dong wrote: > Hi, > > After I make temp-install on HEAD with a

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-05-27 Thread Paul Guo
On Tue, May 14, 2019 at 11:06 AM Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > Hello. > > At Mon, 13 May 2019 17:37:50 +0800, Paul Guo wrote in < > caeet0zf9yn4daxyuflzocayyxuff1ms_oqwea+rwv3gha5q...@mail.gmail.com> > > Thanks for the reply. > >

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-05-13 Thread Paul Guo
Thanks for the reply. On Tue, May 7, 2019 at 2:47 PM Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > > + if (!(stat(parent_path, ) == 0 && S_ISDIR(st.st_mode))) > + { > > This patch is allowing missing source and destination directory > even in consistent state. I don't

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-04-30 Thread Paul Guo
:487, lsn: 0/01638EB8, prev 0/01638E40, desc: DROP dir pg_tblspc/16384/PG_12_201904281/16386 I'm not familiar with the TAP test details previously. I learned a lot about how to test such case from Kyotaro's patch series. On Sun, Apr 28, 2019 at 3:33 PM Paul Guo wrote: > > On Wed,

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-04-28 Thread Paul Guo
ab.ntt.co.jp> > > At Tue, 23 Apr 2019 13:31:58 +0800, Paul Guo wrote in > > > > Hi Kyotaro, ignoring the MakePGDirectory() failure will fix this > database > > > create redo error, but I suspect some other kind of redo, which > depends on > > > t

Re: [Patch] Check file type before calling AllocateFile() for files out of pg data directory to avoid potential issues (e.g. hang).

2019-04-24 Thread Paul Guo
On Wed, Apr 24, 2019 at 10:36 PM Tom Lane wrote: > Alvaro Herrera writes: > > On 2019-Apr-24, Paul Guo wrote: > >> On Wed, Apr 24, 2019 at 12:49 PM Andres Freund > wrote: > >>> This seems like a bad idea to me. IMO we want to support using a pipe > >&

Re: [Patch] Check file type before calling AllocateFile() for files out of pg data directory to avoid potential issues (e.g. hang).

2019-04-23 Thread Paul Guo
On Wed, Apr 24, 2019 at 12:49 PM Andres Freund wrote: > Hi, > > On 2019-04-24 12:46:15 +0800, Paul Guo wrote: > > This is, in theory, not a 100% bug, but it is probably not unusual to see > > conflicts of files between postgresql sqls and other applications on the >

[Patch] Check file type before calling AllocateFile() for files out of pg data directory to avoid potential issues (e.g. hang).

2019-04-23 Thread Paul Guo
Hello, Postgres hackers. I happened to see a hang issue when running a simple copy query. The root cause and repro way are quite simple. mkfifo /tmp/a run sql: copy (select generate_series(1, 10)) to '/tmp/a'; It hangs at AllocateFile()->fopen() because that file was previously created as a

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-04-22 Thread Paul Guo
258021727.horiguchi.kyot...@lab.ntt.co.jp> > > At Mon, 22 Apr 2019 12:36:43 +0800, Paul Guo wrote in > > > > Please see my replies inline. Thanks. > > > > > > On Fri, Apr 19, 2019 at 12:38 PM Asim R P wrote: > > > > > > > On Wed, Ap

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-04-21 Thread Paul Guo
Please see my replies inline. Thanks. On Fri, Apr 19, 2019 at 12:38 PM Asim R P wrote: > On Wed, Apr 17, 2019 at 1:27 PM Paul Guo wrote: > > > > create db with tablespace > > drop database > > drop tablespace. > > Essentially, that sequence of operati

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-04-18 Thread Paul Guo
them and installcheck passes. Thanks. On Wed, Mar 20, 2019 at 1:23 PM Paul Guo wrote: > > > On Wed, Mar 20, 2019 at 1:20 PM Michael Paquier > wrote: > >> On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote: >> > This is a good suggestion also. Will do it. >> >

standby recovery fails (tablespace related) (tentative patch and discussion)

2019-04-17 Thread Paul Guo
Hello postgres hackers, Recently my colleagues and I encountered an issue: a standby can not recover after an unclean shutdown and it's related to tablespace. The issue is that the standby re-replay some xlog that needs tablespace directories (e.g. create a database with tablespace), but the

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
On Wed, Mar 20, 2019 at 1:20 PM Michael Paquier wrote: > On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote: > > This is a good suggestion also. Will do it. > > Please note also that we don't care about recovery.conf since v12 as > recovery parameters are now GUCs

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
On Tue, Mar 19, 2019 at 2:18 PM Michael Paquier wrote: > On Tue, Mar 19, 2019 at 02:09:03PM +0800, Paul Guo wrote: > > The first patch adds an option to automatically generate recovery conf > > contents in related files, following pg_basebackup. In the patch, > >

Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
Hello, Postgres hackers, Please see the attached patches. The first patch adds an option to automatically generate recovery conf contents in related files, following pg_basebackup. In the patch, GenerateRecoveryConf(), WriteRecoveryConf() and escape_quotes() are almost same as them on

Re: Batch insert in CTAS/MatView code

2019-03-12 Thread Paul Guo
On Mon, Mar 11, 2019 at 2:58 AM David Fetter wrote: > On Wed, Mar 06, 2019 at 10:06:27PM +0800, Paul Guo wrote: > > Hello, Postgres hackers, > > > > The copy code has used batch insert with function heap_multi_insert() to > > speed up. It seems that Create Table As

Re: Batch insert in CTAS/MatView code

2019-03-10 Thread Paul Guo
Sorry for the late reply. To Michael. Thank you. I know this commitfest is ongoing and I'm not targeting for this. On Thu, Mar 7, 2019 at 4:54 PM Heikki Linnakangas wrote: > On 06/03/2019 22:06, Paul Guo wrote: > > The patch also modifies heap_multi_insert() a bit to do a bit further

Batch insert in CTAS/MatView code

2019-03-06 Thread Paul Guo
Hello, Postgres hackers, The copy code has used batch insert with function heap_multi_insert() to speed up. It seems that Create Table As or Materialized View could leverage that code also to boost the performance also. Attached is a patch to implement that. That was done by Taylor (cc-ed) and

Re: A WalSnd issue related to state WALSNDSTATE_STOPPING

2018-11-26 Thread Paul Guo
> > On Thu, Nov 22, 2018 at 06:31:04PM +0800, Paul Guo wrote: > > Yes, please see the attached patch. > > Thanks, I have reviewed your patch, and could not resist to change > SyncRepReleaseWaiters() on top of the rest with a comment to not be > confused about the W

Re: A WalSnd issue related to state WALSNDSTATE_STOPPING

2018-11-22 Thread Paul Guo
On Thu, Nov 22, 2018 at 1:29 PM Michael Paquier wrote: > On Wed, Nov 21, 2018 at 04:09:41PM +0900, Michael Paquier wrote: > > The checkpointer initializes a shutdown checkpoint where it tells to all > > the WAL senders to stop once all the children processes are gone, so it > > seems to me that

A WalSnd issue related to state WALSNDSTATE_STOPPING

2018-11-20 Thread Paul Guo
Hello, Recently I encountered a panic (assert failure) on a cassert enabled build of Greenplum database (an MPP database based on postgres). The based postgresql version is 9.4 stable (9.4.19). The panic is not complex. Please see below for more details. This issue seems to be a bug in postgresql

Re: [Patch] Create a new session in postmaster by calling setsid()

2018-09-13 Thread Paul Guo
On Thu, Sep 13, 2018 at 8:20 AM, Michael Paquier wrote: > On Wed, Sep 12, 2018 at 03:55:00PM -0400, Tom Lane wrote: > > Although pg_ctl could sneak it in between forking and execing, it seems > > like it'd be cleaner to have the postmaster proper do it in response to > > a switch that pg_ctl

Re: [Patch] Create a new session in postmaster by calling setsid()

2018-08-07 Thread Paul Guo
On Thu, Aug 2, 2018 at 10:30 PM, Tom Lane wrote: > Paul Guo writes: > > [ make the postmaster execute setsid() too ] > > I'm a bit skeptical of this proposal. Forcing the postmaster to > dissociate from its controlling terminal is a good thing in some > scenarios, b

[Patch] Create a new session in postmaster by calling setsid()

2018-08-01 Thread Paul Guo
Hello, Recently I encountered an issue during testing and rootcaused it as the title mentioned. postmaster children have done this (creating a new session) by calling InitPostmasterChild(), but postmaster itself does not. This could lead to some issues (e..g signal handling). The test script

Re: pg_ugprade test failure on data set with column with default value with type bit/varbit

2018-07-31 Thread Paul Guo
Thanks. I updated the patch as attached. Double-checked those tests passed. 2018-07-30 9:38 GMT+08:00 Thomas Munro : > On Thu, May 17, 2018 at 8:20 PM, Paul Guo wrote: > > Thanks. I tentatively submitted a patch (See the attachment). > > Hi Paul, > > It looks lik

Re: [PATCH] Use access() to check file existence in GetNewRelFileNode().

2018-05-18 Thread Paul Guo
2018-05-17 21:18 GMT+08:00 Michael Paquier <mich...@paquier.xyz>: > On Thu, May 17, 2018 at 05:23:28PM +0800, Paul Guo wrote: > > F_OK seems to be better than R_OK because we want to check file existence > > (not read permission) before creating the relation file wi

Re: [PATCH] Use access() to check file existence in GetNewRelFileNode().

2018-05-17 Thread Paul Guo
F_OK seems to be better than R_OK because we want to check file existence (not read permission) before creating the relation file with the path later. 2018-05-17 17:09 GMT+08:00 Michael Paquier <mich...@paquier.xyz>: > On Thu, May 17, 2018 at 04:09:27PM +0800, Paul Guo wrote: > >

Re: pg_ugprade test failure on data set with column with default value with type bit/varbit

2018-05-17 Thread Paul Guo
to be addressed in the future. (Maybe when anyone checks in test cases pg_upgrade needs to be considered also?) 2018-05-11 3:08 GMT+08:00 Robert Haas <robertmh...@gmail.com>: > On Fri, Mar 30, 2018 at 5:36 AM, Paul Guo <paul...@gmail.com> wrote: > > There is no diff in functiona

  1   2   >