Re: Logical archiving

2020-12-06 Thread Craig Ringer
Reply follows inline. I addressed your last point first, so it's out of order. On Fri, 4 Dec 2020 at 15:33, Andrey Borodin wrote > If OLAP cannot consume data fast enough - we are out of space due to repl slot. There is a much simpler solution to this than logical PITR. What we should be doing

Re: Single transaction in the tablesync worker?

2020-12-06 Thread Craig Ringer
On Sat, 5 Dec 2020, 10:03 Amit Kapila, wrote: > On Fri, Dec 4, 2020 at 7:12 PM Ashutosh Bapat > wrote: > > > > On Thu, Dec 3, 2020 at 7:24 PM Amit Kapila > wrote: > > > > > > On Thu, Dec 3, 2020 at 7:04 PM Ashutosh Bapat > > > wrote: > > > > > > > > On Thu, Dec 3, 2020 at 2:55 PM Amit Kapila

Re: Single transaction in the tablesync worker?

2020-12-03 Thread Craig Ringer
On Thu, 3 Dec 2020 at 17:25, Amit Kapila wrote: > Is there any fundamental problem if > we commit the transaction after initial copy and slot creation in > LogicalRepSyncTableStart and then allow the apply of transactions as > it happens in apply worker? No fundamental problem. Both approaches a

Re: pg_ctl.exe file deleted automatically

2020-12-03 Thread Craig Ringer
On Thu, 3 Dec 2020, 13:06 Joel Mariadasan (jomariad), wrote: > Hi, > > > > We are using Windows 2019 server. > > > > Sometimes we see the *pg_ctl.exe* getting automatically deleted. > > Due to this, while starting up Postgres windows service we are getting the > error. > > “Error 2: The system ca

Re: Two fsync related performance issues?

2020-12-02 Thread Craig Ringer
On Wed, 2 Dec 2020 at 15:41, Michael Paquier wrote: > On Tue, Dec 01, 2020 at 07:39:30PM +0800, Craig Ringer wrote: > > On Wed, 14 Oct 2020, 13:06 Michael Paquier, wrote: > >> Yeah, it is safer to assume that it is the responsability of the > >> backup tool to ensu

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2020-12-02 Thread Craig Ringer
On Tue, 27 Oct 2020 at 16:34, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-09-25 09:40, Craig Ringer wrote: > > While working on extensions I've often wanted to enable cache clobbering > > for a targeted piece of code, without paying the price

Re: Two fsync related performance issues?

2020-12-01 Thread Craig Ringer
On Wed, 14 Oct 2020, 13:06 Michael Paquier, wrote: > On Wed, Oct 14, 2020 at 02:48:18PM +1300, Thomas Munro wrote: > > On Wed, Oct 14, 2020 at 12:53 AM Michael Banck > > wrote: > >> One question about this: Did you consider the case of a basebackup being > >> copied/restored somewhere and the re

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2020-12-01 Thread Craig Ringer
On Mon, 30 Nov 2020, 20:38 Anastasia Lubennikova, < a.lubennik...@postgrespro.ru> wrote: > On 27.10.2020 11:34, Peter Eisentraut wrote: > > On 2020-09-25 09:40, Craig Ringer wrote: > >> While working on extensions I've often wanted to enable cache > >> c

Re: TAP test utility module 'PG_LSN.pm'

2020-11-30 Thread Craig Ringer
On Tue, 1 Dec 2020 at 13:58, Michael Paquier wrote: > > On Tue, Dec 01, 2020 at 12:03:41PM +0800, Craig Ringer wrote: > > I'd like to share the attached PG_LSN.pm module that I use when > > writing TAP tests. I suggest that it be considered for inclusion in > > co

TAP test utility module 'PG_LSN.pm'

2020-11-30 Thread Craig Ringer
Hi all I'd like to share the attached PG_LSN.pm module that I use when writing TAP tests. I suggest that it be considered for inclusion in core. It defines a Perl datatype PG_LSN with operator support, so you can write things like cmp_ok($got_lsn, "<", $expected_lsn, "testname") in TAP test

Re: Printing backtrace of postgres processes

2020-11-30 Thread Craig Ringer
On Tue, 1 Dec 2020 at 11:31, Andres Freund wrote: > > Hi, > > On 2020-11-30 13:35:46 +0800, Craig Ringer wrote: > > I find that when I most often want a backtrace of a running, live > > backend, it's because the backend is doing something that isn't > >

Re: Printing backtrace of postgres processes

2020-11-30 Thread Craig Ringer
On Tue, 1 Dec 2020 at 07:04, Tom Lane wrote: > I'd feel better about it if the mechanism had you specify exactly > one target process, and were restricted to a superuser requestor. Er, rather. I actually assumed the former was the case already, not having looked closely yet.

Re: proposal: function pg_setting_value_split() to parse shared_preload_libraries etc.

2020-11-29 Thread Craig Ringer
On Mon, 30 Nov 2020 at 03:24, Anastasia Lubennikova wrote: > > On 23.10.2020 05:06, Ian Lawrence Barwick wrote: > > Having just submitted this, I realised I'm focussing on the GUCs which call > > "SplitDirectoriesString()" (as my specific uses case is for > > "shared_preload_libraries") > > but th

Re: Printing backtrace of postgres processes

2020-11-29 Thread Craig Ringer
> Surely this is *utterly* unsafe. You can't do that sort of stuff in > a signal handler. Not safely, anyway. The signal handler could be called in the middle of a malloc(), a pfree(), or all sorts of other exciting circumstances. It'd have to be extremely careful to use only local resources on t

Notes on physical replica failover with logical publisher or subscriber

2020-11-29 Thread Craig Ringer
Hi all I recently wrote some notes on interaction between physical replication failover/promotion and logical replication publisher and/or standby. As you probably all know, right now we don't support physical failover for logical replication publishers at all, either for in-core logical replicat

RFC: extension_search_path to supplement dynamic_library_path

2020-11-29 Thread Craig Ringer
Hi all We have a dynamic_library_path setting to allow postgres to search additional locations for loadable libraries (since 8.1, commit 761a0bb69b). This permits libraries to be located in alternate locations for loading by shared_preload_libraries, LOAD commands, implicit loading via function ca

Re: Asynchronous Append on postgres_fdw nodes.

2020-11-29 Thread Craig Ringer
"On Thu, Nov 26, 2020 at 9:28 AM movead...@highgo.ca wrote: > > > I test the patch and occur several issues as blow: > > Issue one: > Get a Assert error at 'Assert(bms_is_member(i, node->as_needrequest));' in > ExecAppendAsyncRequest() function when I use more than two foreign table > on different

Re: POC: postgres_fdw insert batching

2020-11-29 Thread Craig Ringer
On Fri, 27 Nov 2020, 14:06 tsunakawa.ta...@fujitsu.com, wrote: > > Also, I'm afraid it requires major surgery or reform of executor. I don't want it to delay the release of reasonably good (10x) improvement with the synchronous interface.) Totally sensible. If it isn't feasible without signific

Re: Printing LSN made easy

2020-11-29 Thread Craig Ringer
On Mon, Nov 30, 2020 at 1:10 AM Tom Lane wrote: > Michael Paquier writes: > > On Fri, Nov 27, 2020 at 04:10:27PM +0530, Ashutosh Bapat wrote: > >> Off list Craig Ringer suggested introducing a new format specifier > >> similar to %m for LSN but I did not get

Re: Add docs stub for recovery.conf

2020-11-29 Thread Craig Ringer
On Sat, Nov 14, 2020 at 1:42 AM Bruce Momjian wrote: > > Clearly we have need for documenting these renamings somewhere. We were > going to go with a simple URL redirect and a "tip" for > default/pre-installed roles, but I like the idea of doing something more > wholistic that covers all of our r

Re: POC: postgres_fdw insert batching

2020-11-27 Thread Craig Ringer
On Sat, 28 Nov 2020, 10:10 Tomas Vondra, wrote: > > > On 11/27/20 7:05 AM, tsunakawa.ta...@fujitsu.com wrote: > > However, the FDW interface as it's implemented today is not designed to > allow that, I believe (we pretty much just invoke the FWD callbacks as > if it was a local AM). It assumes th

Re: POC: postgres_fdw insert batching

2020-11-26 Thread Craig Ringer
On Fri, Nov 27, 2020 at 10:47 AM tsunakawa.ta...@fujitsu.com < tsunakawa.ta...@fujitsu.com> wrote: Covering this one first: I expect postgresExecForeignBatchInsert() would be able to use the libpq > batching API, because it receives an array of tuples and can generate and > issue INSERT statement

Re: POC: postgres_fdw insert batching

2020-11-26 Thread Craig Ringer
On Fri, Nov 27, 2020 at 3:34 AM Tomas Vondra wrote: > Not sure how is this related to app developers? I think the idea was > that the libpq features might be useful between the two PostgreSQL > instances. I.e. the postgres_fdw would use the libpq batching to send > chunks of data to the other si

Re: [PATCH] LWLock self-deadlock detection

2020-11-25 Thread Craig Ringer
On Wed, Nov 25, 2020 at 9:23 PM Ashutosh Bapat wrote: > On Wed, Nov 25, 2020 at 11:47 AM Craig Ringer > wrote: > >> I am also seeing a pattern > >> Assert(!LWLockHeldByMe()); > >> LWLockAcquire() > >> > >> at some places. Should we cha

Re: [PATCH] LWLock self-deadlock detection

2020-11-24 Thread Craig Ringer
On Tue, Nov 24, 2020 at 10:11 PM Ashutosh Bapat < ashutosh.bapat@gmail.com> wrote: > This looks useful. LWLockCheckSelfDeadlock() could use LWLockHeldByMe > variant instead of copying that code with possibly a change in that > function to return the required information. > Yes, possibly so. I

Re: POC: postgres_fdw insert batching

2020-11-24 Thread Craig Ringer
On Thu, Oct 8, 2020 at 10:40 AM tsunakawa.ta...@fujitsu.com < tsunakawa.ta...@fujitsu.com> wrote: > > Thank you for picking up this. I'm interested in this topic, too. (As an > aside, we'd like to submit a bulk insert patch for ECPG in the near future.) > > As others referred, Andrey-san's fast

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-11-24 Thread Craig Ringer
On Wed, Nov 25, 2020 at 2:43 AM Alexey Kondratov wrote: > On 2020-11-24 06:52, Bharath Rupireddy wrote: > > Thanks for the review comments. > > > > On Mon, Nov 23, 2020 at 9:57 PM Alexey Kondratov > > wrote: > >> > >> > v1-0001-postgres_fdw-function-to-discard-cached-connections.patch > >> > >>

Re: ResourceOwner refactoring

2020-11-19 Thread Craig Ringer
and safer to author in general, like providing a simple way to do error trapping and recovery in an extension mainloop without copy/pasting a bunch of PostgresMain code, better default signal handlers, startup/shutdown that shares more with user backends, etc. Right now it's quite tricky to

Re: Add LWLock blocker(s) information

2020-11-19 Thread Craig Ringer
thin a tranche except for individual locks and named tranches. By the way, the LWLock tracepoints currently fire T_NAME(lock) which calls GetLWTrancheName() for each tracepoint hit, so long as Pg is built with --enable-dtrace, even when nothing is actually tracing them. We might want to consider g

[PATCH] LWLock self-deadlock detection

2020-11-19 Thread Craig Ringer
t of it even if nobody likes the idea of merging it. As written the check runs only for --enable-cassert builds or when LOCK_DEBUG is defined. From 0ec0beb294f4c5ed35dbb35260f53b069563638f Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Thu, 19 Nov 2020 14:24:55 +0800 Subject: [PATCH] LWLock self-dea

Re: Detecting File Damage & Inconsistencies

2020-11-17 Thread Craig Ringer
On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs wrote: > > What I'm proposing is an option to add 16 bytes onto each COMMIT > record > Would it make sense to write this at the time we write a topxid assignment to WAL instead? Otherwise it won't be accessible to streaming-mode logical decoding.

Re: abstract Unix-domain sockets

2020-11-17 Thread Craig Ringer
On Fri, Oct 9, 2020 at 3:28 PM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > During the discussion on Unix-domain sockets on Windows, someone pointed > out[0] abstract Unix-domain sockets. > This reminds me on a somewhat random note that SSPI mode authentication should work out of

Re: Add docs stub for recovery.conf

2020-11-15 Thread Craig Ringer
On Sun, Nov 15, 2020 at 1:49 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > On Fri, Nov 13, 2020 at 10:42 AM Bruce Momjian wrote: > >> I think the big problem, and I have seen this repeatedly, is showing up >> with a patch without discussing whether people actually want the >> featur

Re: Add docs stub for recovery.conf

2020-11-12 Thread Craig Ringer
On Fri, Nov 13, 2020 at 11:50 AM Bruce Momjian wrote: > > So you are saying you don't think you are getting sufficient thought > > into your proposal, and getting just a reflex? Just because we don't > > agree with you don't mean we didn't think about it. In fact, we have > > thought about it

Re: Add docs stub for recovery.conf

2020-11-12 Thread Craig Ringer
On Fri, Nov 13, 2020 at 11:31 AM Craig Ringer wrote: > > Can anyone tell me why the solution I proposed is not acceptable, and why > we have to invent a different one instead? The website redirect is good > and all, but doesn't really solve the problem, and I still don'

Re: Add docs stub for recovery.conf

2020-11-12 Thread Craig Ringer
On Thu, Nov 12, 2020 at 11:25 PM Stephen Frost wrote: > > > Now, the pgweb feature that Jonathan wrote recently might actually be > > exactly what we need to fix that, and to address the issue with > > recovery config documentation that Craig raises. > > After chatting with Jonathan about this fo

Re: Add docs stub for recovery.conf

2020-11-11 Thread Craig Ringer
On Thu, Nov 12, 2020 at 4:01 AM Bruce Momjian wrote: > On Wed, Nov 11, 2020 at 08:59:40PM +0100, Daniel Gustafsson wrote: > > > On 11 Nov 2020, at 20:44, Bruce Momjian wrote: > > > On Tue, Nov 10, 2020 at 01:38:14PM +0800, Craig Ringer wrote: > > > > >>

Re: Add docs stub for recovery.conf

2020-11-11 Thread Craig Ringer
On Thu, Nov 12, 2020 at 3:44 AM Bruce Momjian wrote: > On Tue, Nov 10, 2020 at 01:38:14PM +0800, Craig Ringer wrote: > > Hi all > > > > I noticed that when recovery.conf was removed in 2dedf4d9a8 (yay!) the > docs for > > it were removed completely as well. T

Re: PATCH: Report libpq version and configuration

2020-11-10 Thread Craig Ringer
On Tue, Nov 10, 2020 at 2:22 PM Craig Ringer wrote: > > The main things I'd really like to get in place are a way to get the > version as an ELF data symbol, and a simple way to ID the binary. > > So the minimal change would be to declare: > > const char LIBPQ_VERS

Re: PATCH: Report libpq version and configuration

2020-11-09 Thread Craig Ringer
On Tue, Nov 10, 2020 at 12:33 AM Tom Lane wrote: > Alvaro Herrera writes: > > Well, if we can make it run in more systems than just Linux, then it > > seems worth having. The submitted patch seems a little bit on the > > naughty side. > > I agree that the facility seems possibly useful, as long

Add docs stub for recovery.conf

2020-11-09 Thread Craig Ringer
hem instead of removing them. That'll help prevent us from breaking links on the wider web, in 3rd party documentation, etc. From 9d6db17c8bff093c08a90129d0ba181554b5d3e2 Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Tue, 10 Nov 2020 10:51:20 +0800 Subject: [PATCH] Link the old recovery.co

[PATCH] TAP test showing that pg_replication_slot_advance() works on standby

2020-11-09 Thread Craig Ringer
h committing, but it's here so searchers can find it at least. From 218c607b64db493b975bb398528d6a952beeb32f Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Tue, 10 Nov 2020 13:06:41 +0800 Subject: [PATCH] Extend TAP test for pg_replication_slot_advance() to cover standby Show that pg_replica

Re: Internal key management system

2020-10-28 Thread Craig Ringer
On Thu, Oct 29, 2020 at 1:22 AM Stephen Frost wrote: > > Most importantly - I don't think the SQL key adds anything really > > crucial that we cannot do at the SQL level with an extension. An > > extension "pg_wrap" could provide pg_wrap() and pg_unwrap() already, > > using a single master key

Re: Internal key management system

2020-10-28 Thread Craig Ringer
On Wed, Oct 28, 2020 at 12:02 PM Craig Ringer wrote: > On Wed, Oct 28, 2020 at 9:43 AM Bruce Momjian wrote: > > > >> I don't know much about how to hook into that stuff so if you have an >> idea, I am all ears. > > > Yeah, I have a reasonable idea. The mai

Re: Internal key management system

2020-10-27 Thread Craig Ringer
On Wed, Oct 28, 2020 at 9:43 AM Bruce Momjian wrote: > > I don't know much about how to hook into that stuff so if you have an > idea, I am all ears. Yeah, I have a reasonable idea. The main thing will be to re-read the patch and put it into more concrete terms, which I'll try to find time for

Re: Internal key management system

2020-10-27 Thread Craig Ringer
On Tue, 27 Oct 2020, 19:15 Bruce Momjian, wrote: > We could implement a 'command:' prefix now, and maybe > a 'pass:' one, and allow other methods like 'pkcs11' later. > We don't need to do anything except provide a way to tell OpenSSL where to get the KEK from, for situations where having Pg gen

Re: Internal key management system

2020-10-27 Thread Craig Ringer
On Mon, Oct 26, 2020 at 11:02 PM Stephen Frost wrote: TL;DR: * Important to check that key rotation is possible on a replica, i.e. primary and standby can have different cluster passphrase and KEK encrypting the same WAL and heap keys; * with a HSM we can't read the key out, so a pluggable KEK

Re: PATCH: Report libpq version and configuration

2020-10-26 Thread Craig Ringer
On Tue, Oct 27, 2020 at 12:56 AM Tom Lane wrote: > > Alvaro Herrera writes: > > On 2020-Oct-26, Craig Ringer wrote: > >> also adds PQlibInfoPrint() which dumps PQlibInfo() keys/values to stdout. > > > Sounds useful. I'd have PQlibInfoPrint(FILE *) instead, so y

Re: PATCH: Report libpq version and configuration

2020-10-26 Thread Craig Ringer
On Tue, Oct 27, 2020 at 12:41 AM Alvaro Herrera wrote: > On 2020-Oct-26, Craig Ringer wrote: > > > Patch 0001 adds PQlibInfo(), which returns an array of key/value > > description items reporting on configuration like the full version > string, > > SSL support, gs

Re: Internal key management system

2020-10-26 Thread Craig Ringer
On Mon, Oct 19, 2020 at 11:16 AM Masahiko Sawada < masahiko.saw...@2ndquadrant.com> wrote: The patch introduces only key management infrastructure but with no > key. Currently, there is no interface to dynamically add a new > encryption key. I'm a bit confused by the exact intent and use cases b

PATCH: Report libpq version and configuration

2020-10-26 Thread Craig Ringer
27;--enable-debug' '--enable-cassert' '--enable-tap-tests' '--enable-dtrace' 'CC=/usr/lib64/ccache/gcc' 'CFLAGS=-Og -ggdb3' 'CPPFLAGS=' 'CPP=/usr/lib64/ccache/gcc -E' USE_SSL: 0 ENABLE_GSS: 0 ENABLE_THREAD_SAFETY: 1 HAVE_UNIX_

Re: Use standard SIGHUP and SIGTERM handlers in autoprewarm module

2020-10-22 Thread Craig Ringer
On Wed, Oct 7, 2020 at 8:39 PM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > On Wed, Oct 7, 2020 at 8:00 AM Bharath Rupireddy > wrote: > > > > On Tue, Oct 6, 2020 at 11:41 AM Bharath Rupireddy > > wrote: > > > > > > On Tue, Oct 6, 2020 at 11:20 AM Fujii Masao < > masao.fu.

Re: Make procedure OUT parameters work with JDBC

2020-10-19 Thread Craig Ringer
On Mon, 19 Oct 2020, 19:16 Andrew Dunstan, wrote: > > On 10/19/20 5:19 AM, Peter Eisentraut wrote: > > A follow-up to the recently added support for OUT parameters for > > procedures. The JDBC driver sends OUT parameters with type void. > > This makes sense when calling a function, so that the p

Re: [PATCH] We install pg_regress and isolationtester but not pg_isolation_regress

2020-10-15 Thread Craig Ringer
On Fri, 16 Oct 2020, 09:00 Michael Paquier, wrote: > On Thu, Oct 15, 2020 at 01:06:54PM -0400, Tom Lane wrote: > > Other than src/test/modules/brin, the ISOLATION users don't look > > much like real extensions (rather than test scaffolding), either. > > If you discount test scaffolding modules th

Re: Add primary keys to system catalogs

2020-10-06 Thread Craig Ringer
On Sun, 4 Oct 2020, 01:32 John Naylor, wrote: > > On Sat, Oct 3, 2020 at 9:27 AM Craig Ringer > wrote: > > So a big +1 from me for the idea. Especially if we ensure psql > recognises when the relation 'oid' attribute has a declared PK and includes > it in the c

Re: Add primary keys to system catalogs

2020-10-03 Thread Craig Ringer
On Sat, 3 Oct 2020, 14:40 Peter Eisentraut, < peter.eisentr...@2ndquadrant.com> wrote: > I saw someone ask once for a schema diagram of the system catalogs. > Things like that have occasionally been produced manually, but they are > not regularly updated. That made me wonder, why can't we add pri

Re: Improving connection scalability: GetSnapshotData()

2020-10-01 Thread Craig Ringer
tests. Does anybody have a good > idea for how to best, and without undue effort, to integrate this into > PostgresNode.pm? I don't really have a great idea, so I think I'd leave > it with a local helper in the new test? 2ndQ has some infra for that and various other TAP enhanc

PoC patch: expose TCP socket stats for walsenders

2020-09-30 Thread Craig Ringer
ker that periodically requests stats and writes them to an unlogged relation for inspection. Or have monitoring tools poll it, that sort of thing. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise From ddfba877e3c386598e3790b67413

pg_proc.dat "proargmodes is not a 1-D char array"

2020-09-29 Thread Craig Ringer
ARR_DIMS(arr)[0] != numargs || in funcapi.c. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: [PATCH] We install pg_regress and isolationtester but not pg_isolation_regress

2020-09-29 Thread Craig Ringer
On Tue, 29 Sep 2020 at 22:09, Alvaro Herrera wrote: > On 2018-Jun-01, Craig Ringer wrote: > > > On 28 May 2018 at 15:06, Craig Ringer wrote: > > > > > Per topic, the Pg makefiles install pg_regress (for use by extensions) > and > > > htey install the

[PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2020-09-25 Thread Craig Ringer
geted test with TESTS=foo. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise From 372defde443d178fb4a9c8cf4092dea7debf72ea Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Tue, 22 Sep 2020 09:51:00 +0800 Subject: [PATCH v1]

Proposals for making it easier to write correct bgworkers

2020-09-09 Thread Craig Ringer
TopTransactionContext); if (txn->pushed_snapshot) PopActiveSnapshot(); CommitTransactionCommand(); } else { /* Caller didn't change context between start and end */ Assert(CurrentMemoryContext == txn->save_mctx); } (void) MemoryContextSwitchTo(txn->save_mctx); } -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: [PATCH] Detect escape of ErrorContextCallback stack pointers (and from PG_TRY() )

2020-09-08 Thread Craig Ringer
27;t pop until PG_END_TRY(), like in the demo patch attached. That works, but adds another local to each PG_TRY() that's otherwise useless. If we really felt like it we could use it to produce a poor-man's backtrace through each PG_TRY() at the cost of one sizeof(void) per PG_TRY() + som

Re: [PATCH] Detect escape of ErrorContextCallback stack pointers (and from PG_TRY() )

2020-09-07 Thread Craig Ringer
On Fri, 4 Sep 2020 at 14:55, Craig Ringer wrote: > Using a test program: > > return_stack_escape.c:14:3: warning: Address of stack memory associated > with local variable 'g' is still referred to by the global variable > 'guard_ptr' upon returning to t

Re: [PATCH] Detect escape of ErrorContextCallback stack pointers (and from PG_TRY() )

2020-09-03 Thread Craig Ringer
On Fri, 4 Sep 2020 at 14:13, Craig Ringer wrote: > > I actually had a pretty good look around for static analysis options to > see if I could find anything that might help us out before I landed up with > this approach. > Apparently not good enough. https://clang.llvm.or

Re: [PATCH] Detect escape of ErrorContextCallback stack pointers (and from PG_TRY() )

2020-09-03 Thread Craig Ringer
On Thu, 3 Sep 2020 at 22:28, Tom Lane wrote: > Craig Ringer writes: > > The attached patch series adds support for detecting coding errors where > a > > stack-allocated ErrorContextCallback is not popped from the > > error_context_stack before the variable leaves sc

[PATCH] Detect escape of ErrorContextCallback stack pointers (and from PG_TRY() )

2020-09-03 Thread Craig Ringer
. It's not required for these changes, but it might make sense to adopt it at the same time if we're changing usage across the tree anyway. A further patch (7) switches over to using __has_attribute instead of explicit compiler version checks. Again, optional, more of a RFC. -- Craig Ring

Re: Two fsync related performance issues?

2020-05-26 Thread Craig Ringer
On Tue, 12 May 2020, 08:42 Paul Guo, wrote: > Hello hackers, > > 1. StartupXLOG() does fsync on the whole data directory early in the crash > recovery. I'm wondering if we could skip some directories (at least the > pg_log/, table directories) since wal, etc could ensure consistency. Here > is th

Re: Remove page-read callback from XLogReaderState.

2020-05-26 Thread Craig Ringer
On Tue, 26 May 2020, 15:40 Kyotaro Horiguchi, wrote: > > This patch removes all the three callbacks (open/close/page_read) in > XL_ROUTINE from XLogReaderState. It only has "cleanup" callback > instead. > I actually have a use in mind for these callbacks - to support reading WAL for logical dec

Re: what can go in root.crt ?

2020-05-25 Thread Craig Ringer
cepts only the intermediate certificate of WE ISSUE TO ORGS LIKE YOURS.COM . Assuming the client will accept it; not all clients allow you to configure "certificates I trust to sign peers" separately to "certificates that sign my trusted roots". Because really, in security terms

Re: [PATCH] Fix install-tests target for vpath builds

2020-05-25 Thread Craig Ringer
On Thu, 23 Apr 2020 at 12:55, Craig Ringer wrote: > Patch 0001 fixes this issue with vpath postgres builds: > > $ make -C src/test/regress install-tests > /usr/bin/install: cannot create regular file > 'PGPREFIX/lib/postgresql/regress/PGPREFIX/src/test/regress/expected/er

Re: [PATCH] Skip llvm bytecode generation if LLVM is missing

2020-04-22 Thread Craig Ringer
On Thu, 19 Mar 2020 at 18:47, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-03-15 02:28, Craig Ringer wrote: > > On Fri, 13 Mar 2020 at 15:04, Andres Freund > <mailto:and...@anarazel.de>> wrote: > > > > On 2020-03-13 14:08:12 +0

[PATCH] Fix install-tests target for vpath builds

2020-04-22 Thread Craig Ringer
variable NO_DUMMY_CHECK_TARGET . This lets extensions that want to define a 'check' target do so without having make complain at them about redefined targets. This patch is independent of the others and can apply on master directly. -- Craig Ringer http://www.2n

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-15 Thread Craig Ringer
On Thu, 16 Apr 2020 at 13:23, Craig Ringer wrote: > > Just discarding the prepared xacts is not the answer though. > ... however, I have wondered a few times about making vacuum smarter about cases where the xmin is held down by prepared xacts or by replication slots. If we could r

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-15 Thread Craig Ringer
agree that there's a problem with users having to watch the logs or query obscure state to understand what the system is doing and why bloat is being created by abandoned prepared xacts. Just discarding the prepared xacts is not the answer though. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Include sequence relation support in logical replication

2020-04-15 Thread Craig Ringer
On Thu, 16 Apr 2020 at 07:44, Andres Freund wrote: > > > I would like to ask if you have some suggestions or ideas that can make > subscriber receives the current value without the need to > > > > disabling the 32 increment behavior? > > It simply shouldn't expect that to be the case. What do yo

Re: [PATCH] Support built control file in PGXS VPATH builds

2020-04-08 Thread Craig Ringer
On Mon, 30 Mar 2020 at 11:50, Craig Ringer wrote: > > > On Mon, 9 Mar 2020, 17:27 Peter Eisentraut, < > peter.eisentr...@2ndquadrant.com> wrote: > >> On 2020-02-07 04:14, Craig Ringer wrote: >> > The attached patch fixes this by having PGXS resolve >&

Re: Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread Craig Ringer
ust knowing what the oldest xmin is doesn't help you > find *where* it is. Maybe what we need is a view showing all of > these potential sources of an old xmin. Strongly agree. https://www.postgresql.org/message-id/camsr+ygss6jbhmehbxqmdc1xj7sobdsq62ywaekohn-kbqy...@mail.gmail.co

Re: Missing errcode() in ereport

2020-03-29 Thread Craig Ringer
On Fri, 20 Mar 2020, 01:59 Andres Freund, wrote: > Hi, > > On 2020-03-17 10:09:18 -0400, Tom Lane wrote: > > We might want to spend some effort thinking how to find or prevent > > additional bugs of the same ilk ... > > Yea, that'd be good. Trying to help people new to postgres write their > fir

Re: [PATCH] Support built control file in PGXS VPATH builds

2020-03-29 Thread Craig Ringer
On Mon, 9 Mar 2020, 17:27 Peter Eisentraut, < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-02-07 04:14, Craig Ringer wrote: > > The attached patch fixes this by having PGXS resolve > > $(EXTENSION).control along the VPATH. > > Simpler patch: > > diff --

Re: [PATCH] Skip llvm bytecode generation if LLVM is missing

2020-03-14 Thread Craig Ringer
On Fri, 13 Mar 2020 at 15:04, Andres Freund wrote: > On 2020-03-13 14:08:12 +0800, Craig Ringer wrote: > > The alternative would be to detect a missing clang and emit a much more > > informative error than the current one that explicitly suggests retrying > > with > >

Re: logical replication empty transactions

2020-03-12 Thread Craig Ringer
On Tue, 10 Mar 2020 at 02:30, Andres Freund wrote: > Hi, > > On 2020-03-06 13:53:02 +0800, Craig Ringer wrote: > > On Mon, 2 Mar 2020 at 19:26, Amit Kapila > wrote: > > > > > One thing that is not clear to me is how will we advance restart_lsn > > > i

Re: [PATCH] Skip llvm bytecode generation if LLVM is missing

2020-03-12 Thread Craig Ringer
e that explicitly suggests retrying with make with_llvm=no or setting with_llvm=no in the environment. The whole thing is a mess caused by this "enterprise-y" repository split between core and "extras" and I'm rather frustrated by the whole thing, but the current situat

Re: [PATCH] Skip llvm bytecode generation if LLVM is missing

2020-03-11 Thread Craig Ringer
On Thu, 12 Mar 2020 at 03:43, Andres Freund wrote: > On 2020-03-11 11:25:28 +0800, Craig Ringer wrote: > > I propose that per the attached patch PGXS should simply skip adding > > the automatic dependency for .bc files if clang cannot be found. > > Extensions may still

Re: [PATCH] Skip llvm bytecode generation if LLVM is missing

2020-03-10 Thread Craig Ringer
On Wed, 11 Mar 2020 at 13:47, Pavel Stehule wrote: > > st 11. 3. 2020 v 6:43 odesílatel Julien Rouhaud napsal: >> >> Le mer. 11 mars 2020 à 05:28, Laurenz Albe a >> écrit : >>> >>> On Wed, 2020-03-11 at 11:25 +0800, Craig Ringer wrote: >>> >

Re: custom postgres launcher for tests

2020-03-10 Thread Craig Ringer
On Fri, 21 Feb 2020 at 17:05, Ivan N. Taranov wrote: > > On Fri, Feb 21, 2020 at 4:49 AM Craig Ringer wrote: > > > I thought I saw a related patch to this that proposed to add a pg_ctl > > argument. Was that you too? I can't find it at the moment. I've had it

Re: Proposal: PqSendBuffer removal

2020-03-10 Thread Craig Ringer
faifaiOn Sat, 7 Mar 2020 at 02:45, Andres Freund wrote: > > Hi, > > On March 5, 2020 1:23:21 PM PST, Aleksei Ivanov wrote: > >Thank you for your reply! > > > >Yes, you are right there will be a separate call to send the data, but > >is > >copying data each time more costly operation than just one

[PATCH] Skip llvm bytecode generation if LLVM is missing

2020-03-10 Thread Craig Ringer
dependency and Makefile.llvm is included, so it all works. But I don't think it's worth the hassle and I'd rather just skip automatic bitcode generation if we don't find clang. See also yum packagers report at https://www.postgresql.org/message-id/CAMsr+YHokx0rWLV561z3=gai6

Re: logical replication empty transactions

2020-03-05 Thread Craig Ringer
timely response from the replica(s), which delays synchronous replication. You need to send empty txns when synchronous replication is enabled, or instead poke the walsender to force immediate feedback with reply requested. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadran

Re: Proposal: PqSendBuffer removal

2020-03-05 Thread Craig Ringer
protocol extension so we had a way to recover after interrupted sending of a partial message... -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Add PGURI env var for passing connection string to psql in Docker

2020-02-20 Thread Craig Ringer
o peek at the environment of the containerised process(es) from outside the container. Much more likely than being able to peek at a file, anyway. Then again, Docker relies on dropping capabilities and likes to run as root-that-isn't-root-except-when-it's-root, which doesn

Re: custom postgres launcher for tests

2020-02-20 Thread Craig Ringer
ity concern. And you're right that getting a pg_ctl command line or the like into all the places where we'd want it deep inside TAP tests etc is just too much hassle otherwise. I haven't reviewed the code yet, but for the idea a strong +1000 or so. -- Craig Ringer

Re: Internal key management system

2020-02-18 Thread Craig Ringer
rt of non-backdoored crypto is entirely illegal, we just say "that's your problem" to people in those places. I wholly support this approach. Pretty much everything is illegal somewhere. Patents are pain enough already. (Apologies for thread-breaking reply, this is not from my usually-subscribed account. I do not speak in any way for my employer on this matter.) -- Craig Ringer

Re: PL/Python - lifetime of variables?

2020-02-18 Thread Craig Ringer
velopment practice. Instead, use a "with" block, or a try/finally block, and do explicit cleanup for external resources. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Standards compliance of SET ROLE / SET SESSION AUTHORIZATION

2020-02-16 Thread Craig Ringer
Pooler proxies client access to session like ususal, with no need to specially filter. * When the client releases the session, pooler does a protocol-level RESET SESSION AUTHORIZATION to the pooler user, supplying the reset cookie it gave at S-S-A time. > > regards, tom lane > > -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: [Proposal] Add accumulated statistics for wait event

2020-02-11 Thread Craig Ringer
ools like perf, bpftrace and systemtap. That way we have a zero-overhead-when-unused option that can also be used to aggregate the information per-query, per-user, etc. (I really need to add a bunch more tracepoints to make this easier...) -- Craig Ringer http://www.2

Re: POC: GUC option for skipping shared buffers in core dumps

2020-02-10 Thread Craig Ringer
track down some hard-to-repro crash on a production system I really don't want 20GB coredumps... Please, please apply. Please backpatch, if you can possibly stand to do so. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Postgres 32 bits client compilation fail. Win32 bits client is supported?

2020-02-10 Thread Craig Ringer
mixes of different MSVC toolchains versions on a machine. I now maintain isolated VMs with exactly one MSVC version on each to address the amazing level of breakage and incompatibility that MS's various toolchains seem to deliver. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Is custom MemoryContext prohibited?

2020-02-10 Thread Craig Ringer
On Mon, 10 Feb 2020 at 21:19, Kohei KaiGai wrote: > > 2020年2月10日(月) 13:53 Craig Ringer : > > > > On Thu, 6 Feb 2020 at 11:09, Andres Freund wrote: > > > > > I wasn't advocating for making plannodes.h etc frontend usable. I think > > > that's a

Re: WAL usage calculation patch

2020-02-09 Thread Craig Ringer
lel workers and that sort of thing. (I aim to find time to submit a patch for that.) I haven't yet reviewed the patch. -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise

<    1   2   3   4   5   6   >