Re: [HACKERS] Restricting maximum keep segments by repslots

2017-11-05 Thread Craig Ringer
beit at a performance cost. I'm thinking of this mainly for logical slots. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to y

Re: [HACKERS] ArrayLists instead of List (for some things)

2017-11-02 Thread Craig Ringer
On 3 November 2017 at 12:41, David Rowley wrote: > On 3 November 2017 at 03:26, Craig Ringer wrote: >> On 2 November 2017 at 22:22, David Rowley >> wrote: >>> Maybe, but the new implementation is not going to do well with places >>> where we perform lcons(). Pr

Re: [HACKERS] SSL and Encryption

2017-11-02 Thread Craig Ringer
r pgsql-hackers. For password crypto please go read the SCRAM thread and the PostgreSQL 10 release notes. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresq

Re: [HACKERS] Client Connection redirection support for PostgreSQL

2017-11-02 Thread Craig Ringer
red statements, any WITH HOLD cursors, temp tables, etc you were working with. Strangeness ensues. But we now have a session-intent stuff though. So we could possibly do it at session level. Backends used just for a redirect would be pretty expensive though. -- Craig Ringer

Re: [HACKERS] proposal: schema variables

2017-11-02 Thread Craig Ringer
mail.com#camsr+yf0g8_fehqyfs8gsfneer9opsmovpfnidjovgqzjzh...@mail.gmail.com It'd be nice if you summarised any outcomes from that and addressed it, rather than taking this as a new topic. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: [HACKERS] Custom compression methods

2017-11-02 Thread Craig Ringer
appened with pg_upgrade, with consuming precious toast bits, and a few other things. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make c

Re: [HACKERS] ArrayLists instead of List (for some things)

2017-11-02 Thread Craig Ringer
s of list data structures, often wrapped with a common API. Java's Collections, the STL, you name it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@po

Re: [HACKERS] ArrayLists instead of List (for some things)

2017-11-02 Thread Craig Ringer
ion underneath > once before. Maybe it's time to do that again. I know some systems use hybrid linked array-lists, where linked list cells are multi-element. https://en.wikipedia.org/wiki/Unrolled_linked_list I don't have much experience with them myself. -- Craig Ringer

Re: [HACKERS] Mapping MERGE onto CTEs (Re: MERGE SQL Statement for PG11)

2017-11-02 Thread Craig Ringer
7;t immutable, > they may be considered for inlining. > https://www.postgresql.org/message-id/5351711493487...@web53g.yandex.ru Yep. All theoretical though, I don't think anyone (myself included) stumped up a patch. -- Craig Ringer http://www.2ndQuadrant.com/ Postg

Re: [HACKERS] Statement-level rollback

2017-11-01 Thread Craig Ringer
unlike some of the prior protocol tweaks I've been interested in, it'd be client-initiated so it should be pretty safe. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-01 Thread Craig Ringer
ould be MERGE CONCURRENTLY ... and when you removed the WHEN NOT MATCHED clause it'd ERROR because that's no longer able to be done with the same concurrency-safe semantics? I don't know if this would be helpful TBH, or if it would negate Simon's compatibility goals. Just another

Re: [HACKERS] Statement-level rollback

2017-11-01 Thread Craig Ringer
oint used for statement level rollback we might still have some optimisation opportunities. Downside is that it needs support in each client driver. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers

Re: [HACKERS] PATCH: enabling parallel execution for cursors explicitly (experimental)

2017-11-01 Thread Craig Ringer
into a temp table / tuplestore / etc. It's not clear to me why an unbounded portal fetch, using the tcp socket windows and buffers for flow control, isn't sufficient. Tomas, can you explain the use case a bit more? -- Craig Ringer http://www.2ndQuadrant.com/ Pos

Re: [HACKERS] proposal: extend shm_mq to support more use cases

2017-11-01 Thread Craig Ringer
etaching. These changes would make using shm_mq persistently MUCH easier, without imposing significant cost on existing users. And it'd make it way simpler to build a layer on top for a 1:m 2-way comms system like Ildus is talking about. -- Craig Ringer http://www.

Re: [HACKERS] Re: Anyone have experience benchmarking very high effective_io_concurrency on NVME's?

2017-10-31 Thread Craig Ringer
in the relevant threads, so you know that. To save others the time, see: * https://lwn.net/Articles/724198/ * https://lwn.net/Articles/671649/ -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hack

Re: [HACKERS] Dynamic result sets from procedures

2017-10-31 Thread Craig Ringer
executions. Instead, the multiple result set emulation requires the caller to 'getObject' the 'refcursor' field's result-object, then cast it to ResultSet, and treat it as a new (nested) result set. True multiple result sets would be exposed in PgJDBC via getMoreResul

Re: [HACKERS] PATCH: enabling parallel execution for cursors explicitly (experimental)

2017-10-31 Thread Craig Ringer
fetch all, and let the socket buffering take care of things, reading results only when it wants them, and letting the server block when the windows are full. That's not to say that SQL-level cursor support wouldn't be nice. I'm just trying to better understand what it's solving

Re: [HACKERS] Reading timeline from pg_control on replication slave

2017-10-29 Thread Craig Ringer
On 28 October 2017 at 06:09, Michael Paquier wrote: > On Fri, Oct 27, 2017 at 1:04 AM, Andrey Borodin wrote: >> I'm working on backups from replication salve in WAL-G [0] >> Backups used to use result of pg_walfile_name(pg_start_backup(...)). Call to >> pg_start_backup() works nice, but "pg_walf

Re: [HACKERS] logical decoding of two-phase transactions

2017-10-29 Thread Craig Ringer
n this patch. Hopefully he'll be able to share it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Timeline ID in backup_label file

2017-10-25 Thread Craig Ringer
ng on > backup methods. > > I would find interesting to add at the bottom of the backup_label file > a new field called "START TIMELINE: %d" to put this information in a > more understandable shape. Any opinions? Strong "yes" from me. -- Craig Ringer

Re: [HACKERS] legitimacy of using PG_TRY , PG_CATCH , PG_END_TRY in C function

2017-10-23 Thread Craig Ringer
On 23 October 2017 at 16:16, Craig Ringer wrote: > On 23 October 2017 at 08:30, John Lumby wrote: > >> All works but not perfectly -- at COMMIT, resource_owner issues >> relcache reference leak messages about relation scans not closed >> and also about snapshot still

Re: [HACKERS] legitimacy of using PG_TRY , PG_CATCH , PG_END_TRY in C function

2017-10-23 Thread Craig Ringer
some point I'd really like to expose that in a more general way so it can be used from background workers. Right now AFAICS most background workers have to cope with errors with a proc_exit(1) and getting restarted to try again. Not ideal. -- Craig Ringer http://www.2ndQu

Re: [HACKERS] How to determine that a TransactionId is really aborted?

2017-10-23 Thread Craig Ringer
ut here, but the context is likely something along > the lines of externally storing all transaction ids, and periodically asking > Postgres if they're known-to-be-aborted-by-all-transactions -- one at a time. I think Peter is asking "why?". -- Craig Ringer ht

Re: [HACKERS] Re: Is anything preventing us from allowing write to foreign tables from standby?

2017-10-17 Thread Craig Ringer
; > No interest yet, but no objections too :-) > I'm going to add this to next commitfest. Superficially at least, it sounds like a good idea. We should only need a virtual xid when we're working with foreign tables since we don't do any local heap changes. How's it work wi

Re: [HACKERS] Query got Killed with CTE.

2017-10-17 Thread Craig Ringer
uge cross-product by the looks, and psql ran of of RAM buffering the result. The OOM killer fired (check 'dmesg' to confirm) and killed psql. The server noticed psql going away, and reported the fact. None of this is surprising. What's the problem here? -- Craig Ringer

Re: [HACKERS] SIGSEGV in BRIN autosummarize

2017-10-17 Thread Craig Ringer
ions as you find in the rpm spec. That can be easier when iterating tests and builds. Since the patches are separate, you can skip the tarball and clone the same tag from git instead. Then apply the rpm patches as separate commits. That's typically what I'll do, makes it easier to keep t

Re: [HACKERS] Determine state of cluster (HA)

2017-10-16 Thread Craig Ringer
On 17 October 2017 at 01:02, Joshua D. Drake wrote: > On 10/15/2017 07:39 PM, Craig Ringer wrote: >> >> On 13 October 2017 at 08:50, Joshua D. Drake wrote: >>> >>> -Hackers, >>> >>> I had a long call with a firm developing front end proxy/cache/

Re: [HACKERS] Determine state of cluster (HA)

2017-10-15 Thread Craig Ringer
On 13 October 2017 at 08:50, Joshua D. Drake wrote: > -Hackers, > > I had a long call with a firm developing front end proxy/cache/HA for > Postgres today. Essentially the software is a replacement for PGPool in > entirety but also supports analytics etc... When I was asking them about > pain poin

Re: [HACKERS] Slow synchronous logical replication

2017-10-12 Thread Craig Ringer
records which is safe to skip > (insert/update/delete records not affecting this publication). That sounds like a giant layering violation too. I suggest focusing on reducing the amount of work done when reading WAL, not trying to jump over whole ranges of WAL. -- Craig Ringer

Re: [HACKERS] show precise repos version for dev builds?

2017-10-11 Thread Craig Ringer
on, but not server_version_num, as GUC_REPORT. So if a client wants server_version_num it has to do another round trip to query for it. - In pg_config, where we don't expose any --version-num only --version. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x

Re: [HACKERS] Slow synchronous logical replication

2017-10-11 Thread Craig Ringer
this operation and so can point wal > -sender to the proper LSN without decoding huge part of WAL. > But it seems to be not so easy to implement. Sounds like confusing layering violations to me. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 S

Re: [HACKERS] show precise repos version for dev builds?

2017-10-11 Thread Craig Ringer
ys, some more subtle than others. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Slow synchronous logical replication

2017-10-11 Thread Craig Ringer
On 9 October 2017 at 15:37, Konstantin Knizhnik wrote: > Thank you for explanations. > > On 08.10.2017 16:00, Craig Ringer wrote: >> >> I think it'd be helpful if you provided reproduction instructions, >> test programs, etc, making it very clear when things

Re: [HACKERS] Latches API on windows

2017-10-09 Thread Craig Ringer
your extension load via shared_preload_libraries, registering its latch in shmem_startup_hook . But ... that's an off-the-cuff guess. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hacker

Re: [HACKERS] Help required to debug pg_repack breaking logical replication

2017-10-08 Thread Craig Ringer
you seek, but start by studying src/backend/access/heap/rewriteheap.c . Notably logical_end_heap_rewrite, logical_rewrite_heap_tuple, logical_begin_heap_rewrite. At a wild "I haven't read any of the relevant code in detail yet" stab in the dark, pg_repack is failing to do the bookkeeping re

Re: [HACKERS] Slow synchronous logical replication

2017-10-08 Thread Craig Ringer
streaming physical replication. Hopefully that gives you something to look into, anyway. Maybe you'll be inspired to work on parallelized logical decoding :) -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- S

Re: [HACKERS] Still another race condition in recovery TAP tests

2017-10-06 Thread Craig Ringer
of modules they must first install. Meh, I don't buy that. At worst, all we have to do is provide a script that fetches them, from distro repos if possible, and failing that from CPAN. With cpanminus, that's pretty darn simple too. -- Craig Ringer http://www.2ndQuadran

Re: [HACKERS] [PATCH] A hook for session start

2017-10-05 Thread Craig Ringer
hen a user has hundreds of plpgsql functions that all like to make temp tables. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

[HACKERS] PATCH: Expose generate_qualified_relation_name functionality

2017-10-05 Thread Craig Ringer
place it because it cares about IsBootstrapProcessingMode(). (Prompted by https://dba.stackexchange.com/q/187788/7788) -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services From 011cd6e08d6d29854db637d6beb8709615f376cb Mon Sep 17 00:00

Re: [HACKERS] fork()-safety, thread-safety

2017-10-05 Thread Craig Ringer
On 6 October 2017 at 08:06, Andres Freund wrote: > On 2017-10-06 07:59:40 +0800, Craig Ringer wrote: >> The only thing that gets me excited about a threaded postgres is the >> ability to have a PL/Java, PL/Mono etc that don't suck. We could do >> some really cool things

Re: [HACKERS] fork()-safety, thread-safety

2017-10-05 Thread Craig Ringer
only thing that gets me excited about a threaded postgres is the ability to have a PL/Java, PL/Mono etc that don't suck. We could do some really cool things that just aren't practical right now. Not compelling to a wide audience, really. -- Craig Ringer http://www.2nd

Re: [HACKERS] Postgresql gives error that role goes not exists while it exists

2017-10-03 Thread Craig Ringer
On 3 October 2017 at 20:47, Euler Taveira wrote: > > 2017-10-03 5:49 GMT-03:00 Nick Dro : > > Can someone assists with the issue posted on StackOverflow? > > > > https://stackoverflow.com/questions/46540537/postgresql-9-3-creation-of-group-role-causes-permission-problems > > > > > > Creation of ne

Re: [HACKERS] [PATCH] Off-by-one error in logical slot resource retention

2017-10-01 Thread Craig Ringer
ther > > in the process? > > Since this patch has been in Waiting for Author state for the duration of > the > commitfest without moving, I’m marking it Returned with Feedback. If > there is > still interest in pursuing this patch, please re-submit it to the next > commitfest with the comments addressed. > Thanks. I'll revisit it next CF. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] pg_prepared_xact_status

2017-10-01 Thread Craig Ringer
x27;m not opposed, I just don't really see the point. I'm not seeing where it'd come in useful. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] pg_prepared_xact_status

2017-10-01 Thread Craig Ringer
id. If we get better about that, then we might need some way to ask Pg to keep extra clog. But for now it works well enough. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] pg_prepared_xact_status

2017-09-29 Thread Craig Ringer
o-ordinate. It can't do anything else, since if it unilaterally commits or rolls back it might later find out that the nodes on the other side of the network partition or whatever made the opposite decision and, boom! -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] pg_prepared_xact_status

2017-09-29 Thread Craig Ringer
built-in transaction resolver; then I think it's probably not going to get far. I could see a full DTC resolver in postgres one day, once we have things like working in-core logical rep based multi-master with 2PC support. But that's a looong way off. -- Craig Ringer

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-26 Thread Craig Ringer
t impossible, but probably irritating and verbose. And you'd have none of the DDL required to manage it, so you'd need SQL-function equivalents. I suspect you'd be better off tweaking pglogical to speak the same protocol as pg10, since the pgoutput protocol is an evolution of p

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-26 Thread Craig Ringer
noted, we can likely use pgoutput for that to some extent at least. I think the pressing need is json, going by the zillion plugins out there for it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-26 Thread Craig Ringer
;s done well. That said, I'm all in favour of a generic json output plugin that shares infrastructure with logical replication, so people who are on inflexible environments have a fallback option. I just don't care to write it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Craig Ringer
>>> >>> https://www.2ndquadrant.com/en/resources/2ndqpostgres/ >>> >> >> This doesn't seem like a good way to argue. >> >> > Sorry, that wasn't supposed to be negative. My point was that 2ndQuadrant > has a distribution of Post

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Craig Ringer
much care for that. Personally I'd be more friendly toward Amazon / Google / etc wanting us to include things for their convenience if they actually usefully contributed to development and maintenance of Pg. -- Craig Ringer

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-24 Thread Craig Ringer
t some time ago but ran into some issues and time constraints. Because of the need to support older versions I'm now committed to an approach using direct libpq connections and function calls instead, but it seems like a real shame to do that when the replication protocol connection is *r

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-24 Thread Craig Ringer
or Amazon then? But now there's a new home-invented plugin that we should adopt, ignoring any of the existing ones. Why? https://github.com/apigee-labs/transicator/tree/master/pgoutput > No README? Why did this need to be invented, rather than using an existing plugin? I don't

Re: [HACKERS] Assertion failure when the non-exclusive pg_stop_backup aborted.

2017-09-21 Thread Craig Ringer
t. > Another one to watch out for is that elog(...) and ereport(...) invoke CHECK_FOR_INTERRUPTS. That's given me exciting surprises before when combined with assertion checking and various exit cleanup hooks. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Error: dsa_area could not attach to a segment that has been freed

2017-09-20 Thread Craig Ringer
On 21 September 2017 at 05:50, Thomas Munro wrote: > On Thu, Sep 21, 2017 at 12:59 AM, Robert Haas > wrote: > > On Wed, Sep 20, 2017 at 5:54 AM, Craig Ringer > wrote: > >> By the way, dsa.c really needs a cross-reference to shm_toc.c and vice > >> vers

Re: [HACKERS] Error: dsa_area could not attach to a segment that has been freed

2017-09-20 Thread Craig Ringer
On 20 September 2017 at 17:52, Craig Ringer wrote: > On 20 September 2017 at 16:55, Thomas Munro > wrote: > >> On Wed, Sep 20, 2017 at 6:14 PM, Gaddam Sai Ram >> wrote: >> > Thank you very much! That fixed my issue! :) >> > I was in an assumption

Re: [HACKERS] Error: dsa_area could not attach to a segment that has been freed

2017-09-20 Thread Craig Ringer
g > long lived DSA area you have nothing like that. We need, IMO, a DSA-backed heirachical MemoryContext system. We can't use the exact MemoryContext API as-is due to the need for far pointers though :( -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] Utilities for quoting literals and identifiers in Pg TAP tests

2017-09-19 Thread Craig Ringer
Hi all Here's a little utility class I wrote for value and identifier quoting for use in TAP tests. Might be handy for others. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services PGValues.pm Description: Per

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Craig Ringer
On 20 September 2017 at 12:16, Craig Ringer wrote: > The thought I had in mind upthread was to get rid of logicalrep slots >> in favor of expanding the underlying bgworker slot with some additional >> fields that would carry whatever extra info we need about a logicalrep >>

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Craig Ringer
On 20 September 2017 at 12:06, Amit Kapila wrote: > On Wed, Sep 20, 2017 at 9:23 AM, Tom Lane wrote: > > Craig Ringer writes: > >> On 19 September 2017 at 18:04, Petr Jelinek < > petr.jeli...@2ndquadrant.com> > >> wrote: > >>> If y

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Craig Ringer
On 20 September 2017 at 11:53, Tom Lane wrote: > Craig Ringer writes: > > On 19 September 2017 at 18:04, Petr Jelinek < > petr.jeli...@2ndquadrant.com> > > wrote: > >> If you are asking why they are not identified by the > >> BackgroundWorkerHandle,

Re: [HACKERS] Show backtrace when tap tests fail

2017-09-19 Thread Craig Ringer
On 20 September 2017 at 06:36, David Steele wrote: > > I just use: > > $SIG{__DIE__} = sub {Carp::confess @_}; > That's what I patched into my TestLib.pm too, until I learned of Carp::Always. I'd rather have Carp::Always, but it's definitely an O

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Craig Ringer
d (b) the postmaster remembering bgworker registrations across crash restart with no way to tell it not to. Maybe Petr remembers the details? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Craig Ringer
help. At some point we'll also want to be able to enumerate background workers and get handles for existing workers. Also, let background workers recover from errors without exiting, which means factoring a bunch of stuff out of PostgresMain. But both of those are bigger jobs. -- Craig Ringer

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-09-14 Thread Craig Ringer
emit raw flags by default, so FROZEN would't be shown at all, only COMMITTED|INVALID. If the bool to decode combined flags is set, then it'll show things like FROZEN, and hide COMMITTED|INVALID. Similar for other combos. -- Craig Ringer http://www.2ndQuadrant.com/

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2017-09-12 Thread Craig Ringer
On 13 September 2017 at 13:44, Vaishnavi Prabakaran < vaishnaviprabaka...@gmail.com> wrote: > Thanks for explaining. Will change this too in next version. > > Thankyou, a lot, for picking up this patch. -- Craig Ringer http://www.2ndQuadrant.com/ Postgre

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2017-09-12 Thread Craig Ringer
PQbatchSyncQueue >> > + >> > + PQbatchSyncQueue >> > + >> > + >> >> I wonder why this isn't framed as PQbatchIssue/Send/...()? Syncing seems >> to mostly make sense from a protocol POV. >> >> > Renamed to PQ

Re: [HACKERS] WIP: Failover Slots

2017-09-06 Thread Craig Ringer
On 14 August 2017 at 11:56, Craig Ringer wrote: > > I don't want to block failover slots on decoding on standby just because > decoding on standby would be nice to have. > However, during discussion with Tomas Munro a point has come up that does block failover slots as cur

Re: [HACKERS] [PATCH] Fix drop replication slot blocking instead of returning error

2017-08-29 Thread Craig Ringer
the new behavior was needed for internal > > future functions since the doc wasn't changed. > > FWIW, I also don't think it's ok to just change the behaviour > unconditionally and without a replacement for existing behaviour. Seems like it just needs a new argument nowait DEFAULT false Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Custom allocators in libpq

2017-08-28 Thread Craig Ringer
s and pass it to a connect function. I pretty much always just put the user's original connstring in 'dbname' and set expand_dbname = true instead. It might make sense to have any new function accept PQconninfoOption*. Or a variant of PQconninfoParse that populates k/v arrays with 'n' extra fields allocated and zeroed on return, I guess. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Make pg_regress print a connstring with sockdir

2017-08-28 Thread Craig Ringer
On 28 August 2017 at 19:45, Tom Lane wrote: > Craig Ringer writes: > > It's a pain having to find the postmaster command line to get the port > > pg_regress started a server on. We print the port in the pg_regress > output, > > why not the socket directory / ho

Re: [HACKERS] psql --batch

2017-08-28 Thread Craig Ringer
so's anything really. null bytes aren't usable for all scripts, and nothing else cannot also be output in the data its self. No easy answers there. In cases where I expect that to be an issue I sometimes use \COPY ... TO STDOUT WITH (FORMAT CSV) though. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] psql --batch

2017-08-28 Thread Craig Ringer
ong option version if -B is added, and it is >> auto-completion friendly. > > > This doesn't really address the original issue though, that it's far from obvious how to easily and correctly script psql. I guess there's always the option of a docs patch for that. *shrug* I'll see what others have to say. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Make pg_regress print a connstring with sockdir

2017-08-28 Thread Craig Ringer
On 28 August 2017 at 15:19, Michael Paquier wrote: > On Mon, Aug 28, 2017 at 4:07 PM, Craig Ringer > wrote: > > == starting postmaster== > > running with PID 30235; connect with: > > psql "host='/tmp/pg_regress-j74

Re: [HACKERS] Make pg_regress print a connstring with sockdir

2017-08-28 Thread Craig Ringer
st 2017 at 14:08, Michael Paquier wrote: > On Mon, Aug 28, 2017 at 2:28 PM, Craig Ringer > wrote: > > It's a pain having to find the postmaster command line to get the port > > pg_regress started a server on. We print the port in the pg_regress > output, > > why

Re: [HACKERS] psql --batch

2017-08-28 Thread Craig Ringer
an the first two, though, as I can at least think of some cases where you might want it. X: skip .psqlrc Reliable, portable scripted psql shouldn't be using the local .psqlrc IMO. It's likely to just break things in exciting ways. But I can see it being reasonable to require this

[HACKERS] Make pg_regress print a connstring with sockdir

2017-08-27 Thread Craig Ringer
e attached? If you'd prefer nicer wording at the expense of two lines, maybe running with PID 16409 connection string: 'port=50848 host=/tmp/blah' -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services From 5b242d

[HACKERS] psql --batch

2017-08-27 Thread Craig Ringer
Hi all I find myself regurgitating the incantation psql -qAtX -v ON_ERRORS_STOP=1 quite a bit. It's not ... super friendly. It strikes me that we could possibly benefit from a 'psql --batch' option. Thoughts? -- Craig Ringer http://www.2ndQuadrant.c

Re: [HACKERS] WIP: Separate log file for extension

2017-08-27 Thread Craig Ringer
be left as stubs initially. So the outcome would be the same, just without the assumption of specific file name and output mechanism baked in. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] shm_mq_wait_internal gets stuck forever on fast shutdown

2017-08-21 Thread Craig Ringer
applying shm_mq to non-bgworker endpoints. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] shm_mq_wait_internal gets stuck forever on fast shutdown

2017-08-20 Thread Craig Ringer
On 21 August 2017 at 10:57, Craig Ringer wrote: > Hi all > > I've noticed a possible bug / design limitation where shm_mq_wait_internal > sleep in a latch wait forever, and the postmaster gets stuck waiting for > the bgworker the wait is running in to exit. > > This ha

[HACKERS] shm_mq_wait_internal gets stuck forever on fast shutdown

2017-08-20 Thread Craig Ringer
function pointer called on each iteration to test whether looping should continue, to be passed to shm_mq_attach. So if you can't supply a bgw handle, you supply that instead. Provide a shm_mq_set_handle equivalent for it too. Any objections to the last approach? -- Craig Ringer

Re: [HACKERS] Updating line length guidelines

2017-08-20 Thread Craig Ringer
fers side by side, or when I'm working in poor conditions where I've set my terminal to "giant old people text" sizes, I remember the advantages of a width limit. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [RFC] What would be difficult to make data models pluggable for making PostgreSQL a multi-model database?

2017-08-20 Thread Craig Ringer
tes, like the path model. With some kind of interaction so the sub-planner for the other model could know to generate a different sub-plan based on the context of the outer plan. I have no idea how that could work. But I think you have about zero chance of achieving what you want by going straight

Re: [HACKERS] Function to move the position of a replication slot

2017-08-16 Thread Craig Ringer
hink it's quite a useful > function to be used by an automated system. E.g. to ensure enough, but > not too much, WAL is available for a tertiary standby both on the actual > primary and a failover node. > I strongly agree. If you really need to move a physical slot back (why?)

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-08-16 Thread Craig Ringer
s we have macros, and I think it'd make sense to use them here too. Eschew direct use of HEAP_XMIN_COMMITTED, HEAP_XMIN_INVALID and HEAP_XMIN_FROZEN in tests. Instead, consistently use HeapXminIsFrozen(), HeapXminIsCommitted(), and HeapXminIsInvalid() or something like that. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Function to move the position of a replication slot

2017-08-16 Thread Craig Ringer
all xacts, or by lower level use of the decoding code. Reasonable? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-08-15 Thread Craig Ringer
erpret the bitmasks (omitting some of > the information) assuming all the bits were set correctly. I agree, and the patch already does half of this: it can output just the raw bit flags, or it can interpret them to show HEAP_XMIN_FROZEN etc. So the required change, which seems to have broad agreement, is to have the "interpret the bits" mode show only HEAP_XMIN_FROZEN when it sees HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID, etc. We can retain raw-flags output as-is for when seriously bogus state is suspected. Any takers? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] What users can do with custom ICU collations in Postgres 10

2017-08-14 Thread Craig Ringer
On 15 August 2017 at 10:16, Michael Paquier wrote: > On Tue, Aug 15, 2017 at 11:10 AM, Craig Ringer > wrote: > > Ooh, this finally gives us a path toward case-insensitive default > database > > collation via CLDR caseLevel. > > > > http://userguide.i

Re: [HACKERS] What users can do with custom ICU collations in Postgres 10

2017-08-14 Thread Craig Ringer
s come first, which we see here: > > Ooh, this finally gives us a path toward case-insensitive default database collation via CLDR caseLevel. http://userguide.icu-project.org/collation http://www.unicode.org/reports/tr35/tr35-collation.html#Algorithm_Case That *definitely* should be

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-08-14 Thread Craig Ringer
all over pg_stat_replication and pg_replication_slots and so on. They're already routinely used for monitoring replication lag in bytes, waiting for a peer to catch up, etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-08-14 Thread Craig Ringer
On 15 August 2017 at 09:11, Moon Insung wrote: > Dear Craig Ringer > > > > Frist, thank you for implementing the necessary function. > > > > but, i have some question. > > > > question 1) vacuum freeze hint bits > > If run a vacuum freeze, bits in

Re: [HACKERS] WIP: Failover Slots

2017-08-13 Thread Craig Ringer
too, just as proposed for the physical case, though no replica->master reporting would be needed for logical failover. So despite my initial expectations they can be moderately similar in broad structure. But I don't think there's going to be much actual code overlap beyond minor things like both wanting a way to query slot state on the upstream. Both *could* use decoding on standby to advance slot positions, but for the physical case that's just a slower (and unfinished) way to do what we already have, wheras it's necessary for logical failover. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Thoughts on unit testing?

2017-08-13 Thread Craig Ringer
t strict unit testing as the rest of Pg's APIs aren't mocked away, but it's very practical small-unit integration testing that helps catch issues. I wouldn't mind having an easier and nicer way to do that built in to Pg, but don't have many ideas about practical, low-maint

Re: [HACKERS] WIP: Failover Slots

2017-08-10 Thread Craig Ringer
vail as https://gist.github.com/ringerc/d4a8fe97f5fd332d8b883d596d61e257 ) To actually use the slot once decoding on standby is supported: a decoding client on "C" can consume xacts and cause slot "X" to advance catalog_xmin, confirmed_flush_lsn, etc. walreceiver on "C" will tell walsender on "B" about the new slot state, and it'll get synced up-tree, then B will tell A. Since slot is already marked permanent, state won't get copied back down-tree, that only happens once when slot is first fully created on master. Some node "D" can exist as a phys rep of "C". If C fails and is replace with D, admin can promote the down-mirror slot on "D" to an owned slot. Make sense? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Thoughts on unit testing?

2017-08-10 Thread Craig Ringer
#x27; and move toward higher level visibility (https://gcc.gnu.org/wiki/Visibility: -fvisibility=hidden and __attribute__((dllexport)) ). It'd make it easier not to forget needed PGDLLEXPORTs, let us hide stuff we consider really internal but still share across a few files, etc.) -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [TRAP: FailedAssertion] causing server to crash

2017-08-10 Thread Craig Ringer
On 10 August 2017 at 23:25, Robert Haas wrote: > On Mon, Aug 7, 2017 at 2:06 AM, Craig Ringer > wrote: > > I think so - specifically, that it's a leftover from a revision where the > > xid limit was advanced before clog truncation. > > > > I'll be finding

Re: [HACKERS] WIP: Failover Slots

2017-08-09 Thread Craig Ringer
On 9 August 2017 at 23:42, Robert Haas wrote: > On Tue, Aug 8, 2017 at 4:00 AM, Craig Ringer > wrote: > >> - When a standby connects to a master, it can optionally supply a list > >> of slot names that it cares about. > > > > Wouldn't that immediately e

Re: [HACKERS] Walsender timeouts and large transactions

2017-08-09 Thread Craig Ringer
; > + } > > + if (now <= TimestampTzPlusMilliseconds(last_reply_timestamp, > wal_sender_timeout / 2)) > > + return; > > + } > > > > If not, what problem prevents? > > We should do CHECK_FOR_INTERRUPTS() independently of

  1   2   3   4   5   6   7   8   9   10   >