Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Dilip Kumar
Hi Thomas, I have started reviewing 0003-Add-undo-log-manager, I haven't yet reviewed but some places I noticed that instead of UndoRecPtr you are directly using UndoLogOffset. Which seems like bugs to me 1. +UndoRecPtr +UndoLogAllocateInRecovery(UndoLogAllocContext *context, + TransactionId

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Thomas Munro
On Wed, Jul 24, 2019 at 9:15 PM Amit Kapila wrote: > I have done some more review of undolog patch series and here are my comments: Hi Amit, Thanks! There a number of actionable changes in your review. I'll be posting a new patch set soon that will address most of your complaints

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Kuntal Ghosh
Hello Thomas, Here are some review comments on 0003-Add-undo-log-manager.patch. I've tried to avoid duplicate comments as much as possible. 1. In UndoLogAllocate, + * time this backend as needed to write to an undo log at all or because s/as/has + * Maintain our tracking of the and the previous

Initdb failure

2019-07-24 Thread vignesh C
Hi, Initdb fails when following path is provided as input:

RE: [Patch] PQconnectPoll() is blocked if target_session_attrs is read-write

2019-07-24 Thread Tsunakawa, Takayuki
From: Matsumura, Ryo [mailto:matsumura@jp.fujitsu.com] > Detail: > If target_session_attrs is set to read-write, PQconnectPoll() calls > PQsendQuery("SHOW transaction_read_only") althogh previous return value > was PGRES_POLLING_READING not WRITING. The current code probably assumes that

Re: double free in ExecHashJoin, 9.6.12

2019-07-24 Thread Thomas Munro
On Thu, Jul 25, 2019 at 2:39 AM Merlin Moncure wrote: > Server is generally running pretty well, and is high volume. This > query is not new and is also medium volume. Database rebooted in > about 4 seconds with no damage; fast enough we didn't even trip alarms > (I noticed this troubleshooting

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread vignesh C
On Thu, Jul 25, 2019 at 7:48 AM Amit Kapila wrote: > > On Wed, Jul 24, 2019 at 11:04 PM vignesh C wrote: > > > > Hi, > > > > I have done some review of undolog patch series > > and here are my comments: > > 0003-Add-undo-log-manager.patch > > > > 1) As undo log is being created in tablespace, >

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-24 Thread Peter Geoghegan
On Wed, Jul 24, 2019 at 3:06 PM Peter Geoghegan wrote: > There seems to be a kind of "synergy" between the nbtsplitloc.c > handling of pages that have lots of duplicates and posting list > compression. It seems as if the former mechanism "sets up the bowling > pins", while the latter mechanism

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2019-07-24 Thread Noah Misch
On Wed, Jul 24, 2019 at 05:27:18PM +0900, Kyotaro Horiguchi wrote: > Sorry in advance for link-breaking message forced by gmail.. Using the archives page "Resend email" link avoids that. > https://www.postgresql.org/message-id/flat/20190202083822.gc32...@gust.leadboat.com > > > 1. The result of

Re: dropdb --force

2019-07-24 Thread Tom Lane
Pavel Stehule writes: > [ drop-database-force-20190708.patch ] I took a brief look at this, but I don't think it's really close to being committable. * The documentation claims FORCE will fail if you don't have privileges to terminate the other session(s) in the target DB. This is a lie; the

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Amit Kapila
On Wed, Jul 24, 2019 at 11:04 PM vignesh C wrote: > > Hi, > > I have done some review of undolog patch series > and here are my comments: > 0003-Add-undo-log-manager.patch > > 1) As undo log is being created in tablespace, > if the tablespace is dropped later, will it have any impact? > Yes, it

Re: benchmarking Flex practices

2019-07-24 Thread Tom Lane
Chapman Flack writes: > On 07/24/19 03:45, John Naylor wrote: >> On Sun, Jul 21, 2019 at 3:14 AM Tom Lane wrote: >>> However, my second reaction was that maybe you were on to something >>> upthread when you speculated about postponing de-escaping of >>> Unicode literals into the grammar. If we

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-24 Thread Kyotaro Horiguchi
I found that CF-bot complaining on this. Seems that some comment fixes by the recent 21039555cd are the cause. No substantial change have been made by this rebasing. regards. On Fri, Jul 12, 2019 at 5:37 PM Kyotaro Horiguchi wrote: > > At Fri, 12 Jul 2019 17:30:41 +0900 (Tokyo Standard Time),

[PATCH] Race condition in logical walsender causes long postgresql shutdown delay

2019-07-24 Thread Craig Ringer
Hi folks I recently tracked down a race in shutdown of logical walsenders that can cause PostgreSQL shutdown to hang for wal_sender_timeout/2 before it continues to a normal shutdown. With a long timeout that can be quite disruptive. TL;DR: The logical walsender may be signalled to stop, then

Re: Should we add xid_current() or a int8->xid cast?

2019-07-24 Thread Thomas Munro
On Thu, Jul 25, 2019 at 12:42 PM Tom Lane wrote: > Andres Freund writes: > > On 2019-07-24 20:34:39 -0400, Tom Lane wrote: > >> Yeah, I would absolutely NOT recommend that you open that can of worms > >> right now. We have looked at adding unsigned integer types in the past > >> and it looked

Re: On the stability of TAP tests for LDAP

2019-07-24 Thread Michael Paquier
On Wed, Jul 24, 2019 at 09:01:47PM +1200, Thomas Munro wrote: > Huh, yeah, I don't know why slapd requires credentials on Debian, when > the version that ships with FreeBSD is OK with an anonymous > connection. Rather than worrying about that, I just adjusted it to > supply the credentials. It

Re: Should we add xid_current() or a int8->xid cast?

2019-07-24 Thread Tom Lane
Andres Freund writes: > On 2019-07-24 20:34:39 -0400, Tom Lane wrote: >> Yeah, I would absolutely NOT recommend that you open that can of worms >> right now. We have looked at adding unsigned integer types in the past >> and it looked like a mess. > I assume Thomas was thinking more of another

Re: Should we add xid_current() or a int8->xid cast?

2019-07-24 Thread Andres Freund
Hi, On 2019-07-24 20:34:39 -0400, Tom Lane wrote: > Yeah, I would absolutely NOT recommend that you open that can of worms > right now. We have looked at adding unsigned integer types in the past > and it looked like a mess. I assume Thomas was thinking more of another bespoke type like xid,

Re: Should we add xid_current() or a int8->xid cast?

2019-07-24 Thread Tom Lane
Andres Freund writes: > On 2019-07-25 12:20:58 +1200, Thomas Munro wrote: >> On Thu, Jul 25, 2019 at 12:06 PM Andres Freund wrote: >>> Seems easiest to just add xid_current(), or add a cast from int8 to xid >>> (probably explicit?) that handles the wraparound logic correctly? >> Yeah, I was

Re: Should we add xid_current() or a int8->xid cast?

2019-07-24 Thread Andres Freund
Hi, On 2019-07-25 12:20:58 +1200, Thomas Munro wrote: > On Thu, Jul 25, 2019 at 12:06 PM Andres Freund wrote: > > we have txid_current(), which returns an int8. But there's no convenient > > way to convert that to type 'xid'. Which is fairly inconvenient, given > > that we expose xids in various

Re: Should we add xid_current() or a int8->xid cast?

2019-07-24 Thread Thomas Munro
On Thu, Jul 25, 2019 at 12:06 PM Andres Freund wrote: > we have txid_current(), which returns an int8. But there's no convenient > way to convert that to type 'xid'. Which is fairly inconvenient, given > that we expose xids in various places. > > My current need for this was just a regression

Re: ON CONFLICT (and manual row locks) cause xmax of updated tuple to unnecessarily be set

2019-07-24 Thread Peter Geoghegan
On Wed, Jul 24, 2019 at 4:24 PM Andres Freund wrote: > as you can see the same xmax is set for both row version, with the new > infomask being HEAP_XMAX_KEYSHR_LOCK | HEAP_XMAX_LOCK_ONLY | HEAP_UPDATED. Meta remark about your test case: I am a big fan of microbenchmarks like this, which execute

Should we add xid_current() or a int8->xid cast?

2019-07-24 Thread Andres Freund
Hi, we have txid_current(), which returns an int8. But there's no convenient way to convert that to type 'xid'. Which is fairly inconvenient, given that we expose xids in various places. My current need for this was just a regression test to make sure that system columns (xmin/xmax in

Re: Compile from source using latest Microsoft Windows SDK

2019-07-24 Thread Michael Paquier
On Wed, Jul 24, 2019 at 10:38:47AM -0400, Andrew Dunstan wrote: > Yeah, on consideration I think Peifeng's patch upthread looks OK. > (Incidentally, this variable is not set in the very old version of VC > running on currawong). Interesting. I am not actually sure in which version of VS this has

RE: seems like a bug in pgbench -R

2019-07-24 Thread Imai, Yoshikazu
On Wed, July 24, 2019 at 7:02 PM, Fabien COELHO wrote: > > but I have one question. Is it better adding any check like if(maxsock > > != -1) before the select? > > > > else/* no explicit delay, select without timeout */ > > { > >nsocks = select(maxsock + 1, _mask, NULL, NULL,

Re: [PATCH] minor bugfix for pg_basebackup (9.6 ~ )

2019-07-24 Thread Michael Paquier
On Wed, Jul 24, 2019 at 11:23:30AM -0400, Alvaro Herrera wrote: > Heh, yesterday I revised the original patch as attached and was about to > push when the bell rang. I like this one because it keeps the comment > to one line and it mentions the function name in charge of the > validation (useful

Re: Statistical aggregate functions are not working with PARTIAL aggregation

2019-07-24 Thread David Rowley
On Thu, 25 Jul 2019 at 11:33, Andres Freund wrote: > > On 2019-07-25 10:36:26 +1200, David Rowley wrote: > > 2) Planner trying to give nodeAgg.c a sorted path to work with on > > DISTINCT / ORDER BY aggs > > That'll have to be a best effort thing though, i.e. there'll always be > cases where

Re: Statistical aggregate functions are not working with PARTIAL aggregation

2019-07-24 Thread Andres Freund
Hi, On 2019-07-25 10:36:26 +1200, David Rowley wrote: > I'd like to do > much more in nodeAgg.c, TBH. It would be good to remove some > code from > nodeAgg.c and put it in the planner. Indeed! > I'd like to see: > > 1) Planner doing the Aggref merging for aggregates with the same > transfn

ON CONFLICT (and manual row locks) cause xmax of updated tuple to unnecessarily be set

2019-07-24 Thread Andres Freund
Hi, Scenario is a very plain upsert: CREATE TABLE upsert(key int primary key); INSERT INTO upsert VALUES(1) ON CONFLICT (key) DO UPDATE SET key = excluded.key; INSERT INTO upsert VALUES(1) ON CONFLICT (key) DO UPDATE SET key = excluded.key; INSERT 0 1 INSERT 0 1 postgres[8755][1]=# SELECT

Re: Statistical aggregate functions are not working with PARTIAL aggregation

2019-07-24 Thread David Rowley
On Thu, 25 Jul 2019 at 06:52, Andres Freund wrote: > Now that master is open for development, and you have a commit bit, are > you planning to go forward with this on your own? I plan to, but it's not a high priority at the moment. I'd like to do much more in nodeAgg.c, TBH. It would be good to

Re: pgbench - allow to create partitioned tables

2019-07-24 Thread Fabien COELHO
# and look at latency: # no parts = 0.071 ms # 1 hash = 0.071 ms (did someone optimize this case?!) # 2 hash ~ 0.126 ms (+ 0.055 ms) # 50 hash ~ 0.155 ms # 100 hash ~ 0.178 ms # 150 hash ~ 0.232 ms # 200 hash ~ 0.279 ms # overhead ~ (0.050 + [0.0005-0.0008] * nparts) ms

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-24 Thread Peter Geoghegan
On Tue, Jul 23, 2019 at 6:22 PM Peter Geoghegan wrote: > Attached is a revised version of your v2 that fixes this issue -- I'll > call this v3. Remember that index that I said was 5.5x smaller with the patch applied, following retail insertions (a single big INSERT ... SELECT ...)? Well, it's

Re: Built-in connection pooler

2019-07-24 Thread Ryan Lambert
Hello Konstantin, > Concerning testing: I do not think that connection pooler needs some kind of special tests. > The idea of built-in connection pooler is that it should be able to handle all requests normal postgres can do. > I have added to regression tests extra path with enabled connection

Re: Memory Accounting

2019-07-24 Thread Tomas Vondra
On Tue, Jul 23, 2019 at 06:18:26PM -0700, Melanie Plageman wrote: On Thu, Jul 18, 2019 at 11:24 AM Jeff Davis wrote: Previous discussion: https://postgr.es/m/1407012053.15301.53.camel%40jeff-desktop This patch introduces a way to ask a memory context how much memory it currently has

Re: Index Skip Scan

2019-07-24 Thread Dmitry Dolgov
> On Mon, Jul 22, 2019 at 7:10 PM Jesper Pedersen > wrote: > > On 7/22/19 1:44 AM, David Rowley wrote: > > Here are the comments I noted down during the review: > > > > cost_index: > > > > I know you've not finished here, but I think it'll need to adjust > > tuples_fetched somehow to account for

Re: initdb recommendations

2019-07-24 Thread Peter Eisentraut
On 2019-07-24 22:18, Tom Lane wrote: >> I think we could just define that if geteuid == getpeereid, then >> authentication succeeds. Possibly make that a setting if someone wants >> to turn it off. > > We would still need to make the proposed buildfarm changes, though, > because Windows. (And

Re: pg_upgrade version checking questions

2019-07-24 Thread Peter Eisentraut
On 2019-07-23 17:30, Daniel Gustafsson wrote: > The reason for moving is that we print default values in usage(), and that > requires the value to be computed before calling usage(). We already do this > for resolving environment values in parseCommandLine(). If we do it in setup, > then we’d

Re: Support for jsonpath .datetime() method

2019-07-24 Thread Peter Eisentraut
On 2019-07-24 00:48, Nikita Glukhov wrote: > It seems that our YY works like RR should: > > SELECT to_date('69', 'YY'); > to_date > > 2069-01-01 > (1 row) > > SELECT to_date('70', 'YY'); > to_date > > 1970-01-01 > (1 row) > > But by the standard first two

Re: initdb recommendations

2019-07-24 Thread Tom Lane
Peter Eisentraut writes: > If I'm logged in as the OS user that owns the data directory, I should > be able to log in to the database system via local socket as any user. > Because why stop me? I can just change pg_hba.conf to let me in. Hmm ... there's probably some minor loss of safety there,

Re: initdb recommendations

2019-07-24 Thread Peter Eisentraut
On 2019-07-22 19:40, Andres Freund wrote: > On 2019-07-22 13:02:13 -0400, Andrew Dunstan wrote: >> There are a few things we could do. We could force trust auth, or we >> could add an ident map that allowed $USER to login as buildfarm. Finding >> all the places we would need to fix that could be a

Re: initdb recommendations

2019-07-24 Thread Andrew Dunstan
On 7/24/19 10:00 AM, Andrew Dunstan wrote: > On 7/23/19 2:12 AM, Peter Eisentraut wrote: >> On 2019-07-22 21:16, Andrew Dunstan wrote: >>> Modulo this issue, experimentation shows that adding '-A trust' to the >>> line in run_build.pl where initdb is called fixes the issue. If we're >>> going to

Re: initdb recommendations

2019-07-24 Thread Peter Eisentraut
On 2019-07-24 16:00, Andrew Dunstan wrote: > I think we also need to change vcregress.pl to use trust explicitly for > upgrade checks, just like the Unix upgrade test script does. That should > help to future-proof us a bit. Right, I'll add that to my patch. -- Peter Eisentraut

Re: psql - add SHOW_ALL_RESULTS option

2019-07-24 Thread Fabien COELHO
Bonjour Daniel, I kind of agree as well, but I was pretty sure that someone would complain if the current behavior was changed. If queries in a compound statement must be kept silent, they can be converted to CTEs or DO-blocks to produce the same behavior without having to configure anything

RE: seems like a bug in pgbench -R

2019-07-24 Thread Fabien COELHO
Hello Yoshikazu, I could not reproduce this issue on head, but I confirm on 11.2. I could reproduce the stuck on 11.4. Attached is a fix to apply on pg11. I confirm the stuck doesn't happen after applying your patch. Ok, thanks for the feedback. + /* under throttling

add a MAC check for TRUNCATE

2019-07-24 Thread Yuli Khodorkovskiy
Hackers, Since all DAC checks should have corresponding MAC, this patch adds a hook to allow extensions to implement a MAC check on TRUNCATE. I have also implemented this access check in the sepgsql extension. One important thing to note is that refpolicy [1] and Redhat based distributions do

Re: Statistical aggregate functions are not working with PARTIAL aggregation

2019-07-24 Thread Andres Freund
Hi, On 2019-05-20 17:27:10 +1200, David Rowley wrote: > On Mon, 20 May 2019 at 13:20, Andres Freund wrote: > > How > > about we have something roughly like: > > > > int numTransFnArgs = -1; > > int numCombineFnArgs = -1; > > Oid

Re: Adding a test for speculative insert abort case

2019-07-24 Thread Andres Freund
Hi, On 2019-06-05 15:49:47 -0700, Melanie Plageman wrote: > On Thu, May 16, 2019 at 8:46 PM Melanie Plageman > wrote: > > > > > Good idea. > > I squashed the changes I suggested in previous emails, Ashwin's patch, my > > suggested updates to that patch, and the index order check all into one >

Re: GiST VACUUM

2019-07-24 Thread Peter Geoghegan
On Wed, Jul 24, 2019 at 11:33 AM Heikki Linnakangas wrote: > That's probably how it's going to go, but hey, doesn't hurt to ask :-). I think that it would be fine to be conservative with nbtree, and only target the master branch. The problem is annoying, certainly, but it's not likely to make a

Re: GiST VACUUM

2019-07-24 Thread Heikki Linnakangas
On 24/07/2019 21:02, Peter Geoghegan wrote: On Wed, Jul 24, 2019 at 10:30 AM Heikki Linnakangas wrote: Pushed this now, to master and REL_12_STABLE. Now, B-tree indexes still have the same problem, in all versions. Any volunteers to write a similar fix for B-trees? I was hoping that you'd

Re: GiST VACUUM

2019-07-24 Thread Peter Geoghegan
On Wed, Jul 24, 2019 at 10:30 AM Heikki Linnakangas wrote: > Pushed this now, to master and REL_12_STABLE. > > Now, B-tree indexes still have the same problem, in all versions. Any > volunteers to write a similar fix for B-trees? I was hoping that you'd work on it. :-) Any reason to think

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-24 Thread Tom Lane
David Rowley writes: > Here's a more polished version with the debug code removed, complete > with benchmarks. A few gripes: You're measuring the number of locks held at completion of the transaction, which fails to account for locks transiently taken and released, so that the actual peak usage

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread vignesh C
Hi, I have done some review of undolog patch series and here are my comments: 0003-Add-undo-log-manager.patch 1) As undo log is being created in tablespace, if the tablespace is dropped later, will it have any impact? +void +UndoLogDirectory(Oid tablespace, char *dir) +{ + if (tablespace ==

Re: Seek failure at end of FSM file during WAL replay (in 11)

2019-07-24 Thread Tom Lane
Michael Paquier writes: > Recently, one of the test beds we use has blown up once when doing > streaming replication like that: > FATAL: could not seek to end of file "base/16386/19817_fsm": No such >file or directory > CONTEXT: WAL redo at 60/8DA22448 for Heap2/CLEAN: remxid 65751197 >

Re: GiST VACUUM

2019-07-24 Thread Heikki Linnakangas
On 22/07/2019 16:09, Heikki Linnakangas wrote: Unless something comes up, I'll commit this tomorrow. Pushed this now, to master and REL_12_STABLE. Now, B-tree indexes still have the same problem, in all versions. Any volunteers to write a similar fix for B-trees? - Heikki

Re: "localtime" value in TimeZone

2019-07-24 Thread Tom Lane
Shay Rojansky writes: > In (certain) out-of-the-box PostgreSQL installations, the timezone GUC is > set to "localtime", which seems to mean to query the OS for the value. > Unless I'm mistaken, the issue with this is that it doesn't allow clients > inspecting the TimeZone GUC to actually know

Re: [PATCH] minor bugfix for pg_basebackup (9.6 ~ )

2019-07-24 Thread Alvaro Herrera
On 2019-Jul-24, Ian Barwick wrote: > It'd be better if such a hypothetical option validated the provided > slot name anwyay, to prevent later surprises. Hmm, but what would we do if the validation failed? > Revised patch attached, which as Alvaro suggests removes the escaping > and adds a

"localtime" value in TimeZone

2019-07-24 Thread Shay Rojansky
Greetings everyone. In (certain) out-of-the-box PostgreSQL installations, the timezone GUC is set to "localtime", which seems to mean to query the OS for the value. Unless I'm mistaken, the issue with this is that it doesn't allow clients inspecting the TimeZone GUC to actually know what timezone

Re: [bug fix] Produce a crash dump before main() on Windows

2019-07-24 Thread Alvaro Herrera
On 2019-Jul-24, Kyotaro Horiguchi wrote: > Hello. > > On Wed, Jul 24, 2019 at 2:31 AM Alvaro Herrera > wrote: > > > > On 2019-Jul-23, Tom Lane wrote: > > > > > Kyotaro Horiguchi writes: > > > > > > My investigation convinced me that there is no way for a process > > > > to detect wheter it is

Re: Compile from source using latest Microsoft Windows SDK

2019-07-24 Thread Andrew Dunstan
On 7/22/19 4:23 AM, Michael Paquier wrote: > On Mon, Jul 22, 2019 at 04:01:46PM +0800, Peifeng Qiu wrote: >>> but it's really only a major issue for VS2019 >> VS2019 will use the latest v10 SDK by default. So no need to install 8.1 >> for VS2019. > Yes, FWIW, I have tested with VS2019 when

double free in ExecHashJoin, 9.6.12

2019-07-24 Thread Merlin Moncure
Server is generally running pretty well, and is high volume. This query is not new and is also medium volume. Database rebooted in about 4 seconds with no damage; fast enough we didn't even trip alarms (I noticed this troubleshooting another issue). We are a couple of bug fixes releases behind

Re: initdb recommendations

2019-07-24 Thread Andrew Dunstan
On 7/23/19 2:12 AM, Peter Eisentraut wrote: > On 2019-07-22 21:16, Andrew Dunstan wrote: >> Modulo this issue, experimentation shows that adding '-A trust' to the >> line in run_build.pl where initdb is called fixes the issue. If we're >> going to rely on a buildfarm client fix that one seems

Re: Psql patch to show access methods info

2019-07-24 Thread Alexander Korotkov
On Wed, Jul 24, 2019 at 9:01 AM Andres Freund wrote: > Based on a quick skim of the thread - which means I most definitely > missed things - there's not been discussion of why we actually want to > add this. Who's the prospective user of this facility? And why wouldn't > they just query

Re: initdb recommendations

2019-07-24 Thread Andrew Dunstan
On 7/22/19 1:40 PM, Andres Freund wrote: > Hi, > > On 2019-07-22 13:02:13 -0400, Andrew Dunstan wrote: >> There are a few things we could do. We could force trust auth, or we >> could add an ident map that allowed $USER to login as buildfarm. Finding >> all the places we would need to fix that

Re: Psql patch to show access methods info

2019-07-24 Thread Alexander Korotkov
Hi! On Wed, Jul 24, 2019 at 9:00 AM Andres Freund wrote: > On 2019-07-23 01:57:29 +0300, Alexander Korotkov wrote: > > It was always scary there is no way in psql to see am/opclass/opfamily > > information rather than query catalog directly. > > What does make that scary? For it's unclear why

Re: Support for jsonpath .datetime() method

2019-07-24 Thread Alexander Korotkov
On Wed, Jul 24, 2019 at 1:50 AM Nikita Glukhov wrote: > So it's unclear what we should do: > - implement YY and RR strictly following the standard only in .datetime() > - fix YY implementation in to_date()/to_timestamp() and implement RR > - use our non-standard templates in .datetime() Also

Re: pgbench tests vs Windows

2019-07-24 Thread Andrew Dunstan
On 7/24/19 3:56 AM, Fabien COELHO wrote: > > Hello Andrew, > >> Unfortunately, this isn't portable, as I've just discovered at the cost >> of quite a bit of time. In particular, you can't assume expr is present >> and in the path on Windows. The Windows equivalent would be something >> like: >>

Re: pg_receivewal documentation

2019-07-24 Thread Jehan-Guillaume de Rorthais
Hi, On Wed, 24 Jul 2019 11:29:28 +0900 Michael Paquier wrote: > On Tue, Jul 23, 2019 at 08:00:41AM -0400, Jesper Pedersen wrote: > > Sure. > > Thanks. Applied down to 9.6 where remote_apply has been introduced, > with tweaks for 9.6 as the tool is named pg_receivexlog there. Sorry to step

Re: psql - add SHOW_ALL_RESULTS option

2019-07-24 Thread Daniel Verite
Fabien COELHO wrote: > >> I'd go further and suggest that there shouldn't be a variable > >> controlling this. All results that come in should be processed, period. > > > > I agree with that. > > I kind of agree as well, but I was pretty sure that someone would complain > if the current

Re: Issue in to_timestamp/to_date while handling the quoted literal string

2019-07-24 Thread Brendan Jurd
Hi Suraj, I think the documentation is reasonably clear about this behaviour, quote: " In to_date, to_number, and to_timestamp, literal text and double-quoted strings result in skipping the number of characters contained in the string; for example "XX" skips two input characters (whether or not

Re: Fetching timeline during recovery

2019-07-24 Thread Jehan-Guillaume de Rorthais
Hello Michael, On Wed, 24 Jul 2019 09:49:05 +0900 Michael Paquier wrote: > On Tue, Jul 23, 2019 at 06:05:18PM +0200, Jehan-Guillaume de Rorthais wrote: > > Please, find in attachment a first trivial patch to support > > pg_walfile_name() and pg_walfile_name_offset() on a standby. > > Previous

Re: benchmarking Flex practices

2019-07-24 Thread Chapman Flack
On 07/24/19 03:45, John Naylor wrote: > On Sun, Jul 21, 2019 at 3:14 AM Tom Lane wrote: >> However, my second reaction was that maybe you were on to something >> upthread when you speculated about postponing de-escaping of >> Unicode literals into the grammar. If we did it like that then Wow,

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Amit Kapila
On Wed, Jul 24, 2019 at 2:45 PM Amit Kapila wrote: > > On Thu, Jul 18, 2019 at 5:10 PM Amit Kapila wrote: > > 7. > +attach_undo_log(UndoLogCategory category, Oid tablespace) > { > .. > if (candidate->meta.tablespace == tablespace) > + { > + logno = *place; > + slot = candidate; > + *place =

Re: pg_receivewal documentation

2019-07-24 Thread Jesper Pedersen
Hi, On 7/23/19 10:29 PM, Michael Paquier wrote: Thanks. Applied down to 9.6 where remote_apply has been introduced, with tweaks for 9.6 as the tool is named pg_receivexlog there. Thanks to everybody involved ! Best regards, Jesper

Issue in to_timestamp/to_date while handling the quoted literal string

2019-07-24 Thread Suraj Kharage
Hi, I noticed the issue in to_timestamp()/to_date() while handling the double quote literal string. If any double quote literal characters found in format, we generate the NODE_TYPE_CHAR in parse format and store that actual character in FormatNode->character. n DCH_from_char, we just increment

RE: seems like a bug in pgbench -R

2019-07-24 Thread Imai, Yoshikazu
Hi Fabien, On Fri, Mar 15, 2019 at 4:17 PM, Fabien COELHO wrote: > >> echo 'select 1' > select.sql > >> > >> while /bin/true; do > >> pgbench -n -f select.sql -R 1000 -j 8 -c 8 -T 1 > /dev/null 2>&1; > >> date; > >> done; > > > > Indeed. I'll look at it over the weekend. > > > >> So I guess

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Amit Kapila
On Wed, Jul 24, 2019 at 2:45 PM Amit Kapila wrote: > > On Thu, Jul 18, 2019 at 5:10 PM Amit Kapila wrote: > > > > On Mon, Jul 1, 2019 at 1:24 PM Thomas Munro wrote: > > > Yep, that was completely wrong. Here's a new version. > > 10. > I think UndoLogAllocate can leak allocation of slots. It

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Amit Kapila
On Thu, Jul 18, 2019 at 5:10 PM Amit Kapila wrote: > > On Mon, Jul 1, 2019 at 1:24 PM Thomas Munro wrote: > > > > On Fri, Jun 28, 2019 at 6:09 AM Robert Haas wrote: > > > I happened to open up 0001 from this series, which is from Thomas, and > > > I do not think that the pg_buffercache changes

Re: On the stability of TAP tests for LDAP

2019-07-24 Thread Thomas Munro
On Wed, Jul 24, 2019 at 7:50 PM Michael Paquier wrote: > Perhaps this worked on freebsd? Now that I test it, the test gets > stuck on my Debian box: > # waiting for slapd to accept requests... > # Running: ldapsearch -h localhost -p 49534 -s base -b > dc=example,dc=net -n 'objectclass=*' >

Re: [bug fix] Produce a crash dump before main() on Windows

2019-07-24 Thread Kyotaro Horiguchi
Hello. On Wed, Jul 24, 2019 at 2:31 AM Alvaro Herrera wrote: > > On 2019-Jul-23, Tom Lane wrote: > > > Kyotaro Horiguchi writes: > > > > My investigation convinced me that there is no way for a process > > > to detect wheter it is running as a service (except the process > > > directly called

Re: [PATCH] Improve performance of NOTIFY over many databases (issue blocking on AccessExclusiveLock on object 0 of class 1262 of database 0)

2019-07-24 Thread Martijn van Oosterhout
On Tue, 23 Jul 2019 at 23:32, Tom Lane wrote: > > Martijn van Oosterhout writes: > > I mean tracking the listening backends specifically, so you can > > replace the loops: > > for (i=0; i < MaxBackends; i++) > > with > > for (i=QUEUE_FIRST_LISTENING_BACKEND; i; i = > >

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2019-07-24 Thread Kyotaro Horiguchi
Sorry in advance for link-breaking message forced by gmail.. https://www.postgresql.org/message-id/flat/20190202083822.gc32...@gust.leadboat.com > 1. The result of the test is valid only until we release the SLRU ControlLock, >which we do before SlruScanDirCbDeleteCutoff() uses the cutoff to

Re: pgbench - allow to create partitioned tables

2019-07-24 Thread Fabien COELHO
Hello Simon, While doing some performance tests and reviewing patches, I needed to create partitioned tables. Given the current syntax this is time consumming. Good idea. I wonder why we didn't have it already. Probably because I did not have to create partitioned table for some

Re: pgbench tests vs Windows

2019-07-24 Thread Fabien COELHO
Hello Andrew, Unfortunately, this isn't portable, as I've just discovered at the cost of quite a bit of time. In particular, you can't assume expr is present and in the path on Windows. The Windows equivalent would be something like: \setshell two\   @set /a c = 1 + :one  && echo %c%

Re: On the stability of TAP tests for LDAP

2019-07-24 Thread Michael Paquier
On Wed, Jul 24, 2019 at 05:47:13PM +1200, Thomas Munro wrote: > Thanks, here's v2. Perhaps this worked on freebsd? Now that I test it, the test gets stuck on my Debian box: # waiting for slapd to accept requests... # Running: ldapsearch -h localhost -p 49534 -s base -b dc=example,dc=net -n

Re: benchmarking Flex practices

2019-07-24 Thread John Naylor
On Sun, Jul 21, 2019 at 3:14 AM Tom Lane wrote: > > John Naylor writes: > > The pre-existing ecpg var "state_before" was a bit confusing when > > combined with the new var "state_before_quote_stop", and the former is > > also used with C-comments, so I decided to go with > >

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-24 Thread David Rowley
On Wed, 24 Jul 2019 at 16:16, David Rowley wrote: > > On Wed, 24 Jul 2019 at 15:05, David Rowley > wrote: > > To be able to reduce the threshold down again we'd need to make a > > hash_get_num_entries(LockMethodLocalHash) call before performing the > > guts of LockReleaseAll(). We could then

RE: [PATCH] Fix Proposal - Deadlock Issue in Single User Mode When IO Failure Occurs

2019-07-24 Thread Kyotaro Horiguchi
Sorry in advance for link-breaking message force by gmail.. https://www.postgresql.org/message-id/flat/cy4pr2101mb0804ce9836e582c0702214e8aa...@cy4pr2101mb0804.namprd21.prod.outlook.com I assume that we are in a consensus about the problem we are to fix here. > 0a 0004`8080cc30

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Dilip Kumar
On Wed, Jul 24, 2019 at 11:28 AM Rushabh Lathia wrote: > > Hi, > > I have stated review of > 0008-Provide-interfaces-to-store-and-fetch-undo-records.patch, here are few > quick comments. Thanks for the review, I will work on them soon and post the updated patch along with other comments. I have

Seek failure at end of FSM file during WAL replay (in 11)

2019-07-24 Thread Michael Paquier
Hi all, Recently, one of the test beds we use has blown up once when doing streaming replication like that: FATAL: could not seek to end of file "base/16386/19817_fsm": No such file or directory CONTEXT: WAL redo at 60/8DA22448 for Heap2/CLEAN: remxid 65751197 LOG: startup process (PID

Re: Problem during Windows service start

2019-07-24 Thread Kyotaro Horiguchi
Sorry in advance for link-breaking message, but the original mail was too old and gmail doesn't allow me to craft required headers to link to it. https://www.postgresql.org/message-id/CAKm4Xs71Ma8bV1fY6Gfz9Mg3AKmiHuoJNpxeDVF_KTVOKoy1WQ%40mail.gmail.com > Please find the proposed patch for

Re: pg_basebackup delays closing of stdout

2019-07-24 Thread Andres Freund
Hi, On 2019-07-23 22:16:26 -0400, Jeff Janes wrote: > Ever since pg_basebackup was created, it had a comment like this: > > * End of chunk. If requested, and this is the base tablespace > * write configuration file into the tarfile. When done, close the > * file (but not stdout).

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Andres Freund
On 2019-07-22 14:21:36 +0530, Amit Kapila wrote: > Another thing is changing subxids to fxids can increase the size of > two-phase file for a xact having many sub-transactions which again > might be okay, but not completely sure. I can't see that being a problem.

Re: Psql patch to show access methods info

2019-07-24 Thread Andres Freund
Hi, On 2019-07-15 22:03:31 +0300, Nikita Glukhov wrote: > + > + > + \dAc[+] > +[ class="parameter">access-method-pattern > + [ class="parameter">input-type-pattern]] > + > + > + > + > +Shows info index

Re: initdb recommendations

2019-07-24 Thread Andres Freund
Hi, On 2019-07-22 13:02:13 -0400, Andrew Dunstan wrote: > There are a few things we could do. We could force trust auth, or we > could add an ident map that allowed $USER to login as buildfarm. Finding > all the places we would need to fix that could be a fun project ... Perhaps we could

Re: ANALYZE: ERROR: tuple already updated by self

2019-07-24 Thread Andres Freund
Hi, On 2019-06-18 17:08:37 -0700, Andres Freund wrote: > On 2019-06-18 18:48:58 -0500, Justin Pryzby wrote: > > Ah: the table is an inheritence parent. If I uninherit its child, there's > > no > > error during ANALYZE. MV stats on the child are ok: > > It's a "classical" inheritance parent,

Re: Psql patch to show access methods info

2019-07-24 Thread Andres Freund
Hi, On 2019-07-23 01:57:29 +0300, Alexander Korotkov wrote: > It was always scary there is no way in psql to see am/opclass/opfamily > information rather than query catalog directly. What does make that scary? > I'm going to push it. Probably, someone find that commands syntax and > output

Re: Psql patch to show access methods info

2019-07-24 Thread Andres Freund
Hi, On 2019-07-15 22:03:31 +0300, Nikita Glukhov wrote: > + > + > + \dAc[+] > +[ class="parameter">access-method-pattern > + [ class="parameter">input-type-pattern]] > + > + > + > + > +Shows info index

Re: Change atoi to strtol in same place

2019-07-24 Thread Andres Freund
On 2019-07-24 16:57:42 +1200, David Rowley wrote: > On Wed, 24 Jul 2019 at 16:02, Joe Nelson wrote: > > > In general, I think it's a good idea to fix those places, but I wonder > > > if we need to change the error messages too. > > > > I'll leave that decision for the community to debate. I did,

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-24 Thread Andres Freund
Hi, On 2019-07-21 21:37:28 +1200, David Rowley wrote: > select.sql: > \set p 1 > select * from ht where a = :p > > Master: > > $ pgbench -n -f select.sql -T 60 -M prepared postgres > tps = 10172.035036 (excluding connections establishing) > tps = 10192.780529 (excluding connections