Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-20 Thread Kyotaro Horiguchi
Hello. I looked through the latest patch. At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi wrote in > At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote in > > - When reusing an index build, instead of storing the dropped relid in the > > IndexStmt and opening the dropped

Re: Autovacuum on partitioned table

2020-02-20 Thread Masahiko Sawada
On Fri, 21 Feb 2020 at 15:14, yuzuko wrote: > > Hello Amit-san, > > Thanks for your comments. > > > * White-space noise in the diff (space used where tab is expected); > > please check with git diff --check and fix. > Fixed it. > > > * Names changes_tuples, m_changes_tuples should be

Re: [Proposal] Global temporary tables

2020-02-20 Thread Prabhat Sahu
Hi, I have started testing the "Global temporary table" feature, from "gtt_v11-pg13.patch". Below is my findings: -- session 1: postgres=# create global temporary table gtt1(a int); CREATE TABLE -- seeeion 2: postgres=# truncate gtt1 ; ERROR: could not open file "base/13585/t3_16384": No such

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-20 Thread Michael Paquier
On Thu, Feb 20, 2020 at 05:38:15PM +0100, Bernd Helmle wrote: > So i propose a different approach like the attached patch tries to > implement: instead of just blindly iterating over directory contents > and filter them out, reference the tablespace location and > TABLESPACE_VERSION_DIRECTORY

Re: PATCH: Add uri percent-encoding for binary data

2020-02-20 Thread Anders Åstrand
Thanks for keeping this alive even though I disappeared after submitting it! I can write documentation this weekend. Thanks again. //Anders On Thu, 20 Feb 2020, 23:28 Alvaro Herrera, wrote: > On 2019-Oct-07, Anders Åstrand wrote: > > > Attached is a patch for adding uri as an encoding option

Re: Add PGURI env var for passing connection string to psql in Docker

2020-02-20 Thread Craig Ringer
On Fri, 21 Feb 2020 at 08:03, Michael Leonhard wrote: > 1. parse a perfectly good URI You have a URI with embedded password, which to me is not a perfectly good URI at all. I think the problem really lies with the input: separate your secret credentials out to start with, don't munge them into a

Re: Autovacuum on partitioned table

2020-02-20 Thread yuzuko
Hello Amit-san, Thanks for your comments. > * White-space noise in the diff (space used where tab is expected); > please check with git diff --check and fix. Fixed it. > * Names changes_tuples, m_changes_tuples should be changed_tuples and > m_changed_tuples, respectively? Yes, I modified it.

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-20 Thread Michael Paquier
On Thu, Feb 20, 2020 at 07:37:15AM -0600, David Steele wrote: > But since the name includes the backend's pid you would need to get lucky > and have a new backend with the same pid create the file after a restart. I > tried it and the old temp file was left behind after restart and first >

Re: pgsql: Add kqueue(2) support to the WaitEventSet API.

2020-02-20 Thread Tom Lane
I wrote: > Clearly, we gotta do something about that too. Maybe bumping up > NUM_RESERVED_FDS would be a good idea, but I feel like more-honest > accounting for permanently-eaten FDs would be a better idea. And > in any case we can't suppose that there's a fixed upper limit on > the number of

Re: client-side fsync() error handling

2020-02-20 Thread Michael Paquier
On Thu, Feb 20, 2020 at 10:10:11AM +0100, Peter Eisentraut wrote: > OK, added in new patch. Thanks, that looks good. > The frontends do neither right now, or at least the error handling is very > inconsistent and inscrutable. It would be possible in theory to add a retry > option, but that

Re: allow running parts of src/tools/msvc/ under not Windows

2020-02-20 Thread Michael Paquier
On Thu, Feb 20, 2020 at 09:31:32AM -0500, Tom Lane wrote: > Peter Eisentraut writes: >> The main benefit is that if you make "blind" edits in the Perl files, >> you can verify them easily, first by seeing that the Perl code runs, >> second, depending on the circumstances, by diffing the created

Re: bad wal on replica / incorrect resource manager data checksum in record / zfs

2020-02-20 Thread Amit Kapila
On Thu, Feb 20, 2020 at 7:40 PM Alex Malek wrote: > > On Thu, Feb 20, 2020, 6:16 AM Amit Kapila wrote: >> >> On Thu, Feb 20, 2020 at 3:06 AM Alex Malek wrote: >> > >> > Some interesting data points while debugging: >> > We had lowered the ZFS recordsize from 128K to 32K and for that week the

Re: Clean up some old cruft related to Windows

2020-02-20 Thread Michael Paquier
On Thu, Feb 20, 2020 at 12:39:56PM +0100, Juan José Santamaría Flecha wrote: > Actually, you can still use the vcvars% scripts to configure architecture, > platform_type and winsdk_version with current VS [1]. We still support the build down to MSVC 2013, so I think that it is good to mention the

Re: Portal->commandTag as an enum

2020-02-20 Thread John Naylor
On Thu, Feb 20, 2020 at 5:16 AM Mark Dilger wrote: > > > On Feb 19, 2020, at 3:31 AM, John Naylor > > wrote: > > thought it through. For one advantage, I think it might be nicer to > > have indexing.dat and toasting.dat instead of having to dig the info > > out of C macros. This was evident

Re: custom postgres launcher for tests

2020-02-20 Thread Craig Ringer
On Tue, 11 Feb 2020 at 19:33, Ivan Taranov wrote: > > This patch allow to use custom postgres launcher for tests (tap) > by setting environment variable PGLAUNCHER. I thought I saw a related patch to this that proposed to add a pg_ctl argument. Was that you too? I can't find it at the moment.

Re: Memory-Bounded Hash Aggregation

2020-02-20 Thread Jeff Davis
On Wed, 2020-02-19 at 20:16 +0100, Tomas Vondra wrote: > 1) explain.c currently does this: > > I wonder if we could show something for plain explain (without > analyze). > At least the initial estimate of partitions, etc. I know not showing > those details until after execution is what e.g. sort

Re: Protocol problem with GSSAPI encryption?

2020-02-20 Thread Andrew Gierth
> "Stephen" == Stephen Frost writes: >> I figure something along these lines for the fix. Anyone in a >> position to test this? Stephen> At least at first blush, I tend to agree with your analysis Stephen> and patch. Stephen> I'll see about getting this actually set up and tested in

Re: Add PGURI env var for passing connection string to psql in Docker

2020-02-20 Thread Michael Leonhard
Hi Tom, Thanks for your reply. A new PGURI env var would have the same security risks as the existing PGPASSWORD env var, but no more. It would be a usability improvement for folks using Docker. Docker provides some special security benefits. I believe that we can improve security for users by

Re: pgsql: Add kqueue(2) support to the WaitEventSet API.

2020-02-20 Thread Tom Lane
I wrote: > Thomas Munro writes: >> ... like coypu, where NUM_RESERVED_FDS is >> the only thing ensuring we have some spare fds. I don't know the >> history but it looks like NUM_RESERVED_FDS was deliberately or >> accidentally tuned to be just enough to be able to run the regression >> tests

Re: Improve heavyweight locks instead of building new lock managers?

2020-02-20 Thread Thomas Munro
On Thu, Feb 20, 2020 at 5:14 PM Andres Freund wrote: > 16 files changed, 569 insertions(+), 1053 deletions(-) Nice! Some comments on 0001, 0003, 0004: > Subject: [PATCH v1 1/6] Add additional prev/next & detached node functions to +extern void dlist_check(const dlist_head *head); +extern

Re: PATCH: Add uri percent-encoding for binary data

2020-02-20 Thread Alvaro Herrera
On 2019-Oct-07, Anders Åstrand wrote: > Attached is a patch for adding uri as an encoding option for > encode/decode. It uses what's called "percent-encoding" in rfc3986 > (https://tools.ietf.org/html/rfc3986#section-2.1). Thanks. Seems useful. I made a few cosmetic tweaks and it looks almost

Re: pgsql: Add kqueue(2) support to the WaitEventSet API.

2020-02-20 Thread Tom Lane
Thomas Munro writes: > One thing I've been planning to do for 13 is to get rid of all the > temporary create/destroy WaitEventSets from the main backend loops. > My goal was cutting down on stupid system calls, but this puts a new > spin on it. I have a patch set to do a bunch of that[1], but

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2020-02-20 Thread Peter Geoghegan
On Thu, Feb 20, 2020 at 10:58 AM Peter Geoghegan wrote: > I think that there is a good chance that it just won't matter. The > number of indexes that won't be able to support deduplication will be > very small in practice. The important exceptions are INCLUDE indexes > and nondeterministic

Re: pgsql: Add kqueue(2) support to the WaitEventSet API.

2020-02-20 Thread Thomas Munro
On Fri, Feb 21, 2020 at 8:56 AM Tom Lane wrote: > I wrote: > > It seems fairly obvious now that I look at it, but: the epoll and kqueue > > variants of CreateWaitEventSet are both *fundamentally* unsafe, because > > they assume that they can always get a FD when they want one, which is > > not a

Re: Add PGURI env var for passing connection string to psql in Docker

2020-02-20 Thread Tom Lane
Michael Leonhard writes: > I need to pass a connection string to psql inside Docker [2]. I can > pass it as a process argument, but this exposes the password to other > processes on my machine: > $ docker run --rm -i -t postgres:11 psql "$(cat db_uri)" Yeah, if you include the password in the

Add PGURI env var for passing connection string to psql in Docker

2020-02-20 Thread Michael Leonhard
Hi PostgreSQL Hackers, Please forgive me if this is not the preferred place to suggest a new feature. I found that a lot of items in the psql TODO list [1] were posted to this email list. I need to pass a connection string to psql inside Docker [2]. I can pass it as a process argument, but this

Open Source Hackathon Mentorship Invitation

2020-02-20 Thread Misha Patel
Hello, My name is Misha Patel and I’m reaching out on behalf of the HackIllinois Outreach team. HackIllinois is a 36-hour collegiate Open Source hackathon that takes place annually at the University of Illinois Urbana-Champaign. This year, it will be from February 28th-March 1st, 2020. Our

Re: pgsql: Add kqueue(2) support to the WaitEventSet API.

2020-02-20 Thread Tom Lane
I wrote: > It seems fairly obvious now that I look at it, but: the epoll and kqueue > variants of CreateWaitEventSet are both *fundamentally* unsafe, because > they assume that they can always get a FD when they want one, which is > not a property that we generally want backend code to have. The

Re: pgsql: Add kqueue(2) support to the WaitEventSet API.

2020-02-20 Thread Tom Lane
[ redirecting to -hackers ] I wrote: > =?utf-8?Q?R=C3=A9mi_Zara?= writes: > Le 20 févr. 2020 à 12:15, Thomas Munro a écrit : >>> Remi, any chance you could run gmake installcheck under >>> contrib/postgres_fdw on that host, to see if this is repeatable? Can >>> you tell us about the relevant

Re: plan cache overhead on plpgsql expression

2020-02-20 Thread Pavel Stehule
st 19. 2. 2020 v 8:09 odesílatel Amit Langote napsal: > On Wed, Feb 19, 2020 at 3:56 PM Amit Langote > wrote: > > On Wed, Feb 19, 2020 at 3:38 PM Pavel Stehule > wrote: > > > st 19. 2. 2020 v 7:30 odesílatel Pavel Stehule < > pavel.steh...@gmail.com> napsal: > > >> út 18. 2. 2020 v 17:08

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2020-02-20 Thread Peter Geoghegan
On Thu, Feb 20, 2020 at 7:38 AM Anastasia Lubennikova wrote: > I don't think this patch really needs more nitpicking ) But when has that ever stopped it? :-) > User can discover this with a complex query to pg_index and pg_opclass. > To simplify this, we can probably wrap this into function

Re: Bug in pg_restore with EventTrigger in parallel mode

2020-02-20 Thread Fabrízio de Royes Mello
On Thu, Feb 20, 2020 at 4:52 AM Michael Paquier wrote: > > That sounds right, as event triggers could interact with GRANT and > REFRESH of matviews, so they should be logically last. Looking at the > recent commit history, this would be similar to 3eb9a5e as we don't > really have a way to treat

Re: error context for vacuum to include block number

2020-02-20 Thread Tom Lane
Alvaro Herrera writes: > Another point is that this patch seems to be leaking memory each time > you set relation/index/namespace name, since you never free those and > they are changed over and over. One other point is that this code seems to be trying to ensure that the error context callback

Removing obsolete configure checks

2020-02-20 Thread Tom Lane
Over in the thread at [1] we agreed to remove assorted code that copes with missing , on the grounds that C99 requires that header so we should not have to cater anymore for platforms without it. This logic could obviously be carried further. I scraped the buildfarm configure logs to see what

Re: error context for vacuum to include block number

2020-02-20 Thread Alvaro Herrera
This is, by far, the most complex error context callback we've tried to write ... Easy stuff first: In the error context function itself, you don't need the _() around the strings: errcontext() is marked as a gettext trigger and it does the translation itself, so the manually added _() is just

Fwd: bad wal on replica / incorrect resource manager data checksum in record / zfs

2020-02-20 Thread Alex Malek
On Thu, Feb 20, 2020, 6:16 AM Amit Kapila wrote: > On Thu, Feb 20, 2020 at 3:06 AM Alex Malek wrote: > > > > > > Hello Postgres Hackers - > > > > We are having a reoccurring issue on 2 of our replicas where replication > stops due to this message: > > "incorrect resource manager data checksum

Re: Parallel copy

2020-02-20 Thread David Fetter
On Thu, Feb 20, 2020 at 02:36:02PM +0100, Tomas Vondra wrote: > On Thu, Feb 20, 2020 at 04:11:39PM +0530, Amit Kapila wrote: > > On Thu, Feb 20, 2020 at 5:12 AM David Fetter wrote: > > > > > > On Fri, Feb 14, 2020 at 01:41:54PM +0530, Amit Kapila wrote: > > > > This work is to parallelize the

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-20 Thread Bernd Helmle
Am Dienstag, den 18.02.2020, 15:15 +0900 schrieb Michael Paquier: > Fair point. Now, while the proposed patch is right to use > TABLESPACE_VERSION_DIRECTORY, shouldn't we use strncmp based on the > length of TABLESPACE_VERSION_DIRECTORY instead of de->d_name? It > seems also to me that the code

Re: Fix compiler warnings on 64-bit Windows

2020-02-20 Thread Tom Lane
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= writes: > On Mon, Feb 17, 2020 at 4:52 PM Tom Lane wrote: >> Anyway, I'll have a go at updating gaur to use 4.5.x. There is a >> sane-looking stdint.h on my second-oldest dinosaur, prairiedog. >> Don't know about the situation on Windows,

Re: WAL usage calculation patch

2020-02-20 Thread Kirill Bychik
> вт, 18 февр. 2020 г. в 06:23, Thomas Munro : > > On Mon, Feb 10, 2020 at 8:20 PM Craig Ringer wrote: > > > On Wed, 5 Feb 2020 at 21:36, Kirill Bychik > > > wrote: > > > > Patch is separated in two parts: core changes and pg_stat_statements > > > > additions. Essentially the extension has its

Re: Disallow cancellation of waiting for synchronous replication

2020-02-20 Thread Michail Nikolaev
Hello. Just want to share some thoughts about how it looks from perspective of a high availability web-service application developer. Because sometimes things look different from other sides. And everything looks like disaster to be honest. But let's take it one at a time. First - the problem

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2020-02-20 Thread Anastasia Lubennikova
On 19.02.2020 22:16, Peter Geoghegan wrote: On Wed, Feb 19, 2020 at 8:14 AM Anastasia Lubennikova wrote: Thank you for this work. I've looked through the patches and they seem to be ready for commit. I haven't yet read recent documentation and readme changes, so maybe I'll send some more

Re: allow running parts of src/tools/msvc/ under not Windows

2020-02-20 Thread Tom Lane
Peter Eisentraut writes: > On 2020-02-13 16:36, Tom Lane wrote: >> Yeah, I'm wondering exactly how this helps. IME the typical sort of >> breakage is "the MSVC build doesn't know that file X needs to be >> included when building Y". It seems like just building the project >> files will teach

Re: Minor improvement to partition_bounds_copy()

2020-02-20 Thread Julien Rouhaud
On Thu, Feb 20, 2020 at 09:38:26PM +0900, Amit Langote wrote: > Fujita-san, > > On Thu, Feb 20, 2020 at 8:36 PM Etsuro Fujita wrote: > > partition_bounds_copy() sets the hash_part and natts variable in each > > iteration of a loop to copy the datums in the datums array, which > > would not be

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-20 Thread David Steele
On 2/20/20 12:55 AM, Michael Paquier wrote: On Wed, Feb 19, 2020 at 12:37:00PM -0600, David Steele wrote: As far as I can see, the pg_internal.init.XX will not be cleaned up by PostgreSQL on startup. I've only tested this in 9.6 so far, but I don't see any differences in the code since then

Re: Parallel copy

2020-02-20 Thread Tomas Vondra
On Thu, Feb 20, 2020 at 04:11:39PM +0530, Amit Kapila wrote: On Thu, Feb 20, 2020 at 5:12 AM David Fetter wrote: On Fri, Feb 14, 2020 at 01:41:54PM +0530, Amit Kapila wrote: > This work is to parallelize the copy command and in particular "Copy > from 'filename' Where ;" command. Apropos of

Re: Minor improvement to partition_bounds_copy()

2020-02-20 Thread Amit Langote
Fujita-san, On Thu, Feb 20, 2020 at 8:36 PM Etsuro Fujita wrote: > partition_bounds_copy() sets the hash_part and natts variable in each > iteration of a loop to copy the datums in the datums array, which > would not be efficient. Attached is small patch for avoiding that. That looks good to

Re: Experimenting with transactional memory for SERIALIZABLE

2020-02-20 Thread Thomas Munro
On Thu, Feb 20, 2020 at 11:38 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > Can you share some numbers about how not well it perform and how many > hardware transactions were aborted with a fallback? I'm curious because > from this paper [1] I've got an impression that the bigger (in terms of

Re: Clean up some old cruft related to Windows

2020-02-20 Thread Juan José Santamaría Flecha
On Thu, Feb 20, 2020 at 4:23 AM Michael Paquier wrote: > > + You can change certain build options, such as the targeted CPU > + architecture, build type, and the selection of the SDK by using either > + VSVARS32.bat or VsDevCmd.bat > depending > + on your Visual Studio release. All commands

Minor improvement to partition_bounds_copy()

2020-02-20 Thread Etsuro Fujita
partition_bounds_copy() sets the hash_part and natts variable in each iteration of a loop to copy the datums in the datums array, which would not be efficient. Attached is small patch for avoiding that. Best regards, Etsuro Fujita partition_bounds_copy.patch Description: Binary data

Re: bad wal on replica / incorrect resource manager data checksum in record / zfs

2020-02-20 Thread Amit Kapila
On Thu, Feb 20, 2020 at 3:06 AM Alex Malek wrote: > > > Hello Postgres Hackers - > > We are having a reoccurring issue on 2 of our replicas where replication > stops due to this message: > "incorrect resource manager data checksum in record at ..." > This has been occurring on average once every

Re: Add PostgreSQL home page to --help output

2020-02-20 Thread Daniel Gustafsson
> On 20 Feb 2020, at 10:53, Daniel Gustafsson wrote: > >> On 20 Feb 2020, at 10:15, Peter Eisentraut >> wrote: >> >> On 2020-02-13 14:24, Greg Stark wrote: >>> Sounds like a fine idea. But personally I would prefer it without the <> >>> around the it, just a url on a line by itself. I think

Re: Parallel copy

2020-02-20 Thread Amit Kapila
On Thu, Feb 20, 2020 at 5:12 AM David Fetter wrote: > > On Fri, Feb 14, 2020 at 01:41:54PM +0530, Amit Kapila wrote: > > This work is to parallelize the copy command and in particular "Copy > > from 'filename' Where ;" command. > > Apropos of the initial parsing issue generally, there's an

Re: Experimenting with transactional memory for SERIALIZABLE

2020-02-20 Thread Dmitry Dolgov
> On Thu, Feb 20, 2020 at 04:55:12PM +1300, Thomas Munro wrote: > Hello hackers, > > Here's a *highly* experimental patch set that tries to skip the LWLock > protocol in predicate.c and use HTM[1] instead. HTM is itself a sort > of hardware-level implementation of SSI for shared memory. My >

Re: Add PostgreSQL home page to --help output

2020-02-20 Thread Daniel Gustafsson
> On 20 Feb 2020, at 10:15, Peter Eisentraut > wrote: > > On 2020-02-13 14:24, Greg Stark wrote: >> Sounds like a fine idea. But personally I would prefer it without the <> >> around the it, just a url on a line by itself. I think it would be clearer, >> look cleaner, and be easier to select

RE: [PoC] Non-volatile WAL buffer

2020-02-20 Thread Takashi Menjo
Dear Amit, Thank you for your advice. Exactly, it's so to speak "do as the hackers do when in pgsql"... I'm rebasing my branch onto master. I'll submit an updated patchset and performance report later. Best regards, Takashi -- Takashi Menjo NTT Software Innovation Center > -Original

Re: Autovacuum on partitioned table

2020-02-20 Thread Amit Langote
On Thu, Feb 20, 2020 at 5:32 PM Amit Langote wrote: > On Thu, Feb 20, 2020 at 4:50 PM Amit Langote wrote: > > * I may be missing something, but why doesn't do_autovacuum() fetch a > > partitioned table's entry from pgstat instead of fetching that for > > individual children and adding? That is,

Re: Add PostgreSQL home page to --help output

2020-02-20 Thread Peter Eisentraut
On 2020-02-13 14:24, Greg Stark wrote: Sounds like a fine idea. But personally I would prefer it without the <> around the it, just a url on a line by itself. I think it would be clearer, look cleaner, and be easier to select to copy/paste elsewhere. I'm on the fence about this one, but I

Re: client-side fsync() error handling

2020-02-20 Thread Peter Eisentraut
On 2020-02-13 12:52, Michael Paquier wrote: durable_rename() calls fsync_fname(), so it would be covered by this change. The other file access calls in there can be handled by normal error handling, I think. Is there any specific scenario you have in mind? The old file flush is handled by

Re: Autovacuum on partitioned table

2020-02-20 Thread Amit Langote
On Thu, Feb 20, 2020 at 4:50 PM Amit Langote wrote: > * I may be missing something, but why doesn't do_autovacuum() fetch a > partitioned table's entry from pgstat instead of fetching that for > individual children and adding? That is, why do we need to do the > following: > > +/* > +

RE: Complete data erasure

2020-02-20 Thread asaba.takan...@fujitsu.com
Hello Tom, From: Tom Lane > Tomas Vondra writes: > > I think it depends how exactly it's implemented. As Tom pointed out in > > his message [1], we can't do the erasure itself in the post-commit is > > not being able to handle errors. But if the files are renamed durably, > > and the erasure

RE: Complete data erasure

2020-02-20 Thread asaba.takan...@fujitsu.com
Greetings, From: asaba.takan...@fujitsu.com > Hello Stephen, > > From: Stephen Frost > > I disagree- it's a feature that's been asked for multiple times and does > > have value in some situations. > > I'm rethinking the need for this feature although I think that it improves the > security. >

Re: allow running parts of src/tools/msvc/ under not Windows

2020-02-20 Thread Peter Eisentraut
On 2020-02-13 16:36, Tom Lane wrote: Julien Rouhaud writes: On Thu, Feb 13, 2020 at 02:24:43PM +0100, Peter Eisentraut wrote: When making build system changes that risk breaking the MSVC build system, it's useful to be able to run the part of the MSVC build tools that read the makefiles and