Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread David Rowley
On Thu, 21 May 2020 at 14:32, Thomas Munro wrote: > Thanks. So it seems like Linux, Windows and anything using ZFS are > OK, which probably explains why we hadn't heard complaints about it. I tried out a different test on a Windows 8.1 machine I have here. I was concerned that the test that

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-20 Thread John Naylor
Hi Jon, I noticed a couple minor inconsistencies: ".datetime" -> elsewhere functions are formatted as `.datetime()` libpq -> `libpq` The link to the release notes on its own line is the same as the inline link, if that makes sense. In other places with links on their own line, the full URL is

Re: Is it useful to record whether plans are generic or custom?

2020-05-20 Thread Fujii Masao
On 2020/05/20 21:56, Atsushi Torikoshi wrote: On Wed, May 20, 2020 at 1:32 PM Kyotaro Horiguchi mailto:horikyota@gmail.com>> wrote: At Tue, 19 May 2020 22:56:17 +0900, Atsushi Torikoshi mailto:ato...@gmail.com>> wrote in > On Sat, May 16, 2020 at 6:01 PM legrand legrand

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-20 Thread David Gilman
I did some more digging. To keep everyone on the same page there are four different ways to order TOCs: 1. topological order, 2. dataLength order, size of the table, is always zero when pg_dump can't seek, 3. dumpId order, which should be thought as random but roughly correlates to topological

Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread Thomas Munro
On Thu, May 21, 2020 at 1:38 PM Ranier Vilela wrote: >> >> On Thu, May 21, 2020 at 11:15 AM Ranier Vilela >> >> wrote: >> >> > postgres=# set max_parallel_workers_per_gather = 0; >> >> > Time: 227238,445 ms (03:47,238) >> >> > postgres=# set max_parallel_workers_per_gather = 1; >> >> > Time:

Re: Optimizer docs typos

2020-05-20 Thread Kyotaro Horiguchi
At Wed, 20 May 2020 19:17:48 +0900, Etsuro Fujita wrote in > On Tue, May 19, 2020 at 7:35 PM Etsuro Fujita wrote: > > On Mon, May 18, 2020 at 7:45 PM Richard Guo wrote: > > > ---query_planner() > > > +---query_planner() > > > make list of base relations used in query > > > split up

Re: Subplan result caching

2020-05-20 Thread Andy Fan
On Wed, May 20, 2020 at 7:47 PM David Rowley wrote: > On Mon, 27 Apr 2020 at 00:37, Andy Fan wrote: > > I was feeling that we may have to maintain some extra status if we use > hash > > table rather than tuple store, but that might be not a major concern. I > can > > wait and see your patch. >

Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread Ranier Vilela
Em qua., 20 de mai. de 2020 às 21:03, Thomas Munro escreveu: > On Thu, May 21, 2020 at 11:51 AM Ranier Vilela > wrote: > > Em qua., 20 de mai. de 2020 às 20:48, Thomas Munro < > thomas.mu...@gmail.com> escreveu: > >> On Thu, May 21, 2020 at 11:15 AM Ranier Vilela > wrote: > >> > postgres=# set

Re: Expand the use of check_canonical_path() for more GUCs

2020-05-20 Thread Kyotaro Horiguchi
At Wed, 20 May 2020 10:05:29 +0200, Peter Eisentraut wrote in > On 2020-05-20 09:13, Michael Paquier wrote: > > On Tue, May 19, 2020 at 01:02:12PM +0200, Peter Eisentraut wrote: > >> That thread didn't resolve why check_canonical_path() is necessary > >> there. > >> Maybe the existing uses

Re: Adding missing object access hook invocations

2020-05-20 Thread Michael Paquier
On Wed, May 20, 2020 at 01:57:31PM -0400, Robert Haas wrote: > I don't really see any reason why this couldn't be committed even at > this late date, but I also don't care that much. I suspect the number > of extension authors who are likely to have to make any code changes > is small. It's

Re: Operator class parameters and sgml docs

2020-05-20 Thread Alexander Korotkov
Hi, Peter! On Thu, May 21, 2020 at 12:37 AM Peter Geoghegan wrote: > Commit 911e7020770 added a variety of new support routines to index > AMs. For example, it added a support function 5 to btree (see > BTOPTIONS_PROC), but didn't document this alongside the other support > functions in

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-20 Thread Tomas Vondra
On Tue, May 19, 2020 at 09:15:40PM -0700, Jeff Davis wrote: On Tue, 2020-05-19 at 19:53 +0200, Tomas Vondra wrote: And if there a way to pre-allocate larger chunks? Presumably we could assign the blocks to tape in larger chunks (e.g. 128kB, i.e. 16 x 8kB) instead of just single block. I

Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread Thomas Munro
On Thu, May 21, 2020 at 11:51 AM Ranier Vilela wrote: > Em qua., 20 de mai. de 2020 às 20:48, Thomas Munro > escreveu: >> On Thu, May 21, 2020 at 11:15 AM Ranier Vilela wrote: >> > postgres=# set max_parallel_workers_per_gather = 0; >> > Time: 227238,445 ms (03:47,238) >> > postgres=# set

Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread Ranier Vilela
Em qua., 20 de mai. de 2020 às 20:48, Thomas Munro escreveu: > On Thu, May 21, 2020 at 11:15 AM Ranier Vilela > wrote: > > postgres=# set max_parallel_workers_per_gather = 0; > > Time: 227238,445 ms (03:47,238) > > postgres=# set max_parallel_workers_per_gather = 1; > > Time: 138027,351 ms

Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread Thomas Munro
On Thu, May 21, 2020 at 11:15 AM Ranier Vilela wrote: > postgres=# set max_parallel_workers_per_gather = 0; > Time: 227238,445 ms (03:47,238) > postgres=# set max_parallel_workers_per_gather = 1; > Time: 138027,351 ms (02:18,027) Ok, so it looks like NT/NTFS isn't suffering from this problem.

Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread Ranier Vilela
Em qua., 20 de mai. de 2020 às 18:49, Thomas Munro escreveu: > On Wed, May 20, 2020 at 11:03 PM Ranier Vilela > wrote: > > Time: 47767,916 ms (00:47,768) > > Time: 32645,448 ms (00:32,645) > > Just to make sure kernel caching isn't helping here, maybe try making > the table 2x or 4x bigger? My

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-20 Thread Jonathan S. Katz
On 5/20/20 6:42 PM, Justin Pryzby wrote: > On Wed, May 20, 2020 at 06:11:08PM -0400, Jonathan S. Katz wrote: >> This release includes more ways to monitor actibity within a PostgreSQL > > activity ...that one is embarrassing. Thanks. > >> partition its "accounts" table, making it easier to

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-20 Thread Justin Pryzby
On Wed, May 20, 2020 at 06:11:08PM -0400, Jonathan S. Katz wrote: > This release includes more ways to monitor actibity within a PostgreSQL activity > partition its "accounts" table, making it easier to benchmark workloads that > contains partitions. contain No need to respond :) Thanks,

Re: factorial function/phase out postfix operators?

2020-05-20 Thread Tom Lane
Alvaro Herrera writes: > On 2020-May-20, Tom Lane wrote: >> I feel like we'd be better advised to somehow >> treat can-be-bare-col-label as an independent classification. > Would it make sense (and possible) to have a keyword category that is > not disjoint wrt. the others? Maybe that ends up

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-20 Thread Jonathan S. Katz
Hi, On 5/18/20 10:29 PM, Jonathan S. Katz wrote: > Hi, > > Attached is a draft of the release announcement for the PostgreSQL 13 > Beta 1 release this week. > > The goal of this release announcement is to make people aware of the new > features that are introduced in PostgreSQL 13 and,

Re: factorial function/phase out postfix operators?

2020-05-20 Thread Alvaro Herrera
On 2020-May-20, Tom Lane wrote: > I too failed to save the results of some experimentation, but I'd > also poked at the type_func_name_keyword category, and it has a similar > situation where only about three keywords cause problems if included > in BareColLabel. So we could possibly get another

Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread Thomas Munro
On Wed, May 20, 2020 at 11:03 PM Ranier Vilela wrote: > Time: 47767,916 ms (00:47,768) > Time: 32645,448 ms (00:32,645) Just to make sure kernel caching isn't helping here, maybe try making the table 2x or 4x bigger? My test was on a virtual machine with only 4GB RAM, so the table couldn't be

Operator class parameters and sgml docs

2020-05-20 Thread Peter Geoghegan
Commit 911e7020770 added a variety of new support routines to index AMs. For example, it added a support function 5 to btree (see BTOPTIONS_PROC), but didn't document this alongside the other support functions in btree.sgml. It looks like the new support functions are fundamentally different to

Re: factorial function/phase out postfix operators?

2020-05-20 Thread Tom Lane
Mark Dilger writes: > ... But if we made a clean distinction between the characters that are > allowed in postfix operators vs. those allowed for infix operators, then we'd > get to have postfix operators without the ambiguity, right? I continue to see little point in half-baked compatibility

Re: factorial function/phase out postfix operators?

2020-05-20 Thread Mark Dilger
> On May 20, 2020, at 11:24 AM, Tom Lane wrote: > > Bottom line is that we can reduce the scope of the col-label problem > this way, but we can't make it go away entirely. Is a partial solution > to that worth a full drop of postfix operators? Possibly, but I'm not > sure. I still feel

Re: SEARCH and CYCLE clauses

2020-05-20 Thread Vik Fearing
On 5/20/20 3:04 PM, Vik Fearing wrote: > I'm looking forward to reviewing this. A few quick things I've noticed so far: 1) There are some smart quotes in the comments that should be converted to single quotes. 2) This query is an infinite loop, as expected: with recursive a as (select 1 as

Re: factorial function/phase out postfix operators?

2020-05-20 Thread Tom Lane
Robert Haas writes: > On Tue, May 19, 2020 at 7:47 PM Tom Lane wrote: >> However, as the patch >> stands, only the remaining fully-unreserved keywords can be used as bare >> column labels. I'd hoped to be able to also use col_name keywords in that >> way (which'd make the set of legal bare

Re: Adding missing object access hook invocations

2020-05-20 Thread Robert Haas
On Mon, Apr 20, 2020 at 9:40 PM Michael Paquier wrote: > Okay. Any other opinions? I am in a 50/50 state about that stuff. I don't really see any reason why this couldn't be committed even at this late date, but I also don't care that much. I suspect the number of extension authors who are

Re: Extension ownership and misuse of SET ROLE/SET SESSION AUTHORIZATION

2020-05-20 Thread Robert Haas
On Thu, Feb 13, 2020 at 5:55 PM Tom Lane wrote: > (1) In the backend, allow SET ROLE to succeed if either the session > userid or the current userid is a member of the desired role. This > would mean that, given the use-case for --role that you are logging > into an account that can "SET ROLE

Re: Two fsync related performance issues?

2020-05-20 Thread Robert Haas
On Tue, May 19, 2020 at 4:31 PM Thomas Munro wrote: > What would a precise version of this look like? Maybe we really only > need to fsync relation files that recovery modifies (as we already > do), plus those that it would have touched but didn't because of the > page LSN (a new behaviour to

Re: factorial function/phase out postfix operators?

2020-05-20 Thread Robert Haas
On Tue, May 19, 2020 at 7:47 PM Tom Lane wrote: > As Robert theorized, it works to move a fairly-small number of unreserved > keywords into a new slightly-reserved category. It wasn't entirely a theoretical argument, since I'm pretty sure I did spend some time experimenting with gram.y back in

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-20 Thread Tom Lane
David Gilman writes: >> I think the PG11 >> commit you mentioned (548e5097) happens to make some databases fail in >> parallel restore that previously worked (I didn't check). > Correct, if you do the bisect around that yourself you'll see > pg_restore start failing with the expected "possibly

Re: Extensions not dumped when --schema is used

2020-05-20 Thread Tom Lane
Guillaume Lelarge writes: > Le mer. 20 mai 2020 à 11:26, Daniel Gustafsson a écrit : >> The question is what --extensions should do: only dump any >> extensions that objects in the schema depend on; require a pattern and only >> dump matching extensions; dump all extensions (probably not) or

Fwd: [PATCH] fix GIN index search sometimes losing results

2020-05-20 Thread Pavel Borisov
1. Really if it's possible to avoid bool callbacks at all and shift everywhere to ternary it makes code quite beautiful and even. But I also think we are still not obliged to drop support for (legacy or otherwise) bool callbacks and also consistent functions form some old extensions (I don't know

Re: pgindent vs dtrace on macos

2020-05-20 Thread Tom Lane
Peter Eisentraut writes: > If I run pgindent on a built tree on macos, I get this error > Failure in ./src/backend/utils/probes.h: Error@375: Stuff missing from > end of file > The file in question is built by the dtrace command. I have attached it > here. > Is this something to fix in

Re: SEARCH and CYCLE clauses

2020-05-20 Thread Vik Fearing
On 5/20/20 1:46 PM, Peter Eisentraut wrote: > I have implemented the SEARCH and CYCLE clauses. YES! > This is standard SQL syntax attached to a recursive CTE to compute a > depth- or breadth-first ordering and cycle detection, respectively. This > is just convenience syntax for what you can

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-05-20 Thread Simon Riggs
On Wed, 20 May 2020 at 12:44, David Rowley wrote: > Hackers, > > Over on [1], Heikki mentioned about the usefulness of caching results > from parameterized subplans so that they could be used again for > subsequent scans which have the same parameters as a previous scan. > On [2], I mentioned

Re: Is it useful to record whether plans are generic or custom?

2020-05-20 Thread Atsushi Torikoshi
On Wed, May 20, 2020 at 1:32 PM Kyotaro Horiguchi wrote: > At Tue, 19 May 2020 22:56:17 +0900, Atsushi Torikoshi > wrote in > > On Sat, May 16, 2020 at 6:01 PM legrand legrand < > legrand_legr...@hotmail.com> > > wrote: > > > > BTW, I'd also appreciate other opinions about recording the number

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-20 Thread David Gilman
Your understanding of the issue is mostly correct: > I think the PG11 > commit you mentioned (548e5097) happens to make some databases fail in > parallel restore that previously worked (I didn't check). Correct, if you do the bisect around that yourself you'll see pg_restore start failing with

SEARCH and CYCLE clauses

2020-05-20 Thread Peter Eisentraut
I have implemented the SEARCH and CYCLE clauses. This is standard SQL syntax attached to a recursive CTE to compute a depth- or breadth-first ordering and cycle detection, respectively. This is just convenience syntax for what you can already do manually. The original discussion about

Re: Subplan result caching

2020-05-20 Thread David Rowley
On Mon, 27 Apr 2020 at 00:37, Andy Fan wrote: > I was feeling that we may have to maintain some extra status if we use hash > table rather than tuple store, but that might be not a major concern. I can > wait and see your patch. I've posted the patch and lots of details about it in

Hybrid Hash/Nested Loop joins and caching results from subplans

2020-05-20 Thread David Rowley
Hackers, Over on [1], Heikki mentioned about the usefulness of caching results from parameterized subplans so that they could be used again for subsequent scans which have the same parameters as a previous scan. On [2], I mentioned that parameterized nested loop joins could see similar gains with

Re: Add A Glossary

2020-05-20 Thread Laurenz Albe
On Wed, 2020-05-20 at 13:17 +0200, Jürgen Purtz wrote: > > FWIW, I feel somewhat like Alvaro on that point; I use those terms > > synonymously, > > perhaps distinguishing between a "started cluster" and a "stopped cluster". > > After all, "cluster" refers to "a cluster of databases", which are

Re: Add A Glossary

2020-05-20 Thread Jürgen Purtz
On 19.05.20 08:17, Laurenz Albe wrote: On Mon, 2020-05-18 at 18:08 +0200, Jürgen Purtz wrote: cluster/instance: PG (mainly) consists of a group of processes that commonly act on shared buffers. The processes are very closely related to each other and with the buffers. They exist altogether or

Re: Parallel Seq Scan vs kernel read ahead

2020-05-20 Thread Ranier Vilela
Em qua., 20 de mai. de 2020 às 00:09, Thomas Munro escreveu: > On Wed, May 20, 2020 at 2:23 PM Amit Kapila > wrote: > > Good experiment. IIRC, we have discussed a similar idea during the > > development of this feature but we haven't seen any better results by > > allocating in ranges on the

Re: Optimizer docs typos

2020-05-20 Thread Etsuro Fujita
On Tue, May 19, 2020 at 7:35 PM Etsuro Fujita wrote: > On Mon, May 18, 2020 at 7:45 PM Richard Guo wrote: > > In this same README doc, another suspicious typo to me, which happens in > > section "Optimizer Functions", is in the prefix to query_planner(), > > we should have three dashes, rather

Re: pgindent vs dtrace on macos

2020-05-20 Thread Daniel Gustafsson
> On 20 May 2020, at 11:52, Peter Eisentraut > wrote: > Or should this file be excluded, since it's generated? That would get my vote. Generated files where we don't control the generator can be excluded. cheers ./daniel

Re: Extensions not dumped when --schema is used

2020-05-20 Thread Daniel Gustafsson
> On 20 May 2020, at 11:38, Guillaume Lelarge wrote: > Actually, "dump all extensions" (#3) would make sense to me, and has my vote. Wouldn't that open for another set of problems when running with --schema=bar and getting errors on restoring for relocatable extensions like these:

pgindent vs dtrace on macos

2020-05-20 Thread Peter Eisentraut
If I run pgindent on a built tree on macos, I get this error Failure in ./src/backend/utils/probes.h: Error@375: Stuff missing from end of file The file in question is built by the dtrace command. I have attached it here. Is this something to fix in pgindent? Or should this file be

Re: Extensions not dumped when --schema is used

2020-05-20 Thread Guillaume Lelarge
Le mer. 20 mai 2020 à 11:26, Daniel Gustafsson a écrit : > > On 20 May 2020, at 10:06, Guillaume Lelarge > wrote: > > > I was wondering the reason behind this choice. If anyone knows, I'd be > happy to hear about it. > > Extensions were dumped unconditionally in the beginning, but it was >

Re: Extensions not dumped when --schema is used

2020-05-20 Thread Daniel Gustafsson
> On 20 May 2020, at 10:06, Guillaume Lelarge wrote: > I was wondering the reason behind this choice. If anyone knows, I'd be happy > to hear about it. Extensions were dumped unconditionally in the beginning, but it was changed to match how procedural language definitions were dumped. > *

Re: Extensions not dumped when --schema is used

2020-05-20 Thread Laurenz Albe
On Wed, 2020-05-20 at 10:06 +0200, Guillaume Lelarge wrote: > I've discovered something today that I didn't really expect. > When a user dumps a database with the --schema flag of pg_dump, > extensions in this schema aren't dumped. As far as I can tell, > the documentation isn't clear about this

Re: explicit_bzero for sslpassword

2020-05-20 Thread Peter Eisentraut
On 2020-05-20 07:56, Michael Paquier wrote: On Tue, May 19, 2020 at 02:33:40PM +0200, Daniel Gustafsson wrote: Since commit 74a308cf5221f we use explicit_bzero on pgpass and connhost password in libpq, but not sslpassword which seems an oversight. The attached performs an explicit_bzero before

Extensions not dumped when --schema is used

2020-05-20 Thread Guillaume Lelarge
Hello, I've discovered something today that I didn't really expect. When a user dumps a database with the --schema flag of pg_dump, extensions in this schema aren't dumped. As far as I can tell, the documentation isn't clear about this ("Dump only schemas matching pattern; this selects both the

Re: Expand the use of check_canonical_path() for more GUCs

2020-05-20 Thread Peter Eisentraut
On 2020-05-20 09:13, Michael Paquier wrote: On Tue, May 19, 2020 at 01:02:12PM +0200, Peter Eisentraut wrote: That thread didn't resolve why check_canonical_path() is necessary there. Maybe the existing uses could be removed? This would impact log_directory, external_pid_file,

Re: Problem with pg_atomic_compare_exchange_u64 at 32-bit platforms

2020-05-20 Thread Konstantin Knizhnik
On 20.05.2020 10:36, Noah Misch wrote: On Wed, May 20, 2020 at 10:23:37AM +0300, Konstantin Knizhnik wrote: On 20.05.2020 06:05, Noah Misch wrote: On Tue, May 19, 2020 at 04:07:29PM +0300, Konstantin Knizhnik wrote: Definition of pg_atomic_compare_exchange_u64 requires alignment of expected

Re: Problem with pg_atomic_compare_exchange_u64 at 32-bit platforms

2020-05-20 Thread Noah Misch
On Wed, May 20, 2020 at 10:23:37AM +0300, Konstantin Knizhnik wrote: > On 20.05.2020 06:05, Noah Misch wrote: > >On Tue, May 19, 2020 at 04:07:29PM +0300, Konstantin Knizhnik wrote: > >>Definition of pg_atomic_compare_exchange_u64 requires alignment of expected > >>pointer on 8-byte boundary. > >>

Re: Problem with pg_atomic_compare_exchange_u64 at 32-bit platforms

2020-05-20 Thread Konstantin Knizhnik
On 20.05.2020 08:10, Andres Freund wrote: Hi, On May 19, 2020 8:05:00 PM PDT, Noah Misch wrote: On Tue, May 19, 2020 at 04:07:29PM +0300, Konstantin Knizhnik wrote: Definition of pg_atomic_compare_exchange_u64 requires alignment of expected pointer on 8-byte boundary.

Re: Problem with pg_atomic_compare_exchange_u64 at 32-bit platforms

2020-05-20 Thread Konstantin Knizhnik
On 20.05.2020 06:05, Noah Misch wrote: On Tue, May 19, 2020 at 04:07:29PM +0300, Konstantin Knizhnik wrote: Definition of pg_atomic_compare_exchange_u64 requires alignment of expected pointer on 8-byte boundary. pg_atomic_compare_exchange_u64(volatile pg_atomic_uint64 *ptr,                 

Re: Expand the use of check_canonical_path() for more GUCs

2020-05-20 Thread Michael Paquier
On Tue, May 19, 2020 at 01:02:12PM +0200, Peter Eisentraut wrote: > That thread didn't resolve why check_canonical_path() is necessary there. > Maybe the existing uses could be removed? This would impact log_directory, external_pid_file, stats_temp_directory, where it is still useful to show to

Re: Expand the use of check_canonical_path() for more GUCs

2020-05-20 Thread Michael Paquier
On Tue, May 19, 2020 at 09:32:15AM -0400, Tom Lane wrote: > Hm, I'm pretty certain that data_directory does not need this because > canonicalization is done elsewhere; the most that you could accomplish > there is to cause problems. Dunno about the rest. Hmm. I missed that this is getting done