Re: Commit/abort WAL records with dropped rels missing XLR_SPECIAL_REL_UPDATE

2020-08-14 Thread Amit Kapila
On Fri, Aug 14, 2020 at 2:17 PM Heikki Linnakangas wrote: > > While hacking on pg_rewind, I noticed that commit and abort WAL records > are never marked with the XLR_SPECIAL_REL_UPDATE flag. But if the record > contains "dropped relfilenodes", surely it should be? > Right. > It's harmless as

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-14 Thread Amit Kapila
On Sat, Aug 15, 2020 at 4:14 AM Thomas Munro wrote: > > On Fri, Aug 14, 2020 at 6:14 PM Amit Kapila wrote: > > Yeah, that makes sense. I will take care of that later today or > > tomorrow. We have not noticed that because currently none of the > > extensions is using those functions. BTW, I

Re: Asynchronous Append on postgres_fdw nodes.

2020-08-14 Thread Etsuro Fujita
On Fri, Aug 14, 2020 at 10:29 AM Thomas Munro wrote: > On Thu, Jul 2, 2020 at 3:20 PM Etsuro Fujita wrote: > > I'd like to join the party, but IIUC, we don't yet reach a consensus > > on which one is the right way to go. So I think we need to discuss > > that first. > > Either way, we

Re: run pgindent on a regular basis / scripted manner

2020-08-14 Thread Amit Kapila
On Sat, Aug 15, 2020 at 1:57 AM Alvaro Herrera wrote: > > On 2020-Aug-13, Magnus Hagander wrote: > > > That is: > > 1. Whenever a patch is pushed on master on the main repo a process kicked > > off (or maybe wait 5 minutes to coalesce multiple patches if there are) > > 2. This process checks out

Re: display offset along with block number in vacuum errors

2020-08-14 Thread Amit Kapila
On Fri, Aug 14, 2020 at 4:06 PM Amit Kapila wrote: > > On Mon, Aug 10, 2020 at 10:24 AM Masahiko Sawada > wrote: > > > > It's true that heap_page_is_all_visible() is called from only > > lazy_vacuum_page() but I'm concerned it would lead misleading since > > it's not actually removing tuples but

Re: Switch to multi-inserts for pg_depend

2020-08-14 Thread Michael Paquier
On Fri, Aug 14, 2020 at 02:23:16PM -0400, Alvaro Herrera wrote: > It seems a bit silly to worry about allocating just the exact amount > needed; the current approach looked fine to me. Okay, thanks. > The logic to keep track > number of used slots used is baroque, though -- that could use a lot

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-14 Thread Thomas Munro
On Fri, Aug 14, 2020 at 6:14 PM Amit Kapila wrote: > Yeah, that makes sense. I will take care of that later today or > tomorrow. We have not noticed that because currently none of the > extensions is using those functions. BTW, I noticed that after > failure, the next run is green, why so? Is the

Re: fixing old_snapshot_threshold's time->xid mapping

2020-08-14 Thread Thomas Munro
On Fri, Aug 14, 2020 at 1:04 PM Thomas Munro wrote: > On Fri, Aug 14, 2020 at 12:52 PM Thomas Munro wrote: > > Here's a rebase. > > And another, since I was too slow and v6 is already in conflict... > sorry for the high frequency patches. And ... now that this has a commitfest entry, cfbot told

Re: Dependencies for partitioned indexes are still a mess

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-14, Alvaro Herrera wrote: > On 2020-Aug-12, Alvaro Herrera wrote: > > > Hmm, we do make the FK constraint depend on the ATTACH for the direct > > children; what I think we're lacking is dependencies on descendants > > twice-removed (?) or higher. This mock patch seems to fix this

Re: run pgindent on a regular basis / scripted manner

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-13, Magnus Hagander wrote: > That is: > 1. Whenever a patch is pushed on master on the main repo a process kicked > off (or maybe wait 5 minutes to coalesce multiple patches if there are) > 2. This process checks out master, and runs pgindent on it > 3. When done, this gets committed

Re: run pgindent on a regular basis / scripted manner

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-13, Stephen Frost wrote: > For my 2c, anyway, I like the idea of having folks update the typedefs > themselves when they've got a patch that needs a new typedef to be > indented correctly. Well, let's for starters encourage committers to update typedefs. Personally I've stayed away

Re: Loose ends after CVE-2020-14350 (extension installation hazards)

2020-08-14 Thread Chapman Flack
On 08/14/20 15:38, Tom Lane wrote: > (3) If the SQL syntax is really just "WITH variable value [, ...]" > then I'm afraid we're going to have a lot of parse-ambiguity problems > with wedging full SET syntax into that. The ability for the righthand There is precedent in the SET command for

Re: [bug+patch] Inserting DEFAULT into generated columns from VALUES RTE

2020-08-14 Thread Mikhail Titov
Previously submitted patch got somehow trailing spaces mangled on the way out. This is an attempt to use application/octet-stream MIME instead of text/x-patch to preserve those for regression tests. On Thu, Aug 13, 2020 at 12:11 AM, Pavel Stehule wrote: > please, assign your patch to commitfest

Re: Loose ends after CVE-2020-14350 (extension installation hazards)

2020-08-14 Thread Tom Lane
Chapman Flack writes: > On 08/14/20 14:50, Tom Lane wrote: >> SAVEPOINT s1; >> SET LOCAL search_path = pg_catalog, pg_temp; >> ... protected code here ... >> RELEASE SAVEPOINT s1; >> but this does not work because SET LOCAL persists to the end of the >> outer transaction. Maybe we could invent a

Re: Loose ends after CVE-2020-14350 (extension installation hazards)

2020-08-14 Thread Chapman Flack
On 08/14/20 14:50, Tom Lane wrote: > SAVEPOINT s1; > SET LOCAL search_path = pg_catalog, pg_temp; > ... protected code here ... > RELEASE SAVEPOINT s1; > > but this does not work because SET LOCAL persists to the end of the > outer transaction. Maybe we could invent a

Loose ends after CVE-2020-14350 (extension installation hazards)

2020-08-14 Thread Tom Lane
Yesterday's releases included some fixes meant to make it harder for a malicious user to sabotage an extension installation/update script. There are some things remaining to be done in the area, though: 1. We don't have a way to make things adequately secure for extensions that depend on other

Re: Switch to multi-inserts for pg_depend

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-14, Michael Paquier wrote: > Regarding the maximum number of slots allocated. Do people like the > current approach taken by the patch to do a single loop of the > dependency entries at the cost of more allocating perhaps too much for > the array holding the set of TupleTableSlots

Re: Dependencies for partitioned indexes are still a mess

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-12, Alvaro Herrera wrote: > Hmm, we do make the FK constraint depend on the ATTACH for the direct > children; what I think we're lacking is dependencies on descendants > twice-removed (?) or higher. This mock patch seems to fix this problem > by adding dependencies recursively on all

Re: Parallel query hangs after a smart shutdown is issued

2020-08-14 Thread Tom Lane
Arseny Sher writes: > FWIW, I've also looked through the patch and it's fine. Moderate testing > also found no issues, check-world works, bgws are started during smart > shutdown as expected. And surely this is better than the inital > shorthack of allowing only parallel workers. Thanks,

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-14 Thread Bruce Momjian
On Fri, Aug 14, 2020 at 10:23:27PM +0900, Michael Paquier wrote: > We have nothing in core, yet, that helps with this kind of problem > with binary upgrades. In the last year, Julien and I worked on an > upgrade case where a glibc upgrade was involved with pg_upgrade used > for PG, and it could

Re: Parallel query hangs after a smart shutdown is issued

2020-08-14 Thread Arseny Sher
Tom Lane writes: > Thomas Munro writes: >> On Fri, Aug 14, 2020 at 4:45 AM Tom Lane wrote: >>> After some more rethinking and testing, here's a v5 that feels >>> fairly final to me. I realized that the logic in canAcceptConnections >>> was kind of backwards: it's better to check the main

Re: Implement a new data type

2020-08-14 Thread Tom Lane
mohand oubelkacem makhoukhene writes: > I would like to implement a new data type next to char, number, varchar... > for example a special "Money" type, but > I don't want to use extensions and the Create type command. I want to > implement it directly inside source code, > because I want to

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2020-08-14 Thread Ibrar Ahmed
On Mon, Aug 3, 2020 at 2:29 PM Anastasia Lubennikova < a.lubennik...@postgrespro.ru> wrote: > On 31.07.2020 23:28, Robert Haas wrote: > > On Tue, Jul 14, 2020 at 1:51 PM Anastasia Lubennikova > > wrote: > >> Questions from the first review pass: > >> > >> 1) Do we need XLH_INSERT_ALL_VISIBLE_SET

Implement a new data type

2020-08-14 Thread mohand oubelkacem makhoukhene
Hello I would like to implement a new data type next to char, number, varchar... for example a special "Money" type, but I don't want to use extensions and the Create type command. I want to implement it directly inside source code, because I want to implement my new type at lower level, in

Re: jsonb, collection & postgres_fdw

2020-08-14 Thread Tom Lane
Konstantin Knizhnik writes: > I still do not completely understand current criteria of shippable > functions. > I understood Tom's explanation, but: > postgres=# create table t1(t text collate "C"); > CREATE TABLE > postgres=# create foreign table ft1(t text collate "ru_RU") server > pg_fdw

Re: Inconsistent behavior of smart shutdown handling for queries with and without parallel workers

2020-08-14 Thread Tom Lane
Bharath Rupireddy writes: > Is there any way the bgworkers(for that matter, any postmaster's child > process) knowing that there's a smart shutdown pending? This is > useful, if any of the bgworker(if not parallel workers) want to > differentiate the two modes i.e. smart and fast shutdown modes

Re: Parallel query hangs after a smart shutdown is issued

2020-08-14 Thread Tom Lane
Thomas Munro writes: > On Fri, Aug 14, 2020 at 4:45 AM Tom Lane wrote: >> After some more rethinking and testing, here's a v5 that feels >> fairly final to me. I realized that the logic in canAcceptConnections >> was kind of backwards: it's better to check the main pmState restrictions >> first

Re: Autonomous database is coming to Postgres?

2020-08-14 Thread Dmitry Dolgov
> On Fri, Aug 14, 2020 at 08:55:53AM -0400, Bruce Momjian wrote: > > On Thu, Aug 13, 2020 at 03:26:33AM +, tsunakawa.ta...@fujitsu.com wrote: > > Hello, > > > > I'm not sure if I should have posted this to pgsql-advocacy, but this is > > being developed so I posted here. > > Does anyone know

Re: Newline after --progress report

2020-08-14 Thread Tom Lane
Heikki Linnakangas writes: > While hacking on pg_rewind, this in pg_rewind's main() function caught > my eye: Good catch. > Attached is a patch to fix this, as well as a similar issue in > pg_checksums. pg_basebackup and pgbench also print progres reports like > this, but they seem correct

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-14 Thread Michael Paquier
On Fri, Aug 14, 2020 at 09:00:06AM -0400, Bruce Momjian wrote: > On Tue, Aug 11, 2020 at 02:58:30PM +0200, Magnus Hagander wrote: >> I assumed it had code for that stuff already. Mainly because I assumed it >> supported doing pg_upgrade, which requires similar things no? > > While pg_upgrade

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-14 Thread Bruce Momjian
On Tue, Aug 11, 2020 at 02:58:30PM +0200, Magnus Hagander wrote: > On Tue, Aug 4, 2020 at 11:42 AM Dave Page wrote: > That would require fairly large changes to the installer to allow it to > login to the database server (whether that would work would be dependent > on > how

Re: Autonomous database is coming to Postgres?

2020-08-14 Thread Bruce Momjian
On Thu, Aug 13, 2020 at 03:26:33AM +, tsunakawa.ta...@fujitsu.com wrote: > Hello, > > I'm not sure if I should have posted this to pgsql-advocacy, but this is > being developed so I posted here. > > Does anyone know if this development come to open source Postgres, or only to > the cloud

Re: display offset along with block number in vacuum errors

2020-08-14 Thread Amit Kapila
On Thu, Aug 6, 2020 at 7:41 PM Amit Kapila wrote: > > On Wed, Aug 5, 2020 at 12:47 AM Mahendra Singh Thalor > wrote: > > > > Apart from these, I fixed Justin's comment of extra brackets(That was > > due to "patch -p 1 < file", as 002_fix was not applying directly). I > > haven't updated the

Re: display offset along with block number in vacuum errors

2020-08-14 Thread Amit Kapila
On Fri, Aug 7, 2020 at 7:18 AM Amit Kapila wrote: > > On Thu, Aug 6, 2020 at 7:51 PM Justin Pryzby wrote: > > > > On Thu, Aug 06, 2020 at 07:39:21PM +0530, Amit Kapila wrote: > > > On Wed, Jul 29, 2020 at 1:09 AM Justin Pryzby > > > wrote: > > > > > > > > > > > > lazy_check_needs_freeze

Re: display offset along with block number in vacuum errors

2020-08-14 Thread Amit Kapila
On Mon, Aug 10, 2020 at 10:24 AM Masahiko Sawada wrote: > > It's true that heap_page_is_all_visible() is called from only > lazy_vacuum_page() but I'm concerned it would lead misleading since > it's not actually removing tuples but just checking after vacuum. I > guess that the errcontext should

Re: Yet another fast GiST build (typo)

2020-08-14 Thread Pavel Borisov
I see this feature quite useful in concept and decided to test it. On a real database of 7 million rows I observed speedup of 4 times in case of single column index on points only and 2.5 times speedup in case of index on points with several included columns. Standard deviation between in series

Commit/abort WAL records with dropped rels missing XLR_SPECIAL_REL_UPDATE

2020-08-14 Thread Heikki Linnakangas
While hacking on pg_rewind, I noticed that commit and abort WAL records are never marked with the XLR_SPECIAL_REL_UPDATE flag. But if the record contains "dropped relfilenodes", surely it should be? It's harmless as far as the backend and all the programs in PostgreSQL repository are

Re: Fix an old description in high-availability.sgml

2020-08-14 Thread Michael Paquier
On Fri, Aug 14, 2020 at 04:53:54PM +0900, Masahiko Sawada wrote: > The following sentence in high-availability.sgml is not true: > > The background writer is active during recovery and will perform > restartpoints (similar to checkpoints on the primary) and normal block > cleaning

Re: Switch to multi-inserts for pg_depend

2020-08-14 Thread Michael Paquier
On Thu, Aug 13, 2020 at 11:45:52AM -0400, Alvaro Herrera wrote: > MAX_CATALOG_INSERT_BYTES sounds decent to me. I mentioned dependency.h > because I was uncaffeinatedly thinking that this was used with API > defined there -- but in reality it's used with indexing.h functions, and > it seems to me

Re: Terminate the idle sessions

2020-08-14 Thread Li Japin
On Aug 14, 2020, at 2:15 PM, Bharath Rupireddy mailto:bharath.rupireddyforpostg...@gmail.com>> wrote: I think, since the idle_session_timeout is by default disabled, we have no problem. My thought is what if a user enables the feature(knowingly or unknowingly) on the remote backend? If the

Newline after --progress report

2020-08-14 Thread Heikki Linnakangas
While hacking on pg_rewind, this in pg_rewind's main() function caught my eye: progress_report(true); printf("\n"); It is peculiar, because progress_report() uses fprintf(stderr, ...) for all its printing, and in fact the only other use of printf() in pg_rewind is in printing the

Re: massive FPI_FOR_HINT load after promote

2020-08-14 Thread Simon Riggs
On Mon, 10 Aug 2020 at 23:56, Alvaro Herrera wrote: > > The problem was simply that when a page is > examined by a seqscan, we do HeapTupleSatisfiesVisibility of each tuple > in isolation; and for each tuple we call SetHintBits(). And only the > first time the FPI happens; by the time we get to

Fix an old description in high-availability.sgml

2020-08-14 Thread Masahiko Sawada
Hi, The following sentence in high-availability.sgml is not true: The background writer is active during recovery and will perform restartpoints (similar to checkpoints on the primary) and normal block cleaning activities. I think this is an oversight of the commit 806a2ae in 2011;

Re: Collation versioning

2020-08-14 Thread Michael Paquier
On Fri, Aug 14, 2020 at 02:21:58PM +1200, Thomas Munro wrote: > Thanks Julien. I'm planning to do a bit more testing and review, and > then hopefully commit this next week. If anyone else has objections > to this design, now would be a good time to speak up. The design to use pg_depend for the

Re: jsonb, collection & postgres_fdw

2020-08-14 Thread Konstantin Knizhnik
On 14.08.2020 09:40, Bharath Rupireddy wrote: On Thu, Aug 13, 2020 at 8:54 PM Konstantin Knizhnik wrote: Right now jsonb functions are treated as non-shippable by postgres_fdw and so predicates with them are not pushed down to foreign server: I wonder if there is some way of making

Re: jsonb, collection & postgres_fdw

2020-08-14 Thread Bharath Rupireddy
On Thu, Aug 13, 2020 at 8:54 PM Konstantin Knizhnik wrote: > > Right now jsonb functions are treated as non-shippable by postgres_fdw > and so predicates with them are not pushed down to foreign server: > > I wonder if there is some way of making postgres_fdw to push this this > function to

Re: Terminate the idle sessions

2020-08-14 Thread Bharath Rupireddy
On Tue, Aug 11, 2020 at 8:45 AM Li Japin wrote: > > I’ve attached a new version that add “idle_session_timeout” in the default > postgresql.conf. > Hi, I would like to just mention a use case I thought of while discussing [1]: In postgres_fdw: assuming we use idle_in_session_timeout on remote

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-14 Thread Amit Kapila
On Fri, Aug 14, 2020 at 10:11 AM Thomas Munro wrote: > > On Thu, Aug 13, 2020 at 6:38 PM Amit Kapila wrote: > > I have pushed that patch last week and attached are the remaining > > patches. I have made a few changes in the next patch > > 0001-Extend-the-BufFile-interface.patch and have some

Re: Parallel query hangs after a smart shutdown is issued

2020-08-14 Thread Thomas Munro
On Fri, Aug 14, 2020 at 4:45 AM Tom Lane wrote: > I wrote: > > Hmmm ... maybe that should be more like > > if (smartShutState != SMART_NORMAL_USAGE && > > backend_type == BACKEND_TYPE_NORMAL) > > After some more rethinking and testing, here's a v5 that feels > fairly final to