Re: Devel docs on website reloading

2020-11-18 Thread Heikki Linnakangas
On 18/11/2020 12:56, Magnus Hagander wrote: Today, we build and update the docs on https://www.postgresql.org/docs/devel/ are rebuilt and deployed by the buildfarm automatically every 4 hours. If there are no changes at all made to the docs, they are *still* kicked out of all caches and the

Re: don't allocate HashAgg hash tables when running explain only

2020-11-18 Thread Heikki Linnakangas
On 13/11/2020 18:10, Alexey Bashtanov wrote: Hi, I got somewhat scared when my explain took a few seconds to complete and used a few gigs of RAM. To reproduce try the following: discard temp; create temp table a as select to_timestamp(generate_series(1, 7000)) i; analyze a; set work_mem to

Re: Add LWLock blocker(s) information

2020-11-18 Thread Heikki Linnakangas
On 11/08/2020 03:41, Andres Freund wrote: On 2020-08-10 18:27:17 -0400, Robert Haas wrote: On Tue, Jun 2, 2020 at 8:25 AM Drouvot, Bertrand wrote: the patch adds into the LWLock struct: last_holding_pid: last pid owner of the lock last_mode: last

Re: ResourceOwner refactoring

2020-11-18 Thread Heikki Linnakangas
On 18/11/2020 10:06, Michael Paquier wrote: On Tue, Nov 17, 2020 at 04:21:29PM +0200, Heikki Linnakangas wrote: Attached patch refactors the ResourceOwner internals to do that. + * Size of the small fixed-size array to hold most-recently remembered resources. */ -#define

Re: [patch] [doc] Clarify that signal functions have no feedback

2020-11-18 Thread Heikki Linnakangas
On 17/11/2020 21:50, David G. Johnston wrote: On Tue, Nov 17, 2020 at 6:13 AM Heikki Linnakangas <mailto:hlinn...@iki.fi>> wrote: You can use the pg_file_settings and pg_hba_file_rules views to check the configuration files for possible errors, before reloading. I agree wi

Re: Protect syscache from bloating with negative cache entries

2020-11-17 Thread Heikki Linnakangas
On 09/11/2020 11:34, Kyotaro Horiguchi wrote: At Fri, 6 Nov 2020 10:42:15 +0200, Heikki Linnakangas wrote in Do you need the "ntaccess == 2" test? You could always increment the counter, and in the code that uses ntaccess to decide what to evict, treat all values >= 2 the same. N

ResourceOwner refactoring

2020-11-17 Thread Heikki Linnakangas
Two recent patches that I have reviewed have reminded me that the ResourceOwner interface is a bit clunky. There are two issues: 1. Performance. It's fast enough in most use, but when I was testing Kyotaro's catcache patches [1], the Resowner functions to track catcache references

Re: [patch] [doc] Clarify that signal functions have no feedback

2020-11-17 Thread Heikki Linnakangas
On 02/11/2020 18:02, David G. Johnston wrote: diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index bf6004f321..43bc2cf086 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -23892,7 +23892,8 @@ SELECT collation for ('foo' COLLATE "de_DE"); The

Re: Split copy.c

2020-11-16 Thread Heikki Linnakangas
On 16/11/2020 04:28, Justin Pryzby wrote: On Tue, Nov 03, 2020 at 03:15:27PM +1300, David Rowley wrote: On Tue, 3 Nov 2020 at 07:35, Andres Freund wrote: On 2020-11-02 19:43:38 +0200, Heikki Linnakangas wrote: On 02/11/2020 19:23, Andres Freund wrote: On 2020-11-02 11:03:29 +0200, Heikki

Re: Refactor pg_rewind code and make it work against a standby

2020-11-15 Thread Heikki Linnakangas
On 15/11/2020 09:07, Tom Lane wrote: I wrote: Not sure if you noticed, but piculet has twice failed the 007_standby_source.pl test that was added by 9c4f5192f: ... Now, I'm not sure what to make of that, but I can't help noticing that piculet uses --disable-atomics while francolin uses

Re: Online checksums patch - once again

2020-11-13 Thread Heikki Linnakangas
On 12/11/2020 15:17, Daniel Gustafsson wrote: On 5 Oct 2020, at 14:14, Heikki Linnakangas wrote: I would expect the checksums worker to be automatically started at postmaster startup. Can we make that happen? A dynamic background worker has to be registered from a regular backend, so it's

Re: [PATCH] Combine same ternary types in GIN and TSearch

2020-11-13 Thread Heikki Linnakangas
On 13/11/2020 11:04, Pavel Borisov wrote: Hi, hackers! For historical reasons, now we have two differently named but similar ternary data types in TSearch and Gin text-related types. Before v13 there was also Gin's private TS_execute() version, from which we eventually shifted to Tsearch's

pg_rewind copies

2020-11-13 Thread Heikki Linnakangas
uch more narrow window, though, and pg_basebackup has the same problem. - Heikki >From f0207e355c6be2d710d2ac11e4895795c3471ef3 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Fri, 13 Nov 2020 11:04:29 +0200 Subject: [PATCH 1/1] pg_rewind: Fetch small files according to new size. There's a

Re: Refactor pg_rewind code and make it work against a standby

2020-11-12 Thread Heikki Linnakangas
On 04/11/2020 11:23, Heikki Linnakangas wrote: I read through the patches one more time, fixed a bunch of typos and such, and pushed patches 1-4. I'm going to spend some more time on testing the last patch. It allows using a standby server as the source, and we don't have any tests for that yet

Re: ModifyTable overheads in generic plans

2020-11-11 Thread Heikki Linnakangas
I'm still a bit confused and unhappy about the initialization of ResultRelInfos and the various fields in them. We've made progress in the previous patches, but it's still a bit messy. /* * If transition tuples will be captured, initialize a map to convert

Re: making update/delete of inheritance trees scale better

2020-11-11 Thread Heikki Linnakangas
On 29/10/2020 15:03, Amit Langote wrote: Rebased over the recent executor result relation related commits. ModifyTablePath didn't get the memo that a ModifyTable can only have one subpath after these patches. Attached patch, on top of your v5 patches, cleans that up. - Heikki diff --git

Re: Clean up optional rules in grammar

2020-11-11 Thread Heikki Linnakangas
On 11/11/2020 11:12, Peter Eisentraut wrote: The attached patch cleans this up to make them all look like the first style. +1 - Heikki

Re: ModifyTable overheads in generic plans

2020-11-11 Thread Heikki Linnakangas
On 10/11/2020 17:32, Heikki Linnakangas wrote: On 10/11/2020 13:12, Amit Langote wrote: On second thought, it seems A would amount to merely a cosmetic adjustment of the API, nothing more. B seems to get the job done for me and also doesn't unnecessarily break compatibility, so I've updated

Re: ModifyTable overheads in generic plans

2020-11-10 Thread Heikki Linnakangas
On 10/11/2020 13:12, Amit Langote wrote: On Wed, Nov 4, 2020 at 11:32 AM Amit Langote wrote: On Tue, Nov 3, 2020 at 9:05 PM Heikki Linnakangas wrote: A) We could change FDW API so that BeginDirectModify takes the same arguments as BeginForeignModify(). That avoids the assumption that it's

Re: Protect syscache from bloating with negative cache entries

2020-11-06 Thread Heikki Linnakangas
code in memory so that a some hot codepath or struct gets split across CPU cache lines. It can be infuriating when benchmarking. At Thu, 5 Nov 2020 11:09:09 +0200, Heikki Linnakangas wrote in (A) original test patch I naively thought that the code path is too short to bury the degradation

Re: Protect syscache from bloating with negative cache entries

2020-11-05 Thread Heikki Linnakangas
On 19/11/2019 12:48, Kyotaro Horiguchi wrote: 1. Inserting a branch in SearchCatCacheInternal. (CatCache_Pattern_1.patch) This is the most straightforward way to add an alternative feature. pattern 1 | 8459.73 | 28.15 # 9% (>> 1%) slower than 7757.58 pattern 1 | 8504.83 | 55.61 pattern 1

Re: Refactor pg_rewind code and make it work against a standby

2020-11-04 Thread Heikki Linnakangas
On 25/09/2020 02:56, Soumyadeep Chakraborty wrote: On Thu, Sep 24, 2020 at 10:27 AM Heikki Linnakangas wrote: 7. Please address the FIXME for the symlink case: /* FIXME: Check if it points to the same target? */ It's not a new issue. Would be nice to fix, of course. I'm not sure what

Re: [PATCH] remove pg_archivecleanup and pg_standby

2020-11-03 Thread Heikki Linnakangas
On 02/11/2020 20:26, Justin Pryzby wrote: On Thu, Oct 29, 2020 at 08:40:31PM +0100, Michael Banck wrote: Am Mittwoch, den 28.10.2020, 21:44 -0500 schrieb Justin Pryzby: Forking this thread: https://www.postgresql.org/message-id/fd93f1c5-7818-a02c-01e5-1075ac0d4...@iki.fi I think these are

Re: Parallel copy

2020-11-03 Thread Heikki Linnakangas
On 03/11/2020 10:59, Amit Kapila wrote: On Mon, Nov 2, 2020 at 12:40 PM Heikki Linnakangas wrote: However, the point of parallel copy is to maximize bandwidth. Okay, but this first-phase (finding the line boundaries) can anyway be not done in parallel and we have seen in some of the initial

Re: ModifyTable overheads in generic plans

2020-11-03 Thread Heikki Linnakangas
On 03/11/2020 10:27, Amit Langote wrote: Please check the attached if that looks better. Great, thanks! Yeah, I like that much better. This makes me a bit unhappy: /* Also let FDWs init themselves for foreign-table result rels */ if

Re: Split copy.c

2020-11-02 Thread Heikki Linnakangas
On 02/11/2020 19:23, Andres Freund wrote: On 2020-11-02 11:03:29 +0200, Heikki Linnakangas wrote: There isn't much common code between COPY FROM and COPY TO, so I propose that we split copy.c into two: copyfrom.c and copyto.c. See attached. I thin that's much nicer. Not quite convinced that's

Re: Yet another fast GiST build

2020-11-02 Thread Heikki Linnakangas
On 30/10/2020 20:20, Andrey Borodin wrote: 27 окт. 2020 г., в 16:43, Heikki Linnakangas написал(а): static bool gbt_inet_abbrev_abort(int memtupcount, SortSupport ssup) { #if SIZEOF_DATUM == 8 return false; #else return true; #endif } Better to not set the 'abbrev_converter

Re: ModifyTable overheads in generic plans

2020-11-02 Thread Heikki Linnakangas
On 30/10/2020 08:13, Amit Langote wrote: /* * Perform WITH CHECK OPTIONS check, if any. */ static void ExecProcessWithCheckOptions(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, TupleTableSlot *slot, WCOKind wco_kind) {

Re: hash_array_extended() needs to pass down collation

2020-11-02 Thread Heikki Linnakangas
On 02/11/2020 09:40, Peter Eisentraut wrote: I noticed that hash_array_extended() does not pass down the collation to the element's collation function, unlike hash_array(). As a consequence, hash partitioning using text arrays as partition key fails. The attached patch fixes this. I propose

Re: Getting rid of aggregate_dummy()

2020-11-02 Thread Heikki Linnakangas
On 01/11/2020 22:47, Tom Lane wrote: With that, we don't actually need aggregate_dummy() to exist at all, because it's never referenced. Having "aggregate_dummy" as the prosrc value for an aggregate function is now just a random convention; any other string would do as well. (We could save a

Re: [patch] [doc] Add SELECT clause literals to queries section headers

2020-11-02 Thread Heikki Linnakangas
On 21/10/2020 18:04, David G. Johnston wrote: Hackers, Re-sending from -docs [1] with attachment in order to add to commitfest. David J. [1] https://www.postgresql.org/message-id/flat/159981394174.31338.7014519396749859167%40wrigleys.postgresql.org edit: attaching the patch Seems like a

Re: Fix a typo in verify_heapam.c

2020-11-02 Thread Heikki Linnakangas
On 02/11/2020 10:05, Masahiko Sawada wrote: Hi all, I've attached a patch that fixes a typo in verify_heapam.c recently added. s/wheter/whether/ This was just fixed in commit 8a15e735b. - Heikki

Re: Parallel copy

2020-11-01 Thread Heikki Linnakangas
On 02/11/2020 09:10, Heikki Linnakangas wrote: On 02/11/2020 08:14, Amit Kapila wrote: We have discussed both these approaches (a) single producer multiple consumer, and (b) all workers doing the processing as you are saying in the beginning and concluded that (a) is better, see some

Re: Parallel copy

2020-11-01 Thread Heikki Linnakangas
On 02/11/2020 08:14, Amit Kapila wrote: On Fri, Oct 30, 2020 at 10:11 PM Heikki Linnakangas wrote: Leader process: The leader process is simple. It picks the next FREE buffer, fills it with raw data from the file, and marks it as FILLED. If no buffers are FREE, wait. Worker process: 1

Re: making update/delete of inheritance trees scale better

2020-10-30 Thread Heikki Linnakangas
On 31/10/2020 00:12, Tom Lane wrote: Heikki Linnakangas writes: But if you do: postgres=# explain verbose update tab set a = 1, b = 2; QUERY PLAN - Update

Re: Parallel copy

2020-10-30 Thread Heikki Linnakangas
On 30/10/2020 22:56, Tomas Vondra wrote: I agree this design looks simpler. I'm a bit worried about serializing the parsing like this, though. It's true the current approach (where the first phase of parsing happens in the leader) has a similar issue, but I think it would be easier to improve

Re: making update/delete of inheritance trees scale better

2020-10-30 Thread Heikki Linnakangas
On 30/10/2020 23:10, Tom Lane wrote: Heikki Linnakangas writes: I also did some quick performance testing with a simple update designed as a worst-case scenario: vacuum tab; update tab set b = b, a = a; In this case, the patch fetches the old tuple, but it wouldn't really need

Re: making update/delete of inheritance trees scale better

2020-10-30 Thread Heikki Linnakangas
On 29/10/2020 15:03, Amit Langote wrote: On Sun, Oct 4, 2020 at 11:44 AM Amit Langote wrote: On Fri, Sep 11, 2020 at 7:20 PM Amit Langote wrote: Here are the commit messages of the attached patches: [PATCH v3 1/3] Overhaul how updates compute a new tuple I tried to assess the performance

Re: Parallel copy

2020-10-30 Thread Heikki Linnakangas
On 30/10/2020 18:36, Heikki Linnakangas wrote: Whether the leader process finds the EOLs or the worker processes, it's pretty clear that it needs to be done ASAP, for a chunk at a time, because that cannot be done in parallel. I think some refactoring in CopyReadLine() and friends would

Re: Parallel copy

2020-10-30 Thread Heikki Linnakangas
On 30/10/2020 18:36, Heikki Linnakangas wrote: I find this design to be very complicated. Why does the line-boundary information need to be in shared memory? I think this would be much simpler if each worker grabbed a fixed-size block of raw data, and processed that. In your patch, the leader

Re: Parallel copy

2020-10-30 Thread Heikki Linnakangas
On 27/10/2020 15:36, vignesh C wrote: Attached v9 patches have the fixes for the above comments. I find this design to be very complicated. Why does the line-boundary information need to be in shared memory? I think this would be much simpler if each worker grabbed a fixed-size block of raw

Re: Parallel copy

2020-10-29 Thread Heikki Linnakangas
On 27/10/2020 15:36, vignesh C wrote: Attached v9 patches have the fixes for the above comments. I did some testing: /tmp/longdata.pl: #!/usr/bin/perl # # Generate three rows: # foo # longdatalongdatalongdata... # bar # # The length of the middle row is given as command line arg. #

Re: Deduplicate aggregates and transition functions in planner

2020-10-29 Thread Heikki Linnakangas
On 28/10/2020 21:59, Andres Freund wrote: On 2020-10-28 21:10:41 +0200, Heikki Linnakangas wrote: Currently, ExecInitAgg() performs quite a lot of work, to deduplicate identical Aggrefs, as well as Aggrefs that can share the same transition state. That doesn't really belong in the executor, we

Deduplicate aggregates and transition functions in planner

2020-10-28 Thread Heikki Linnakangas
, or change the query to still get the old plan. - Heikki >From 0100f5e477af814afa23bf7933393f4b144b835f Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 28 Oct 2020 20:58:14 +0200 Subject: [PATCH v1 1/1] Move peragg and pertrans duplicate finding to prepagg.c. This has the advant

Re: Support for NSS as a libpq TLS backend

2020-10-27 Thread Heikki Linnakangas
On 27/10/2020 22:07, Daniel Gustafsson wrote: /* * Track whether the NSS database has a password set or not. There is no API * function for retrieving password status, so we simply flip this to true in * case NSS invoked the password callback - as that will only happen in case * there is a

Re: partition routing layering in nodeModifyTable.c

2020-10-27 Thread Heikki Linnakangas
On 23/10/2020 12:37, Amit Langote wrote: To explain these numbers a bit, "overheaul update/delete processing" patch improves the performance of that benchmark by allowing the updates to use run-time pruning when executing generic plans, which they can't today. However without

Re: Yet another fast GiST build

2020-10-27 Thread Heikki Linnakangas
On 21/10/2020 20:13, Andrey Borodin wrote: 7 окт. 2020 г., в 17:38, Heikki Linnakangas написал(а): On 07/10/2020 15:27, Andrey Borodin wrote: Here's draft patch with implementation of sortsupport for ints and floats. +static int +gbt_int4_cmp(Datum a, Datum b, SortSupport ssup

Re: Question about make coverage-html

2020-10-27 Thread Heikki Linnakangas
On 27/10/2020 10:09, Peter Smith wrote: Hi hackers. The example of test coverage in the documentation [1] works as advertised. But I wanted to generate test coverage results only of some TAP tests in src/test/subscription. The documentation [1] also says "The make commands also work in

Re: [PoC] Non-volatile WAL buffer

2020-10-26 Thread Heikki Linnakangas
I had a new look at this thread today, trying to figure out where we are. I'm a bit confused. One thing we have established: mmap()ing WAL files performs worse than the current method, if pg_wal is not on a persistent memory device. This is because the kernel faults in existing content of

Re: select_common_typmod

2020-10-26 Thread Heikki Linnakangas
On 20/10/2020 11:58, Peter Eisentraut wrote: While working on another patch, I figured adding a select_common_typmod() to go along with select_common_type() and select_common_collation() would be handy. Typmods were previously combined using hand-coded logic in several places, and not at all in

Re: [doc] remove reference to pg_dump pre-8.1 switch behaviour

2020-10-26 Thread Heikki Linnakangas
On 25/10/2020 23:56, Justin Pryzby wrote: On Fri, Oct 23, 2020 at 11:09:26PM +0300, Heikki Linnakangas wrote: Findings in detail follow. Are you working on a patch for these ? I pushed the patch I included in that email now, to remove the most clear cases. I'm not planning to do anything

Re: bulk typos

2020-10-25 Thread Heikki Linnakangas
On 25/10/2020 21:48, Justin Pryzby wrote: On Sat, Mar 31, 2018 at 05:56:40AM -0500, Justin Pryzby wrote: I needed another distraction so bulk-checked for typos, limited to comments in *.[ch]. I'm not passionate about this, but it serves the purpose of reducing the overhead of fixing them

Re: [doc] remove reference to pg_dump pre-8.1 switch behaviour

2020-10-23 Thread Heikki Linnakangas
On 23/10/2020 17:51, Tom Lane wrote: But anyway, this was about documentation not code. What I'm wondering about is when to drop things like, say, this bit in the regex docs: Two significant incompatibilities exist between AREs and the ERE syntax recognized by pre-7.4 releases of

Re: [doc] remove reference to pg_dump pre-8.1 switch behaviour

2020-10-23 Thread Heikki Linnakangas
On 06/10/2020 15:15, Ian Lawrence Barwick wrote: 2020年10月6日(火) 21:13 Ian Lawrence Barwick : Hi The pg_dump doc page [1], under the -t/--table option, contains a Note documenting the behavioural differences introduced in PostgreSQL 8.2. As it's been almost exactly 14 years since that note was

Re: Parallel copy

2020-10-23 Thread Heikki Linnakangas
I had a brief look at at this patch. Important work! A couple of first impressions: 1. The split between patches 0002-Framework-for-leader-worker-in-parallel-copy.patch and 0003-Allow-copy-from-command-to-process-data-from-file.patch is quite artificial. All the stuff introduced in the first

Re: partition routing layering in nodeModifyTable.c

2020-10-23 Thread Heikki Linnakangas
On 22/10/2020 16:49, Amit Langote wrote: On Tue, Oct 20, 2020 at 9:57 PM Amit Langote wrote: On Mon, Oct 19, 2020 at 8:55 PM Heikki Linnakangas wrote: It's probably true that there's no performance gain from initializing them more lazily. But the reasoning and logic around the initialization

Re: partition routing layering in nodeModifyTable.c

2020-10-23 Thread Heikki Linnakangas
On 23/10/2020 05:56, Amit Langote wrote: On Thu, Oct 22, 2020 at 11:25 PM Alvaro Herrera wrote: On 2020-Oct-22, Amit Langote wrote: 0001 fixes a thinko of the recent commit 1375422c782 that I discovered when debugging a problem with 0003. Hmm, how hard is it to produce a test case that

Re: Probable documentation errors or improvements

2020-10-19 Thread Heikki Linnakangas
On 05/10/2020 22:19, Justin Pryzby wrote: On Fri, Sep 25, 2020 at 09:30:00AM -0500, Justin Pryzby wrote: Split one patch about text search, added another one (sequences), added some info to commit messages, and added here. https://commitfest.postgresql.org/30/2744/ Added an additional patch

Re: Use PointerGetDatum(cstring_to_text_with_len()) instead of CStringGetTextDatum() to avoid duplicate strlen

2020-10-19 Thread Heikki Linnakangas
On 19/10/2020 09:32, Hou, Zhijie wrote: Hi I found some code like the following: StringInfoData s; ... values[6] = CStringGetTextDatum(s.data); The length of string can be found in ' StringInfoData.len', but the macro CStringGetTextDatum will use strlen to count the length again. I think we

Re: partition routing layering in nodeModifyTable.c

2020-10-19 Thread Heikki Linnakangas
On 17/10/2020 18:54, Alvaro Herrera wrote: On 2020-Oct-17, Amit Langote wrote: As I said in my previous email, I don't see how we can make initializing the map any lazier than it already is. If a partition has a different tuple descriptor than the root table, then we know for sure that any

Re: partition routing layering in nodeModifyTable.c

2020-10-19 Thread Heikki Linnakangas
On 19/10/2020 07:54, Amit Langote wrote: On Sun, Oct 18, 2020 at 12:54 AM Alvaro Herrera wrote: On 2020-Oct-17, Amit Langote wrote: Hmm, I don't see ri_PartitionCheckExpr as being a piece of routing information, because it's primarily meant to be used when inserting *directly* into a

Re: partition routing layering in nodeModifyTable.c

2020-10-15 Thread Heikki Linnakangas
On Wed, Oct 14, 2020 at 6:04 PM Heikki Linnakangas wrote: I'll continue with the last couple of patches in this thread. I committed the move of the cross-partition logic to new ExecCrossPartitionUpdate() function, with just minor comment editing and pgindent. I left out the refactoring

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Heikki Linnakangas
On 14/10/2020 21:58, Tom Lane wrote: I noticed that chipmunk failed [1] with a rather interesting log: 2020-10-14 08:57:01.661 EEST [27048:6] pg_regress/prepared_xacts LOG: statement: UPDATE pxtest1 SET foobar = 'bbb' WHERE foobar = 'aaa'; 2020-10-14 08:57:01.721 EEST [27048:7]

Re: partition routing layering in nodeModifyTable.c

2020-10-14 Thread Heikki Linnakangas
On 14/10/2020 09:44, Amit Langote wrote: I like the idea of storing the ResultRelInfo in ForeignScanState, but it would be better if we can document the fact that an FDW may not reliably access until IterateDirectModify(). That's because, setting it in ExecInitForeignScan() will mean *all*

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-10-14 Thread Heikki Linnakangas
On 14/10/2020 06:29, Michael Paquier wrote: With 0001 in place, switching the SHA2 implementation of OpenSSL to use EVP is straight-forward, as the only thing that's actually needed here is to put in place a callback to clean up the EVP contexts allocated by OpenSSL. This is rather similar to

Re: partition routing layering in nodeModifyTable.c

2020-10-13 Thread Heikki Linnakangas
On 13/10/2020 19:09, Heikki Linnakangas wrote: One little idea I had: I think all FDWs that support direct modify will have to carry the resultRelaton index or the ResultRelInfo pointer from BeginDirectModify to IterateDirectModify in the FDW's private struct. It's not complicated, but should

Re: partition routing layering in nodeModifyTable.c

2020-10-13 Thread Heikki Linnakangas
On 13/10/2020 15:03, Amit Langote wrote: On Tue, Oct 13, 2020 at 7:13 PM Heikki Linnakangas wrote: Ok, committed. I'll continue to look at the rest of the patches in this patch series now. I've reviewed the next two patches in the series, they are pretty much ready for commit now. I made

Re: Remove unnecessary else branch

2020-10-13 Thread Heikki Linnakangas
On 13/10/2020 16:30, Li Japin wrote: Hi, I found in guc-file.l we can omit the else branch in AbsoluteConfigLocation(). It will compile the same, so it's just a matter of code readability or taste which style is better here. I think we should leave it alone, it's fine as it is. - Heikki

Re: partition routing layering in nodeModifyTable.c

2020-10-13 Thread Heikki Linnakangas
On 13/10/2020 07:32, Amit Langote wrote: On Tue, Oct 13, 2020 at 1:57 AM Heikki Linnakangas wrote: It occurred to me that if we do that (initialize the array lazily), there's very little need for the PlannedStmt->resultRelations list anymore. It's only used in ExecRelationIsTargetRelat

Re: partition routing layering in nodeModifyTable.c

2020-10-12 Thread Heikki Linnakangas
On 12/10/2020 16:47, Amit Langote wrote: On Mon, Oct 12, 2020 at 8:12 PM Heikki Linnakangas wrote: 1. We have many different cleanup/close routines now: ExecCloseResultRelations, ExecCloseRangeTableRelations, and ExecCleanUpTriggerState. Do we need them all? It seems to me that we could merge

Re: archive status ".ready" files may be created too early

2020-10-12 Thread Heikki Linnakangas
On 07/07/2020 12:02, matsumura@fujitsu.com wrote: At Monday, July 6, 2020 05:13:40 +, "Kyotaro Horiguchi " wrote in after WAL buffer is filled up to the requested position. So when it crosses segment boundary we know the all past corss segment-boundary records are stable. That means

Re: partition routing layering in nodeModifyTable.c

2020-10-12 Thread Heikki Linnakangas
On 09/10/2020 11:01, Amit Langote wrote: On Thu, Oct 8, 2020 at 9:35 PM Amit Langote wrote: On Wed, Oct 7, 2020 at 9:07 PM Heikki Linnakangas wrote: On 07/10/2020 12:50, Amit Langote wrote: I have thought about something like this before. An idea I had is to make es_result_relations array

Re: Yet another fast GiST build

2020-10-07 Thread Heikki Linnakangas
On 07/10/2020 15:27, Andrey Borodin wrote: Here's draft patch with implementation of sortsupport for ints and floats. +static int +gbt_int4_cmp(Datum a, Datum b, SortSupport ssup) +{ + int32KEY *ia = (int32KEY *) DatumGetPointer(a); + int32KEY *ib = (int32KEY *)

Re: partition routing layering in nodeModifyTable.c

2020-10-07 Thread Heikki Linnakangas
On 07/10/2020 12:50, Amit Langote wrote: On Tue, Oct 6, 2020 at 12:45 AM Heikki Linnakangas wrote: It would be better to set it in make_modifytable(), just after calling PlanDirectModify(). Actually, that's how it was done in earlier iterations but I think I decided to move

Re: Yet another fast GiST build

2020-10-06 Thread Heikki Linnakangas
On 06/10/2020 14:05, Pavel Borisov wrote: I've been making tests with memory sanitizer Thanks! and got one another error in regression test create_index: CREATE INDEX gpointind ON point_tbl USING gist (f1); server closed the connection unexpectedly with logfile: gistproc.c:1714:28: runtime

Re: partition routing layering in nodeModifyTable.c

2020-10-05 Thread Heikki Linnakangas
On 13/07/2020 08:47, Amit Langote wrote: It's been over 11 months since there was any significant commentary on the contents of the patches themselves, so perhaps I should reiterate what the patches are about and why it might still be a good idea to consider them. The thread started with some

Re: Online checksums patch - once again

2020-10-05 Thread Heikki Linnakangas
On 05/10/2020 17:25, Álvaro Herrera wrote: On 2020-Oct-05, Heikki Linnakangas wrote: The code in sendFile() in basebackup.c seems suspicious in that regard. It calls DataChecksumsNeedVerify() once before starting to read the file. Isn't it possible for the checksums flag to change while it's

Re: Online checksums patch - once again

2020-10-05 Thread Heikki Linnakangas
Replying to an older message in this thread: + /* + * If we reach this point with checksums in inprogress state, we notify + * the user that they need to manually restart the process to enable + * checksums. This is because we cannot launch a dynamic background worker + * directly from here, it

Re: Online checksums patch - once again

2020-10-05 Thread Heikki Linnakangas
I looked at patch v22, and I can see two main issues: 1. The one that Robert talked about earlier: A backend checks the local "checksums" state. If it's 'off', it writes a page without checksums. How do you guarantee that the local state doesn't change in between? The implicit assumption

Re: Prepared Statements

2020-10-05 Thread Heikki Linnakangas
On 02/10/2020 23:10, Patrick REED wrote: Hi, I am having a hard time pinning down which function creates a prepared statement. Say in some language I create a Prepared Statement and send it off. Before the first time I execute the prepared statement, which function is the one that 'creates'

Re: Error code missing for "wrong length of inner sequence" error

2020-10-02 Thread Heikki Linnakangas
On 02/10/2020 15:06, Daniel Gustafsson wrote: On 2 Oct 2020, at 13:44, Heikki Linnakangas wrote: On 01/10/2020 14:21, Daniel Gustafsson wrote: On that note, wouldn't the dimension check errors in PLySequence_ToArray be just as well off using normal ereport()'s? Only one of them seem to error

Re: Error code missing for "wrong length of inner sequence" error

2020-10-02 Thread Heikki Linnakangas
On 01/10/2020 14:21, Daniel Gustafsson wrote: On 1 Oct 2020, at 12:54, Heikki Linnakangas wrote: Most checks when converting between SQL and Python types use the PLy_elog() function, which uses the genericc ERRCODE_EXTERNAL_ROUTINE_EXCEPTION error code, but I think

Error code missing for "wrong length of inner sequence" error

2020-10-01 Thread Heikki Linnakangas
In PLySequence_ToArray_recurse(), there's this check: if (PySequence_Length(list) != dims[dim]) ereport(ERROR, (errmsg("wrong length of inner sequence: has length %d, but %d was expected",

Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64

2020-09-30 Thread Heikki Linnakangas
On 30/09/2020 19:04, Zidenberg, Tsahi wrote: Ubuntu 20.04 even turned it off by default for gcc-10, which seems like a compatibility step with the main gcc-9 compiler. Ok, I definitely don't want to override that decision. I'm marking this as Rejected in the commitfest. But thanks for the

Re: [PATCH] Add section headings to index types doc

2020-09-30 Thread Heikki Linnakangas
On 30/09/2020 14:25, Dagfinn Ilmari Mannsåker wrote: Michael Paquier writes: On Mon, Aug 10, 2020 at 12:52:17PM +, Jürgen Purtz wrote: The new status of this patch is: Waiting on Author This has not been answered yet, so I have marked the patch as returned with feedback. Updated

Re: pgbench - refactor init functions with buffers

2020-09-30 Thread Heikki Linnakangas
On 09/07/2020 10:05, Fabien COELHO wrote: in favor of *PQExpBuffer(). Attached v7 is rebased v5 which uses PQExpBuffer, per cfbot. Thanks! I pushed this with small changes: - I left out the changes to executeStatement(). I'm not quite convinced it's a good idea or worth it, and it's

Re: Yet another fast GiST build

2020-09-29 Thread Heikki Linnakangas
On 29/09/2020 21:04, Andrey M. Borodin wrote: 28 сент. 2020 г., в 13:12, Heikki Linnakangas написал(а): I did some testing with your patch. It seems that the rightlinks are still not always set. I didn't try to debug why. Yes, there is a bug. Now it seems to me so obvious, yet it took some

Re: Corner-case bug in pg_rewind

2020-09-29 Thread Heikki Linnakangas
On 11/09/2020 09:42, Ian Barwick wrote: Take the following cluster with: - node1 (initial primary) - node2 (standby) - node3 (standby) Following activity takes place (greatly simplified from a real-world situation): 1. node1 is shut down. 2. node3 is promoted 3. node2 is attached to

Re: Improved Cost Calculation for IndexOnlyScan

2020-09-29 Thread Heikki Linnakangas
On 29/09/2020 11:49, Hamid Akhtar wrote: So, not actually random replacement here, rather a change with baserel->allvisfrac taken into consideration (as given below): index_random_page_cost = Min(spc_seq_page_cost + spc_random_page_cost * (1.0 - baserel->allvisfrac),

Re: Improved Cost Calculation for IndexOnlyScan

2020-09-29 Thread Heikki Linnakangas
On 29/09/2020 10:06, Hamid Akhtar wrote: In one of my earlier emails [1], I mentioned that there seems to be a problem with how the cost for index only scans is being calculated. [1] https://www.postgresql.org/message-id/CANugjhsnh0OBMOYc7qKcC%2BZsVvAXCeF7QiidLuFvg6zmHy1C7A%40mail.gmail.com

Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64

2020-09-29 Thread Heikki Linnakangas
On 10/09/2020 09:37, Zidenberg, Tsahi wrote: On 08/09/2020, 1:01, "Tom Lane" wrote: > I wonder what version of gcc you intend this for. AFAICS, older > gcc versions lack this flag at all, while newer ones have it on > by default. (previously sent private reply, sorry) The

Re: Concurrency issue in pg_rewind

2020-09-29 Thread Heikki Linnakangas
On 18/09/2020 10:17, Alexander Kukushkin wrote: At the same time, pg_rewind due to such "fatal" error leaves PGDATA in an inconsistent state with empty pg_control file, this is totally bad and easily fixable. We want the specific file to be absent and it is already absent, why should it be a

Re: Yet another fast GiST build

2020-09-28 Thread Heikki Linnakangas
ere are two. - Heikki >From c388e458b196454d3535d84f6f7a617f0f3f819a Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Mon, 28 Sep 2020 11:01:45 +0300 Subject: [PATCH 1/1] Add functions to 'pageinspect' to inspect GiST indexes. --- contrib/pageinspect/Makefile | 4 +- contrib/pageinspec

Re: Yet another fast GiST build

2020-09-28 Thread Heikki Linnakangas
On 21/09/2020 16:29, Andrey M. Borodin wrote: But thing that bothers me now: when we vacuum leaf page, we bump it's NSN. But we do not bump internal page LSN. Does this means we will follow rightlinks after vacuum? It seems superflous. Sorry, I did not understand what you said above. Vacuum

Re: Refactor pg_rewind code and make it work against a standby

2020-09-24 Thread Heikki Linnakangas
not fully consistent with that either. I'll try to make it more consistent. Thanks for the review! Attached is a new version of the patch set. - Heikki >From ddc02fb949811c36c94bd42520ecdcb672493397 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Thu, 24 Sep 2020 12:23:39 +0300 Subject:

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Heikki Linnakangas
On 24/09/2020 17:21, Daniel Gustafsson wrote: If we really want to support it (which would require more evidence of it being a problem IMO), using the non-OpenSSL sha256 code would be one option I guess? That would technically work, but wouldn't it make the product as whole not FIPS

Re: Refactor pg_rewind code and make it work against a standby

2020-09-24 Thread Heikki Linnakangas
Thanks for the review! I'll post a new version shortly, with your comments incorporated. More detailed response to a few of them below: On 18/09/2020 10:41, Kyotaro Horiguchi wrote: I don't think filemap_finalize needs to iterate over filemap twice. True, but I thought it's more clear that

Re: Yet another fast GiST build

2020-09-21 Thread Heikki Linnakangas
On 21/09/2020 12:06, Andrey M. Borodin wrote: 21 сент. 2020 г., в 13:45, Heikki Linnakangas написал(а): Actually, don't we have a problem with that, even before this patch? Even though we set the LSN to the magic GistBuildLSN value when we build the index, WAL replay will write the LSN

Re: Yet another fast GiST build

2020-09-21 Thread Heikki Linnakangas
On 21/09/2020 02:06, Tom Lane wrote: Justin Pryzby writes: This also appears to break checksums. Fixed, thanks for the report! I was wondering about that, because the typical pattern for use of smgrextend for indexes seems to be RelationOpenSmgr(rel);

Re: Yet another fast GiST build

2020-09-21 Thread Heikki Linnakangas
On 21/09/2020 11:08, Heikki Linnakangas wrote: I think they need to, so that they can stamp the page with the LSN of the WAL record. But GiST build is special in that regard, because it stamps all pages with GistBuildLSN. Actually, don't we have a problem with that, even before this patch

<    2   3   4   5   6   7   8   9   10   11   >