Re: inconsistency and inefficiency in setup_conversion()

2018-10-02 Thread John Naylor
On 10/2/18, Michael Paquier wrote: > v4 does not apply anymore. I am moving this patch to next commit fest, > waiting on author. v5 attached. -John Naylor From ea0a180bde325b0383ce7f0b3d48d1ce9e941393 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Mon, 2 Jul 2018 12:52:07 +0700 S

Re: Sync ECPG scanner with core

2018-10-01 Thread John Naylor
On 9/30/18, Michael Paquier wrote: > It would be nice to add this patch to the next commit fest: > https://commitfest.postgresql.org/20/ Done, thanks. -John Naylor

Re: Sync ECPG scanner with core

2018-10-02 Thread John Naylor
It seems the pach tester is confused by the addition of the demonstration diff file. I'm reattaching just the patchset to see if it turns green. -John Naylor From 107e3c8a0b65b0196ea4370a724c8b2a1b0fdf79 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Sun, 30 Sep 2018 12:51:41 +0700 Subject

Re: WIP: Avoid creation of the free space map for small tables

2018-10-07 Thread John Naylor
On 10/6/18, Thomas Munro wrote: > On Sat, Oct 6, 2018 at 7:47 AM John Naylor wrote: >> A while back, Robert Haas noticed that the space taken up by very >> small tables is dominated by the FSM [1]. Tom suggested that we could >> prevent creation of the FSM until the heap

Re: generating bootstrap entries for array types

2018-09-20 Thread John Naylor
On 9/20/18, Dagfinn Ilmari Mannsåker wrote: > Hi again John, Hi Ilmari, My apologies -- your messages ended up in my spam folder. I only saw this thread again when Tom replied. Thanks for the review! I'll keep a look out for any future messages. -John Naylor

Re: generating bootstrap entries for array types

2018-09-20 Thread John Naylor
On 9/21/18, Tom Lane wrote: > I did some other minor hacking (mostly, fixing the documentation) > and pushed it. Thank you. -John Naylor

Re: doc - add missing documentation for "acldefault"

2018-09-19 Thread John Naylor
On 9/19/18, Tom Lane wrote: > However, I don't object to documenting any function that has its > own pg_description string. Speaking of, that description string seems to have been neglected. I've attached a remedy for that. -John Naylor diff --git a/src/include/catalog/pg_proc.dat b/src/i

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-27 Thread John Naylor
VED_KEYWORD) PG_KEYWORD("select", OPTIMIZE, SELECT, RESERVED_KEYWORD) with a warning emitted if more than one keyword per range has OPTIMIZE. That would require all keyword lists to have that second argument, but selecting a preferred keyword would be optional. -John Naylor

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-27 Thread John Naylor
On 12/26/18, John Naylor wrote: > On 12/26/18, Robert Haas wrote: >> I wonder if we could do something really simple like a lookup based on >> the first character of the scan keyword. It looks to me like there are >> 440 keywords right now, and the most common starting le

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-29 Thread John Naylor
like this: Usage: gen_keywords.pl [--output/-o ] [--prefix/-p ] input_file --output Output directory --prefix String prepended to var names in the output file On 12/27/18, Andrew Dunstan wrote: > I would rather we used the standard perl module Getopt::Long, as > numerous programs we have a

Re: WIP: Avoid creation of the free space map for small tables

2019-01-16 Thread John Naylor
On Wed, Jan 16, 2019 at 8:41 AM Amit Kapila wrote: > > On Fri, Jan 11, 2019 at 3:54 AM John Naylor > wrote: > 1. > Commit message: > > Any pages with wasted free space become visible at next relation extension, > > so we still control table bloat. > > I think

Re: WIP: Avoid creation of the free space map for small tables

2019-01-16 Thread John Naylor
On Wed, Jan 16, 2019 at 11:40 AM John Naylor wrote: On Wed, Jan 16, 2019 at 8:41 AM Amit Kapila wrote: can use a macro for the same? I have changed this in the attached patch, see what you think about it. I have used it at a few other places as well. The macro adds clarity, so I'm

Re: explain plans with information about (modified) gucs

2019-01-16 Thread John Naylor
> [v5] Hi Tomas, Peter suggested upthread to use 'settings' rather than 'gucs' for the explain option (and output?), and you seemed to agree. Are you going to include that in a future version? Speaking of the output, v5's default text doesn't match that of formatted text ('GUCs' / 'GUC').

Re: WIP: Avoid creation of the free space map for small tables

2019-01-17 Thread John Naylor
acro. One minor detail added: use uint8 rather than char for the local map array. This seems to be preferred, especially in this file. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training &

Re: Delay locking partitions during INSERT and UPDATE

2019-01-18 Thread John Naylor
32000tps 10k partitions / master: 82tps 10k partitions / patch: 7000tps So far I haven't gotten quite as good performance as you and Tomas, although it's still a ~85x improvement. -John Naylor

Re: Delay locking partitions during INSERT and UPDATE

2019-01-19 Thread John Naylor
On Sat, Jan 19, 2019 at 10:59 AM Tomas Vondra wrote: > > On 1/19/19 12:05 AM, John Naylor wrote: > > I used a similar test, but with unlogged tables, and "-c 2", and got: > > > > normal table: 32000tps > > 10k partitions / master: 82tps > > 10k partit

Re: Speeding up text_position_next with multibyte encodings

2019-01-14 Thread John Naylor
5d95c5c-1fb7-29ea-1c5d-e7e941a0a14d%40iki.fi -- -John Naylor

Re: explain plans with information about (modified) gucs

2019-01-21 Thread John Naylor
. I made a pass through the config parameters, and don't see anything obviously left out. I have no comments on the source code. One thing stands out: For the docs on auto_explain, all other options have "Only superusers can change this setting.", but log_settings doesn't. -- John

Re: WIP: Avoid creation of the free space map for small tables

2019-01-21 Thread John Naylor
ause the upgraded secondary is non-functional, I can't really answer your question. Not sure if this is normal, but the pg_upgraded new cluster no longer had the replication slot. Re-adding it didn't allow my upgraded secondary to go into recovery, either. (I made sure to copy the reco

is there a reason we don't xlog index free space?

2019-01-21 Thread John Naylor
I knew that XLogRecordPageWithFreeSpace() is only called by heapam.c, but I figured indexes had their own logic to xlog free space. While investigating something else I found that brin indexes, at least, on the secondary have no FSMs. Is this deliberate? -- John Naylorhttps://www

Re: WIP: Avoid creation of the free space map for small tables

2019-01-23 Thread John Naylor
I think the upgrade patch is generally going on track but > might need some more review. The pg_upgrade piece is a nice-to-have feature and not essential, so can go in later. Additional review is also welcome. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-01-23 Thread John Naylor
s --size-only --no-inc-recursive data1 data2 standby # result ls standby/data1 ls standby/data2 The result is that foo_fsm is not copied to standby/data2, contrary to what the docs above imply for other unlinked files. Can anyone shed light on this? -- John Naylorhttps://www.2

Re: WIP: Avoid creation of the free space map for small tables

2019-01-19 Thread John Naylor
On Sat, Jan 19, 2019 at 8:06 AM Amit Kapila wrote: > > On Thu, Jan 17, 2019 at 11:13 PM John Naylor > Few more comments: > 1. > I think we should not allow to create FSM for toast tables as well > till there size reaches HEAP_FSM_CREATION_THRESHOLD. If you try below > test,

Re: WIP: Avoid creation of the free space map for small tables

2018-12-12 Thread John Naylor
l-case code in md.c to create any file if it's opened in bootstrap mode. I removed this and a similar special case (attached), and make check still passes. After digging through the history, I'm guessing this has been useless code since about 2001, when certain special catalogs were removed. -Joh

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-18 Thread John Naylor
On 12/17/18, Tom Lane wrote: > John Naylor writes: >> Since PL/pgSQL uses the core scanner, we'd need to use offsets in its >> reserved_keywords[], too. Those don't change much, so we can probably >> get away with hard-coding the offsets and the giant string in tha

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-21 Thread John Naylor
aintainer-clean but > not distclean. I think this also has some implications for VPATH > builds, but as long as you follow the pattern used for other > derived header files (e.g. fmgroids.h), you should be fine. Done. I also blindly added support for MSVC. -John Naylor src/common/keywor

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-23 Thread John Naylor
On 12/22/18, Tom Lane wrote: > John Naylor writes: >> Using a single file also gave me another idea: Take value and category >> out of ScanKeyword, and replace them with an index into another array >> containing those, which will only be accessed in the event of a hit.

Re: Speeding up text_position_next with multibyte encodings

2018-12-26 Thread John Naylor
On 12/22/18, Heikki Linnakangas wrote: > On 14/12/2018 20:20, John Naylor wrote: > I'm afraid that script doesn't work as a performance test. The > position() function is immutable, so the result gets cached in the plan > cache. All you're measuring is the speed to get the c

Re: automatically assigning catalog toast oids

2018-12-11 Thread John Naylor
On 12/11/18, Andres Freund wrote: > I've attached a patch implementing that. I'm not particularly in love > with FirstGenbkiObjectId as the symbol, but I couldn't think of > something more descriptive. How about FirstAutoAssignedObjectId? -John Naylor

reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-16 Thread John Naylor
o pl_reserved_kwlist.h and reuse the above tooling to generate pl_reserved_kwlist_{offset,string}.h, but that's more complex.) The rest should be just a SMOP. Any issues I left out? -John Naylor

Re: inconsistency and inefficiency in setup_conversion()

2018-12-14 Thread John Naylor
ostgresql.org/message-id/CAJVSVGWtUqxpfAaxS88vEGvi%2BjKzWZb2EStu5io-UPc4p9rSJg%40mail.gmail.com --John Naylor From bdbdb36129708d1d417f8db9009b377c3d4e3e90 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Fri, 14 Dec 2018 15:19:54 -0500 Subject: [PATCH v8 1/2] Add pg_language lookup. This didn't

Re: Speeding up text_position_next with multibyte encodings

2018-12-14 Thread John Naylor
On 12/14/18, John Naylor wrote: > I signed up to be a reviewer, and I will be busy next month, so I went > ahead and fixed the typo in the patch that broke assert-enabled > builds. While at it, I standardized on the spelling "start_ptr" in a > few places to match the rest o

Re: automatically assigning catalog toast oids

2018-12-14 Thread John Naylor
as I can see: "pg_database: it's a feature, not a bug, that template1 is not pinned." -John Naylor

Re: Speeding up text_position_next with multibyte encodings

2018-12-14 Thread John Naylor
rings, and 2-3% faster on long strings. It seems the worst case is not that bad, and could be optimized, as Heikki said. -John Naylor From ca7a228174acb8b85b87642bb6df182139587c05 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Sun, 18 Nov 2018 17:40:09 +0700 Subject: [PATCH v2] Use single-byte Boyer

Re: WIP: Avoid creation of the free space map for small tables

2018-11-30 Thread John Naylor
On 11/29/18, Amit Kapila wrote: > On Thu, Nov 29, 2018 at 3:07 PM John Naylor wrote: >> Done. I tried adding it to several schedules, but for some reason >> vacuuming an empty table failed to truncate the heap to 0 blocks. >> Putting the test in its own group fixed the prob

Re: WIP: Avoid creation of the free space map for small tables

2018-12-07 Thread John Naylor
On 12/6/18, Amit Kapila wrote: > On Thu, Dec 6, 2018 at 10:53 PM John Naylor wrote: >> >> I've added an additional regression test for finding the right block >> and removed a test I thought was redundant. I've kept the test file in >> its own schedule. >>

Re: Thinking about EXPLAIN ALTER TABLE

2018-12-07 Thread John Naylor
ions, this could get unwieldy pretty fast. I imagine there could be a shorthand notation. -John Naylor

docs: outdated reference to recursive expression evaluation

2018-12-08 Thread John Naylor
evaluation in v10, there's probably a better example of recursive routines, but I'm not sure what that would be. -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-12-08 Thread John Naylor
On 12/8/18, Amit Kapila wrote: > On Fri, Dec 7, 2018 at 7:25 PM John Naylor wrote: > I couldn't resist the temptation to figure out what's going on here. > The newly added tests have deletes followed by vacuum and then you > check whether the vacuum has removed the data by c

Re: WIP: Avoid creation of the free space map for small tables

2018-12-02 Thread John Naylor
On 12/1/18, Amit Kapila wrote: > Can you check whether the number of pages after test are the same with > and without a patch in this setup? I did verify that the number of pages was as intended. -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-12-02 Thread John Naylor
lexity to remove the FSM. In the pg_upgrade case, however, it is simple to not copy the FSM. -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-12-02 Thread John Naylor
On 12/3/18, Amit Kapila wrote: > On Mon, Dec 3, 2018 at 9:46 AM Amit Kapila wrote: >> >> On Thu, Nov 29, 2018 at 3:07 PM John Naylor wrote: >> > >> > v8 code: > +fsm_local_set(Relation rel, BlockNumber new_nblocks) > +{ > + BlockNumber blkno, >

Re: automatically assigning catalog toast oids

2018-12-10 Thread John Naylor
cy change, not something that can happen because > one lazy hacker didn't follow the policy. On this point, I believe this could have happened anyway. pg_opclass has a mix of hand- and initdb-assigned oids, and there was nothing previously to stop that from creeping into any other catalog, as far as I can tell. -John Naylor

Re: docs: outdated reference to recursive expression evaluation

2018-12-10 Thread John Naylor
On 12/8/18, Tom Lane wrote: > John Naylor writes: >> In confg.sgml, in the section about max_stack_depth, there's this >> sentence: >> "The safety margin is needed because the stack depth is not checked in >> every routine in the server, but only in key p

Re: WIP: Avoid creation of the free space map for small tables

2018-11-29 Thread John Naylor
> I believe to convince myself it was working, I used the individual >> commands in the sql file in [3], then used the size function on the >> secondary. I'll redo that to verify. I've verified the standby behaves precisely as the primary, as far as the aforementioned script goes. -

Re: WIP: Avoid creation of the free space map for small tables

2018-12-06 Thread John Naylor
On 12/3/18, Amit Kapila wrote: > On Mon, Dec 3, 2018 at 11:15 AM John Naylor wrote: >> Per your recent comment, we no longer check relation size if we waited >> on a relation extension lock, so this is essentially a reversion to an >> earlier version. >> > > f

Re: WIP: Avoid creation of the free space map for small tables

2018-11-20 Thread John Naylor
ockBuffer(), visibilitymap_pin(), and GetVisibilityMapPins() don't call errors at this level. I don't immediately see any additional good places from which to clear the local map. -John Naylor

Re: inconsistency and inefficiency in setup_conversion()

2018-11-26 Thread John Naylor
v7 is a rebase over recent catalog changes. -John Naylor From 52aff43dcc91733c1b941fed4582d9c0602004d0 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Sat, 13 Oct 2018 19:28:08 +0700 Subject: [PATCH v7 1/2] Add pg_language lookup. This didn't seem worth doing before, but an upcoming commit

Re: WIP: Avoid creation of the free space map for small tables

2018-11-26 Thread John Naylor
On 11/24/18, Amit Kapila wrote: > On Fri, Nov 23, 2018 at 11:56 AM John Naylor wrote: >> On 11/16/18, Amit Kapila wrote: >> >> > >> > 3. >> > static int fsm_set_and_search(Relation rel, FSMAddress addr, uint16 >> > slot, >> > -u

Re: WIP: Avoid creation of the free space map for small tables

2018-11-22 Thread John Naylor
parate patch but not attached it for now. Also, we don't quite have a consensus on the threshold value, but I have set it to 4 pages for v8. If this is still considered too expensive (and basic tests show it shouldn't be), I suspect it'd be better to interleave the available block numbers as descr

Re: Sync ECPG scanner with core

2018-11-20 Thread John Naylor
check in the ecpg dir only takes 150ms. Compiling and linking the test executables takes another 2s, and running the tests takes another 5s on my machine. Even without setting up and tearing down the temp instance, preproc is trivial. -John Naylor

Re: [RFC] Removing "magic" oids

2018-11-20 Thread John Naylor
On 11/20/18, Andres Freund wrote: > I'm > sure we'll find some things to adapt around the margins, but imo the > patch as a whole looks pretty reasonable. bki.sgml is now a bit out of date. I've attached a draft fix. -John Naylor diff --git a/doc/src/sgml/bki.sgml b/doc/src/sgml/bki.s

Re: WIP: Avoid creation of the free space map for small tables

2018-11-18 Thread John Naylor
gt; 5. > +/*#define TRACE_TARGETBLOCK */ > > Debugging leftover, do you want to retain this and related stuff > during the development of patch? I modeled this after TRACE_VISIBILITYMAP in visibilitymap.c. It's useful for development, but I don't particularly care whether it's in the final verision. Also, I found an off-by-one error that caused an unnecessary smgrexists() call in tables with threshold + 1 pages. This will be fixed in the next version. -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-11-19 Thread John Naylor
On 11/19/18, Amit Kapila wrote: > On Mon, Nov 19, 2018 at 7:30 AM John Naylor wrote: >> Let's say we have to wait to acquire a relation extension lock, >> because another backend had already started extending the heap by 1 >> block. We call GetPageWithFreeSpace() and now the

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-08 Thread John Naylor
On Tue, Jan 8, 2019 at 3:04 PM Tom Lane wrote: > > I'll take a crack at separating into a module. I'll wait a bit in > > case there are any stylistic suggestions on the patch as it stands. > > I had a go at that myself. I'm sure there's plenty to criticize in > the result, but at least it

Re: WIP: Avoid creation of the free space map for small tables

2019-01-08 Thread John Naylor
On Sun, Dec 30, 2018 at 10:50 PM Amit Kapila wrote: > > On Sun, Dec 30, 2018 at 3:49 AM John Naylor wrote: > > > I also tried to insert more > > > records till 8 pages and same regression is observed! So I guess even > > > HEAP_FSM_CREATION_THRESHOLD = 4 is no

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-09 Thread John Naylor
ers still have a dire warning about ASCII order and binary search. Those could be softened to match the comment in gen_keywordlist.pl. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-01-07 Thread John Naylor
d to your other review comments later this week. -John Naylor diff --git a/src/backend/storage/freespace/freespace.c b/src/backend/storage/freespace/freespace.c index 2ab1cb58b7..d6352aabd9 100644 --- a/src/backend/storage/freespace/freespace.c +++ b/src/backend/storage/freespace/freespace.c @@ -1103,1

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-07 Thread John Naylor
On 1/6/19, Tom Lane wrote: > I've pushed that version (v8 + max_kw_len); if the buildfarm doesn't > fall over, we can move on with looking at hashing. Thank you. The API adjustment looks good, and I'm glad that splitting out the aux info led to even further cleanups. -John Naylor

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-04 Thread John Naylor
push this version beforehand? If it does, for v6 I've also done the rebase, updated the copyright year, and fixed an error in MSVC. -John Naylor From fb846eeabd450b6932e57f7e8d4f0aebc125d178 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Fri, 4 Jan 2019 14:35:40 -0500 Subject: [PAT

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-04 Thread John Naylor
es, there is still interest in using perfect hash functions, so thanks for this. I'm not knowledgeable enough to judge its implementation, so I'll leave that for others. -John Naylor

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-04 Thread John Naylor
patch (applies on top of v6), gzipped to avoid confusing the cfbot. -John Naylor keyword-nostruct.patch.gz Description: GNU Zip compressed data

Re: WIP: Avoid creation of the free space map for small tables

2019-01-10 Thread John Naylor
t messages From 854ba27740ed26e0d3e89f6228186f2b0914b21e Mon Sep 17 00:00:00 2001 From: John Naylor Date: Thu, 10 Jan 2019 16:37:57 -0500 Subject: [PATCH v12 1/2] Avoid creation of the free space map for small heap relations. Previously, all heaps had FSMs. For very small tables, this means that the FSM took up more space th

Re: unnecessary creation of FSM files during bootstrap mode

2019-01-10 Thread John Naylor
Thought I'd ping... (sorry for the top post) On Sat, Dec 15, 2018 at 12:02 AM Amit Kapila wrote: > > As pointed out by John Naylor [1], it seems during bootstrap mode, we > are always creating FSM files which are not required. In commit's > b9d01fe288 and 3908473c80, we have add

Re: WIP: Avoid creation of the free space map for small tables

2019-01-09 Thread John Naylor
ready available. Good to hear. I'll clean up the every-other-page patch and include it in my next version. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-09 Thread John Naylor
On Wed, Jan 9, 2019 at 2:04 PM Tom Lane wrote: > > I wrote: > > John Naylor writes: > >> -There is a bit of a cognitive clash between $case_sensitive in > >> gen_keywordlist.pl and $case_insensitive in PerfectHash.pm. They each > >> make sense in their ow

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-09 Thread John Naylor
On Tue, Jan 8, 2019 at 5:31 PM Tom Lane wrote: > > John Naylor writes: > > In the committed keyword patch, I noticed that in common/keywords.c, > > the array length is defined with > > ScanKeywordCategories[SCANKEYWORDS_NUM_KEYWORDS] > > but

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-09 Thread John Naylor
On Wed, Jan 9, 2019 at 2:44 PM Tom Lane wrote: > [patch to shrink oid index] It would help maintaining its newfound sveltness if we warned if a higher oid was assigned, as in the attached. I used 6200 as a soft limit, but that could be anything similiar. -- John Naylorht

automatically assigning catalog toast oids

2018-12-08 Thread John Naylor
. -John Naylor src/backend/catalog/genbki.pl | 26 ++ src/include/catalog/toasting.h | 32 ++-- 2 files changed, 28 insertions(+), 30 deletions(-) diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index edc8ea9f53

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-19 Thread John Naylor
found some libraries, but that would have more of the same difficulties in practicality that gperf had. [1] https://infoscience.epfl.ch/record/64394/files/triesearches.pdf -John Naylor

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-19 Thread John Naylor
gitignore, but maybe that's an abuse of it. Make check passes, but I don't know how well it stresses keyword use. I'll create a commitfest entry soon. -John Naylor .gitignore| 2 + src/common/gen_keywords.pl| 151 ++ s

Re: Delay locking partitions during INSERT and UPDATE

2019-01-24 Thread John Naylor
On Wed, Jan 23, 2019 at 7:56 PM David Rowley wrote: > On Thu, 24 Jan 2019 at 13:38, John Naylor wrote: > > Hmm, now instead of an 85x speedup over master in the 10k partition > > case, I only get 20x. Anyone else see this? > > What's it like with fsync off? No change. Just

Re: Delay locking partitions during INSERT and UPDATE

2019-01-24 Thread John Naylor
ion of whether the locking behavior with concurrent DDL is as safe as it is now. Anyone have anything else? -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-01-24 Thread John Naylor
On Wed, Jan 23, 2019 at 11:17 PM Amit Kapila wrote: > > On Thu, Jan 24, 2019 at 3:39 AM John Naylor > wrote: > > mkdir -p data1 data2 standby > > > > echo 'heap' > data1/foo > > echo 'fsm' > data1/foo_fsm > > > > # simulate stre

Re: Delay locking partitions during INSERT and UPDATE

2019-01-22 Thread John Naylor
The cfbot reports this patch no longer applies. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-01-27 Thread John Naylor
On Sat, Jan 26, 2019 at 2:14 PM Amit Kapila wrote: > > On Sat, Jan 26, 2019 at 5:05 AM John Naylor > wrote: > > > > So, in v19 we check pg_class.relpages and if it's > > a heap and less than or equal the threshold we call stat on the 0th > > segment to verify

Re: WIP: Avoid creation of the free space map for small tables

2019-01-27 Thread John Naylor
I will take one more pass on Monday morning (28th Jan) > > and will commit unless you or others see any problem. > > Pushed these two patches. Thank you for your input and detailed review! Thank you Mithun for testing! -- John Naylorhttps://www.2ndQuadrant.com/ Pos

Re: WIP: Avoid creation of the free space map for small tables

2019-01-27 Thread John Naylor
truncate the FSM. I wonder if the heap didn't get truncated either. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: Avoid creation of the free space map for small heap relations.

2019-01-27 Thread John Naylor
gt; > > No, it also depends on vacuum to remove rows and then truncate the > relation accordingly. That particular test could be removed -- it's just verifying behavior that's already been there for years and is a direct consquence of normal truncation combined with the addressing schem

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-30 Thread John Naylor
I believe I found a typo in mcv.c, fix attached. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services mcv-comment-fix.patch Description: Binary data

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread John Naylor
On Tue, Feb 26, 2019 at 8:59 AM Amit Kapila wrote: > I will look into it today and respond back with my findings. John, > see if you also get time to investigate this. Will do, but it will likely be tomorrow -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Devel

Re: WIP: Avoid creation of the free space map for small tables

2019-02-20 Thread John Naylor
al data. So, the length will be 1056 which is already > MAXALIGN. I took the new comments added in your latest version of the > patch and added them to the previous version of the patch. Kindly > see if I have not missed anything while merging the patch-versions? OK, that makes sense. Looks

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-26 Thread John Naylor
on the struct. I think the pointer to the README is sufficient. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-26 Thread John Naylor
On Tue, Feb 26, 2019 at 10:28 AM Amit Kapila wrote: > John, others, can you review my findings and patch? I can confirm your findings with your debugging steps. Nice work! -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Train

Re: WIP: Avoid creation of the free space map for small tables

2019-03-06 Thread John Naylor
www.postgresql.org/message-id/CACPNZCu4cOdm3uGnNEGXivy7Gz8UWyQjynDpdkPGabQ18_zK6g%40mail.gmail.com [2] https://www.postgresql.org/message-id/CACPNZCsWa%3Ddd0K%2BFiODwM%3DLEsepAHVJCoSx2Avew%3DxBEX3Ywjw%40mail.gmail.com -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Develop

Re: Why don't we have a small reserved OID range for patch revisions?

2019-02-23 Thread John Naylor
fault, but that could be changed if we agreed on the convention of > 9000 or whatever. In case we don't want to lose track of this, I added it to the March commitfest with a target of v13. (I didn't see a way to add it to the July commitfest) -- John Naylorhttps://www.2ndQuadrant.com

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-27 Thread John Naylor
; What if get some valid block from the first call to > GetPageWithFreeSpace via local map which has required space? I think > in that case we will need the call at hio.c:682. Am I missing > something? Are you referring to the call at line 393? Then the map will be cleared on line 507 before

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-27 Thread John Naylor
o "This prevents us from using the map". Perhaps also a comma after "first". -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-02-22 Thread John Naylor
size expressed as a fraction of current_setting('block_size'). > So instead of "1024" you'd write current_setting('block_size') / 8. > And then display the relation size in terms of pages, not bytes, so > divide pg_relation_size by block size. I've done this for v6, tested o

Re: WIP: Avoid creation of the free space map for small tables

2019-02-23 Thread John Naylor
On Fri, Feb 22, 2019 at 3:59 AM Amit Kapila wrote: > The reason for not pushing much on making the test pass for > nonstandard block sizes is that when I tried existing tests, there > were already some failures. FWIW, I currently see 8 failures (attached). -- John Naylor

outdated reference to tuple header OIDs

2019-03-14 Thread John Naylor
It seems this is a leftover from commit 578b229718e8. Patch attached. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services remove-ref-to-sys-oid-col.patch Description: Binary data

Re: WIP: Avoid creation of the free space map for small tables

2019-03-14 Thread John Naylor
ion header to be consistent with the > nearby code, run pgindent on the code and changed the commit message. Looks good to me, thanks. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Why don't we have a small reserved OID range for patch revisions?

2019-03-12 Thread John Naylor
On Tue, Mar 12, 2019 at 5:36 AM Tom Lane wrote: > This seems committable from my end --- any further comments? I gave it a read and it looks good to me, but I haven't tried to run it. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote

Re: WIP: Avoid creation of the free space map for small tables

2019-03-10 Thread John Naylor
re now not transferring *fsm files for small relations. How > about constructing a test where all relations are greater than 4 pages > and then try to upgrade them. We can check for a cluster with a > different number of relations say 10K, 20K, 50K, 100K. I have not, but

Re: Why don't we have a small reserved OID range for patch revisions?

2019-03-10 Thread John Naylor
event a maintenance headache, I didn't copy any of the formatting logic over. You'll also have to run reformat_dat_files.pl afterwards to restore that. It seems to work, but I haven't tested thoroughly. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQ

Re: WIP: Avoid creation of the free space map for small tables

2019-03-13 Thread John Naylor
5.64s The variation within the builds is up to +/- 0.2s, so there is no difference, as expected. 1 page: masterpatch 5.62s 4.25s Clearly, linking is much slower than stat. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-03-13 Thread John Naylor
ithout the > patch? If it is only with patch, then we definitely need to > investigate. Only with the patch. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: jsonpath

2019-03-19 Thread John Naylor
aw this in the committed patch. This is not default, it's chosen for maximum performance at the expense of binary/memory size. That's fine, but with a little effort you can also make the scanner non-backtracking for additional performance, as in the attached. -- John Naylorhttps://w

Re: jsonpath

2019-03-21 Thread John Naylor
ch case a comment to describe the additional complexity would be helpful. {unicode} { parseUnicode(yytext, yyleng); } {hex_char} { parseHexChars(yytext, yyleng); } {unicodefail} { yyerror(NULL, "Unicode sequence is invalid"); } {hex_fail} { yyerror(NULL, "Hex charact

Re: use Getopt::Long for catalog scripts

2019-02-07 Thread John Naylor
On Thu, Feb 7, 2019 at 6:53 PM David Fetter wrote: > [some style suggestions] I've included these in v2, and also some additional tweaks for consistency. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Servic

<    1   2   3   4   5   6   7   8   9   10   >